Sr Associate Big Data Engineering

Key Responsibilities Data Engineering Architecture Design develop and maintain high-performance data pipelines for structured and unstructured data using Azure Data Bricks and Apache Spark Build and manage scalable data ingestion frameworks for batch and real-time data processing Implement and optimize data lake architecture in Azure Data Lake to support analytics and reporting workloads Develop and optimize data models and queries in Azure Synapse Analytics to power BI and analytics use cases Cloud-Based Data Solutions Architect and implement modern data lakehouses combining the best of data lakes and data warehouses Leverage Azure services like Data Factory Event Hub and Blob Storage for end-to-end data workflows Ensure security compliance and governance of data through Azure Role-Based Access Control RBAC and Data Lake ACLs ETL ELT Development Develop robust ETL ELT pipelines using Azure Data Factory Data Bricks notebooks and PySpark Perform data transformations cleansing and validation to prepare datasets for analysis Manage and monitor job orchestration ensuring pipelines run efficiently and reliably Performance Optimization Optimize Spark jobs and SQL queries for large-scale data processing Implement partitioning caching and indexing strategies to improve performance and scalability of big data workloads Conduct capacity planning and recommend infrastructure optimizations for cost-effectiveness Collaboration Stakeholder Management Work closely with business analysts data scientists and product teams to understand data requirements and deliver solutions Participate in cross-functional design sessions to translate business needs into technical specifications Provide thought leadership on best practices in data engineering and cloud computing Documentation Knowledge Sharing Create detailed documentation for data workflows pipelines and architectural decisions Mentor junior team members and promote a culture of learning and innovation Required Qualifications Experience 7 years of experience in data engineering big data or cloud-based data solutions Proven expertise with Azure Data Bricks Azure Data Lake and Azure Synapse Analytics Technical Skills Strong hands-on experience with Apache Spark and distributed data processing frameworks Advanced proficiency in Python and SQL for data manipulation and pipeline development Deep understanding of data modeling for OLAP OLTP and dimensional data models Experience with ETL ELT tools like Azure Data Factory or Informatica Familiarity with Azure DevOps for CI CD pipelines and version control Big Data Ecosystem Familiarity with Delta Lake for managing big data in Azure Experience with streaming data frameworks like Kafka Event Hub or Spark Streaming Cloud Expertise Strong understanding of Azure cloud architecture including storage compute and networking Knowledge of Azure security best practices such as encryption and key management Preferred Skills Nice to Have Experience with machine learning pipelines and frameworks like MLFlow or Azure Machine Learning Knowledge of data visualization tools such as Power BI for creating dashboards and reports Familiarity with Terraform or ARM templates for infrastructure as code IaC Exposure to NoSQL databases like Cosmos DB or MongoDB Experience with data governance to Weekly Hours 40 Time Type Regular Location Hyderabad Andhra Pradesh India It is the policy of AT T to provide equal employment opportunity EEO to all persons regardless of age color national origin citizenship status physical or mental disability race religion creed gender sex sexual orientation gender identity and or expression genetic information marital status status with regard to public assistance veteran status or any other characteristic protected by federal state or local law In addition AT T will provide reasonable accommodations for qualified individuals with disabilities AT T is a fair chance employer and does not initiate a background check until an offer is made