Data Engineer Lead

About the Role We are seeking an experienced Data Engineer Lead to design, develop, and maintain scalable data solutions on Azure and Databricks as part of our enterprise data modernization initiatives. The ideal candidate will have a strong background in data pipeline development, data integration frameworks , and cloud-based data engineering , with deep expertise in tools such as Databricks, Azure Data Factory, Alteryx, Ab Initio, Talend, and Informatica . This role will lead the design and delivery of high-performance, governed data architectures for large-scale enterprise clients, driving data reliability, compliance, and analytics readiness. Key Responsibilities Design and implement scalable and reusable data pipelines using Databricks (PySpark, Delta Lake) and Azure Data Factory , ensuring high throughput and low latency. Develop and maintain Common Data Platforms (CDP) and Medallion Architecture-based data lakes to consolidate legacy systems and improve data availability. Create and manage backend data services using Python, Spark, or C# , ensuring performance, reliability, and scalability. Develop data ingestion frameworks using metadata-driven and schema-drift–handling designs to support multi-source data integration. Lead data integration and migration projects using Talend, Ab Initio, Alteryx, and Informatica , modernizing traditional ETL workflows into cloud-native pipelines. Implement data quality, lineage, and governance frameworks (Unity Catalog, Data Stewardship, Data Catalog, MDM). Design and maintain GDPR/CCPA-compliant data processing and ensure adherence to global data privacy standards. Build and optimize Feature Store Data Marts to support AI/ML analytics teams and downstream BI consumers. Collaborate with cross-functional teams (Data Scientists, BI Developers, Product Owners) to define and deliver data-driven business outcomes. Mentor junior engineers in best practices of data engineering , CI/CD, and infrastructure-as-code deployments. Utilize Redis and caching mechanisms to optimize performance and reduce load latency in analytic workloads. Take end-to-end ownership of assigned modules ensuring timely delivery, documentation, and performance optimization. Technical Environment Cloud Platforms: Microsoft Azure (Data Factory, Synapse, Data Lake, Databricks), AWS (S3, EC2, Glue, Redshift) Data Engineering Tools: Databricks (DLT, Auto Loader, Unity Catalog), Talend, Ab Initio, Informatica, Alteryx Programming Languages: Python, PySpark, SQL, Unix Shell Scripting Data Modeling: Dimensional, Relational, and Medallion architecture DevOps / CI-CD: GitHub Actions, Azure DevOps, Databricks Asset Bundles (DABs) Data Governance & Security: RBAC/ABAC, Data Stewardship, MDM, GDPR/CCPA compliance Monitoring & Scheduling: Airflow, Logic Apps, Azure Monitor Required Qualifications 12+ years of experience in Data Engineering, Data Architecture, or ETL Development . Proven experience leading Azure + Databricks –based modernization or migration programs. Deep understanding of data integration tools (Talend, Informatica, Ab Initio, Alteryx) and metadata-driven pipeline design . Hands-on experience with PySpark, SQL, and Azure Data Services . Strong understanding of data modeling, data quality, and governance frameworks . Experience with real-time and batch data ingestion , including Kafka, Kinesis, or Event Hub. Excellent problem-solving, communication, and stakeholder management skills. Preferred Qualifications Bachelor’s degree in computer science, or a related field Strong analytical and problem-solving skills Azure Data Engineer Certification or equivalent Familiarity with AI/ML data preparation and Feature Store design Experience implementing cost optimization and cluster governance on Databricks Why Join Us Be part of a data modernization journey with a global enterprise brand. Work with cutting-edge cloud and data engineering tools . Collaborate with a highly skilled, cross-functional team of architects, analysts, and data scientists. Opportunity to lead and innovate within a large-scale, multi-domain data transformation initiative .