Data Engineer Lead

About the Role We are seeking an experiencedData Engineer Leadto design, develop, and maintain scalable data solutions onAzure and Databricksas part of our enterprise data modernization initiatives. The ideal candidate will have a strong background indata pipeline development, data integration frameworks , andcloud-based data engineering , with deep expertise in tools such asDatabricks, Azure Data Factory, Alteryx, Ab Initio, Talend, and Informatica . This role will lead the design and delivery ofhigh-performance, governed data architecturesfor large-scale enterprise clients, driving data reliability, compliance, and analytics readiness.Key Responsibilities Design and implementscalable and reusabledata pipelinesusingDatabricks (PySpark, Delta Lake)andAzure Data Factory , ensuring high throughput and low latency. Develop and maintainCommon Data Platforms (CDP)andMedallion Architecture-based data lakesto consolidate legacy systems and improve data availability. Create and managebackend data servicesusingPython, Spark, or C# , ensuring performance, reliability, and scalability. Develop data ingestion frameworksusing metadata-driven and schema-drift–handling designs to support multi-source data integration. Leaddata integration and migration projectsusingTalend, Ab Initio, Alteryx, and Informatica , modernizing traditional ETL workflows into cloud-native pipelines. Implement data quality, lineage, and governance frameworks(Unity Catalog, Data Stewardship, Data Catalog, MDM). Design and maintainGDPR/CCPA-compliant data processingand ensure adherence to global data privacy standards. Build and optimizeFeature Store Data Martsto support AI/ML analytics teams and downstream BI consumers. Collaborate with cross-functional teams (Data Scientists, BI Developers, Product Owners) to define and deliver data-driven business outcomes. Mentor junior engineers inbest practices of data engineering , CI/CD, and infrastructure-as-code deployments. UtilizeRedisand caching mechanisms to optimize performance and reduce load latency in analytic workloads. Take end-to-end ownership of assigned modules ensuring timely delivery, documentation, and performance optimization.Technical Environment Cloud Platforms:Microsoft Azure (Data Factory, Synapse, Data Lake, Databricks), AWS (S3, EC2, Glue, Redshift) Data Engineering Tools:Databricks (DLT, Auto Loader, Unity Catalog), Talend, Ab Initio, Informatica, Alteryx Programming Languages:Python, PySpark, SQL, Unix Shell Scripting Data Modeling:Dimensional, Relational, and Medallion architecture DevOps / CI-CD:GitHub Actions, Azure DevOps, Databricks Asset Bundles (DABs) Data Governance & Security:RBAC/ABAC, Data Stewardship, MDM, GDPR/CCPA compliance Monitoring & Scheduling:Airflow, Logic Apps, Azure MonitorRequired Qualifications 12+ years of experience inData Engineering, Data Architecture, or ETL Development . Proven experience leadingAzure + Databricks –based modernization or migration programs. Deep understanding ofdata integration tools(Talend, Informatica, Ab Initio, Alteryx) andmetadata-driven pipeline design . Hands-on experience withPySpark, SQL, and Azure Data Services . Strong understanding ofdata modeling, data quality, and governance frameworks . Experience withreal-time and batch data ingestion , including Kafka, Kinesis, or Event Hub. Excellent problem-solving, communication, and stakeholder management skills.Preferred Qualifications Bachelor’s degree in computer science, or a related field Strong analytical and problem-solving skills Azure Data Engineer Certification or equivalent Familiarity withAI/ML data preparation and Feature Store design Experience implementingcost optimization and cluster governanceon DatabricksWhy Join Us Be part of adata modernization journeywith a global enterprise brand. Work with cutting-edgecloud and data engineering tools . Collaborate with ahighly skilled, cross-functional teamof architects, analysts, and data scientists. Opportunity to lead and innovate within a large-scale, multi-domaindata transformation initiative .