Lead Data Engineer

Description :

Job Title : Lead Data Consultant / Senior Data Engineer

Experience Level : 10 Years (with deep Azure experience)

Employment Type : Full-time / Contract

Role Overview :

As a Lead Data Consultant / Senior Data Engineer, you will design, develop, and lead the delivery of enterprise-scale, cloud-native data platforms for our clients, with a particular focus on Azure, Databricks, and modern Lakehouse architectures. Working under the direction of the Principal Data Consultant you will help shape go-to-market technical assets and production-ready solutions across data engineering, system integration, governance, and AI/ML-enablement.

This is a hands-on consulting role requiring both strong implementation expertise and the ability to influence design, compliance, and delivery decisions across diverse enterprise environmentsparticularly in Financial Services and Insurance (FSI).

Key Responsibilities :

- Architect and build Lakehouse solutions with bronze/silver/gold layers using Delta Lake and Databricks

- Implement scalable ETL/ELT pipelines using Azure Data Factory, Airflow, Databricks, and PySpark

- Define enterprise-level data architectures, including lineage, governance, and integration patterns

- Enable agent orchestration and validation pipelines to support dynamic, AIenabled processing flows

- Lead development of data ingestion, transformation, and orchestration frameworks across cloud-native environments

- Guide implementation of data governance and compliance standards specific to FSI, including lineage tracking and access controls

- Collaborate with cross-functional teams (platform, ML, API, compliance) to design integrated data AI systems

- Develop CI/CD pipelines and IaC (Terraform) modules to automate provisioning and deployment of data infrastructure

- Mentor other engineers and ensure reusable, modular, well-documented assets are delivered

Technical Skills & Experience :

Data Platforms & Engineering :

years of experience in data engineering, with deep Azure cloud experience

- Proven experience designing Lakehouse architectures and implementing bronze/silver/gold/curated layers

- Hands-on expertise with Databricks, Delta Lake, PySpark, and SQL

- Experience integrating Kafka, RDBMS, and unstructured data into cloud pipelines

- Knowledge of DLT-Meta frameworks, metadata-driven ELT development, and data mesh concepts

- Familiarity with GenAI, and its integration with structured/unstructured data

System Architecture :

- Ability to design systems spanning data layers, API integrations, and AI orchestration components

- Knowledge of agentic systems, validation workflows, and event-driven orchestration

Governance & Compliance :

- Strong understanding of data lineage, data quality, and access control models

- Experience applying FSI regulatory standards, including auditing and privacy best practices

Cloud & DevOps :

- Proficiency in Azure services, especially ADF, Synapse, Blob Storage, Key Vault, and Managed Identity

- CI/CD using GitHub, GitLab, or Azure DevOps, with automated deployment patterns

- Infrastructure automation using Terraform with Git-based workflows

- Development environments : VSCode, Python scripting, Git version control

Non-Technical & Consulting Skills :

- Agile delivery experience : refinement, estimation, MVP definition, backlog grooming

- Ability to translate technical solutions into high-level architecture artifacts and documentation (Markdown, Confluence, Lucidchart)

- Comfortable reviewing existing codebases and recommending paths for reuse or modernization

- Proven ability to mentor and unblock delivery teams, with experience guiding junior and mid-level engineers

- Strong communication skills for client interaction, executive presentations, and cross-team coordination