Lead Data Engineer

Job Title : Lead Data Consultant / Senior Data Engineer

Experience Level : 10+ Years (with deep Azure experience)

Employment Type : Full-time / Contract

Role Overview :

As a Lead Data Consultant / Senior Data Engineer, you will design, develop, and lead the delivery of enterprise-scale, cloud-native data platforms for our clients, with a particular focus on Azure, Databricks, and modern Lakehouse architectures. Working under the direction of the Principal Data Consultant you will help shape go-to-market technical assets and production-ready solutions across data engineering, system integration, governance, and AI/ML-enablement.

This is a hands-on consulting role requiring both strong implementation expertise and the ability to influence design, compliance, and delivery decisions across diverse enterprise environmentsparticularly in Financial Services and Insurance (FSI).

Key Responsibilities :

• Architect and build Lakehouse solutions with bronze/silver/gold layers using Delta Lake and Databricks

• Implement scalable ETL/ELT pipelines using Azure Data Factory, Airflow, Databricks, and PySpark

• Define enterprise-level data architectures, including lineage, governance, and integration patterns

• Enable agent orchestration and validation pipelines to support dynamic, AIenabled processing flows

• Lead development of data ingestion, transformation, and orchestration frameworks across cloud-native environments

• Guide implementation of data governance and compliance standards specific to FSI, including lineage tracking and access controls

• Collaborate with cross-functional teams (platform, ML, API, compliance) to design integrated data + AI systems

• Develop CI/CD pipelines and IaC (Terraform) modules to automate provisioning and deployment of data infrastructure

• Mentor other engineers and ensure reusable, modular, well-documented assets are delivered

Technical Skills & Experience :

Data Platforms & Engineering :

• 10+ years of experience in data engineering, with deep Azure cloud experience

• Proven experience designing Lakehouse architectures and implementing bronze/silver/gold/curated layers

• Hands-on expertise with Databricks, Delta Lake, PySpark, and SQL

• Experience integrating Kafka, RDBMS, and unstructured data into cloud pipelines

• Knowledge of DLT-Meta frameworks, metadata-driven ELT development, and data mesh concepts

• Familiarity with GenAI, and its integration with structured/unstructured data

System Architecture :

• Ability to design systems spanning data layers, API integrations, and AI orchestration components

• Knowledge of agentic systems, validation workflows, and event-driven orchestration

Governance & Compliance :

• Strong understanding of data lineage, data quality, and access control models

• Experience applying FSI regulatory standards, including auditing and privacy best practices

Cloud & DevOps :

• Proficiency in Azure services, especially ADF, Synapse, Blob Storage, Key Vault, and Managed Identity

• CI/CD using GitHub, GitLab, or Azure DevOps, with automated deployment patterns

• Infrastructure automation using Terraform with Git-based workflows

• Development environments : VSCode, Python scripting, Git version control

Non-Technical & Consulting Skills :

• Agile delivery experience : refinement, estimation, MVP definition, backlog grooming

• Ability to translate technical solutions into high-level architecture artifacts and documentation (Markdown, Confluence, Lucidchart)

• Comfortable reviewing existing codebases and recommending paths for reuse or modernization

• Proven ability to mentor and unblock delivery teams, with experience guiding junior and mid-level engineers

• Strong communication skills for client interaction, executive presentations, and cross-team coordination

Job Type: Full-time

Pay: ₹700, ₹2,842,521.17 per year

Benefits:

• Health insurance
• Paid time off
• Provident Fund
• Work from home

Work Location: Remote