Data Engineer

Description :

Overview

Role : Data Engineer

Location : Bangalore

Job Overview :

We are seeking an experienced Data Architect to define and lead our enterprise data architecture strategy, enabling scalable analytics, machine learning, and business intelligence solutions. The role involves architecting and managing cloud-native data platforms using Databricks and AWS, designing lakehouse architectures, and building high-performance data pipelines for both batch and streaming workloads. You will collaborate closely with cross-functional teams to deliver governed, reliable, and analytics-ready data, support ML workflows, and drive data platform innovation. This position also includes mentoring data engineers and ensuring best practices in data governance, observability, and MLOps.

Key Responsibilities :

- Define and lead the data architecture vision and strategy, ensuring it supports analytics, ML, and business operations at scale.

- Architect and manage cloud-native data platforms using Databricks and AWS, leveraging the lakehouse architecture to unify data engineering and ML workflows.

- Build and optimize large-scale batch and streaming pipelines using Apache Spark, Airflow, and AWS Glue, ensuring high availability and fault tolerance.

- Design and develop data marts, warehouses, and analytics-ready datasets tailored for BI, product, and data science teams.

- Implement robust ETL/ELT pipelines with a focus on reusability, modularity, and automated testing.

- Enforce and scale data governance practices, including data lineage, cataloging, access management, and compliance with security and privacy standards.

- Partner with ML Engineers and Data Scientists to build and deploy ML pipelines, leveraging Databricks MLflow, Feature Store, and MLOps practices.

- Provide architectural leadership across data modeling, data observability, pipeline monitoring, and CI/CD for data workflows.

- Evaluate emerging tools and frameworks, recommending technologies that align with platform scalability and cost-efficiency.

- Mentor data engineers and foster a culture of technical excellence, innovation, and ownership across data teams.

Required Skills & Qualifications :

years of hands-on experience in data engineering, with at least 4 years in a lead or architectlevel role.

- Deep expertise in Apache Spark, with proven experience developing large-scale distributed data processing pipelines.

- Strong experience with Databricks platform and its internal ecosystem (e.g., Delta Lake, Unity Catalog, MLflow, Job orchestration, Workspaces, Clusters, Lakehouse architecture).

- Extensive experience with workflow orchestration using Apache Airflow.

- Proficiency in both SQL and NoSQL databases (e.g., Postgres, DynamoDB, MongoDB, Cassandra) with a deep understanding of schema design, query tuning, and data partitioning.

- Proven background in building data warehouse/data mart architectures using AWS services like Redshift, Athena, Glue, Lambda, DMS, and S3.

- Strong programming and scripting ability in Python (preferred) or other AWS-compatible languages.

- Solid understanding of data modeling techniques, versioned datasets, and performance tuning strategies.

- Hands-on experience implementing data governance, lineage tracking, data cataloging, and compliance frameworks (GDPR, HIPAA, etc.

- Experience with real-time data streaming using tools like Kafka, Kinesis, or Flink.

- Working knowledge of MLOps tooling and workflows, including automated model deployment, monitoring, and ML pipeline orchestration.

- Familiarity with MLflow, Feature Store, and Databricks-native ML tooling is a plus.

- Strong grasp of CI/CD for data and ML pipelines, automated testing, and infrastructure-as-code (Terraform, CDK, etc.

- Excellent communication, leadership, and mentoring skills with a collaborative mindset and the ability to influence across functions.

Preferred Skills :

yrs in data engineering (4 in lead/architect roles)

- Expert in Apache Spark, Databricks, and AWS data stack

- Strong in Python, SQL/NoSQL, and Airflow

- Skilled in data modeling, ETL, governance, and streaming (Kafka/Kinesis)

- Knowledge of MLOps, CI/CD, and IaC (Terraform/CDK)