Senior Data Engineer

About Delhivery:

Delhivery is India's leading fulfillment platform for digital commerce. With a vast logistics network spanning 18,000+ pin codes and over 2,500 cities, Delhivery provides a comprehensive suite of services including express parcel transportation, freight solutions, reverse logistics, cross-border commerce, warehousing, and cutting-edge technology services. Since 2011, we've fulfilled over 550 million transactions and empowered 10,000+ businesses, from startups to large enterprises.

Vision
:

To become the operating system for commerce in India by combining world-class infrastructure, robust logistics operations, and technology excellence
.

About the Role: Senior Data Enginee
r

We're looking for
a Senior Data Engine
er who can design, optimize, and own our high-throughput data infrastructure. You'll work across batch and real-time pipelines, scale distributed processing on petabyte-scale data, and bring AI-assisted tooling into your workflow for debugging, testing, and documentation. This is a hands-on role where you'll work with a wide range of big data technologies (Spark, Kafka, Hive, Hudi/Iceberg, Databricks, EMR), data modeling best practices, and real-time systems to power analytics, data products, and machine learning. As a senior engineer, you'll review complex pipelines, manage SLAs, and mentor junior team members — while leveraging GenAI tools to scale your impact.

What You'll Do

• Build and optimize scalable batch and streaming data pipelines using Apache Spark, Kafka, Flink, Hive, and Airflow.
• Design and implement efficient data lake architectures with Hudi, Iceberg, or Delta for versioning, compaction, schema evolution, and time travel.
• Architect and maintain cloud-native data systems (AWS EMR, S3, Glue, Lambda, Athena), focusing on cost, performance, and availability.
• Model complex analytical and operational data workflows for warehouse and data lake environments.
• Own pipeline observability — define and monitor SLAs, alerts, and lineage across batch and real-time systems.
• Debug performance bottlenecks across Spark, Hive, Kafka, and S3 — optimizing jobs with broadcast joins, file formats, resource configs, and partitioning strategies.
• Leverage AI tools (e.g., Cursor AI, Copilot, Gemini, Windsurf) for Code generation and refactoring of DAGs or Spark jobs
• Debugging logs, stack traces, and SQL errors. Generating tests for data pipelines
• Documenting complex pipeline dependencies and architecture
• Collaborate with product, analytics, data science, and platform teams to deliver end-to-end data products.
• Mentor junior engineers and establish AI-native development workflows, including prompt libraries and automation best practices.

What We're Looking For:

• Experience in building and maintaining large-scale data systems.
• Strong hands-on experience with Apache Spark, Kafka, Hive, and Airflow in production.
• Deep knowledge of the Hadoop ecosystem (HDFS, YARN, MapReduce tuning, NameNodeHA).
• Expert in SQL (windowing, recursive queries, tuning) and experience with NoSQL stores (e.g., DynamoDB, HBase).
• Experience with trino/prestoExperience with cloud-native data platforms — especially AWS Glue, S3 lifecycle policies, EMR, and Athena.
• Working knowledge of file formats and internals like Parquet, Avro, and best practices for efficient storage.
• Familiarity with modern Lakehouse formats (Hudi, Iceberg, Delta Lake) and their compaction, versioning, and schema evolution.
• Hands-on experience managing Databricks or EMR.
• Solid grounding in data modeling, DWH design, and slowly changing dimensions (SCD).
• Strong programming in Python/Scala/Java, and ability to write clean, modular, testable code.
• Proficiency with CI/CD practices, Git, Jenkins/GitHub Actions for data engineering workflows.
• Bonus: Experience with distributed systems, consensus protocols, and real-time data guarantees.
• Passion for AI-native engineering — using and evolving prompt-based workflows for greater efficiency and quality.