Senior Data Engineer

Senior Data Engineer (Azure Databricks / Spark) – Pune

Experience:
8–10 Years

Work Mode:
On-Site Only
(Pune)

Key Responsibilities

• Lead Data Engineering Projects:
  Drive the design, development, and deployment of end-to-end data ingestion pipelines using
  Azure Databricks
  ,
  Apache Spark
  , and related technologies—ensuring scalability, performance, and efficiency.
• Architect Robust Data Solutions:
  Design
  high-performance, resilient, and scalable data architectures
  following best practices for Azure Databricks and Spark environments.
• Mentor & Lead Teams:
  Provide
  technical leadership, guidance, and mentorship
  to data engineers, fostering a culture of innovation, collaboration, and continuous learning.
• Cross-Functional Collaboration:
  Work closely with
  data scientists, analysts, and business stakeholders
  to translate business requirements into data-driven solutions.
• Performance Optimization:
  Tune Spark jobs, optimize Databricks clusters, and troubleshoot data pipeline bottlenecks for maximum
  speed, reliability, and cost efficiency
  .
• Code Quality & Standards:
  Implement and uphold best practices in
  coding standards, version control, testing, and documentation
  .
• Azure Service Integration:
  Integrate pipelines seamlessly with
  Azure Data Lake Storage, Azure Blob Storage, Azure Synapse Analytics, and Azure SQL Data Warehouse
  .
• Ensure Data Quality:
  Implement
  validation and integrity checks
  to guarantee consistency, accuracy, and reliability across all ingested data.
• Continuous Improvement:
  Stay updated with
  emerging tools and trends
  in data engineering, recommending process and tool enhancements proactively.
• Risk & Issue Management:
  Identify potential risks or technical challenges early, and resolve them efficiently to maintain project timelines.

Qualifications

• Education:
  Bachelor's or Master's degree in
  Computer Science, IT, Engineering
  , or a related field.
• Experience:
• 8+ years of
  hands-on data engineering
  experience.
• Proven expertise in
  Azure Databricks, Apache Spark, Python/Scala
  , and data ingestion pipelines.
• Proficiency with
  CI/CD
  , scripting, and
  data processing automation
  .
• Experience working with
  Parquet, Delta, and Iceberg
  file formats.
• Familiarity with
  Kafka
  or equivalent
  streaming technologies
  for real-time data ingestion.
• Strong understanding of
  data governance and security
  principles within Azure.
• Deep experience across
  Azure Data Services
  (Blob Storage, Data Lake, SQL Data Warehouse, Event Hubs, Functions, etc.).
• Practical knowledge of
  modern data architectures
  , data lakes, and data warehouses.
• Working knowledge of
  CI/CD pipelines (Jenkins, Git)
  and
  agile methodologies
  .
• Understanding of
  cloud infrastructure and architectural principles
  , especially within Azure.

Technical Skills

• Expert-level proficiency in
  Spark
  and
  Spark Streaming
  (optimization, debugging, troubleshooting).
• Solid hands-on expertise in
  Azure Databricks
  for distributed data processing.
• Strong programming in
  Python
  and
  Scala
  for data transformation and ETL.
• Advanced
  SQL
  skills for managing and processing large datasets.
• Experience with
  Parquet, Delta Lake, Iceberg, and ORC
  formats.

Leadership & Soft Skills

• Proven ability to
  lead and mentor
  a team of high-performing data engineers.
• Strong
  analytical and problem-solving
  mindset with a proactive approach.
• Excellent
  communication and stakeholder management
  skills.
• Ability to thrive in a
  fast-paced, dynamic environment
  while managing multiple priorities.
• Highly
  collaborative
  , organized, and quality-focused professional.

Important Note

This position is
strictly On-Site
and based in
Pune / Trient locations only
.

Please
do not apply
if you are expecting a
hybrid or remote work model
.

The
maximum compensation
for this role is
₹17–18 LPA
(fixed range).