Senior Data Engineer

Job Title : Senior Data Engineer - Azure | ADF | Databricks | PySpark | AWS

Location : Bangalore, Hyderabad, Chennai (Hybrid Mode)

Experience Required : 5+ Years (Budget around 25 LPA)

Employment Type : Full-Time

Industry : IT Services & Consulting / Data & Analytics

Notice Period : Immediate to 30 days preferred

Job Description :

We are looking for a Senior Data Engineer who is passionate about designing and developing scalable data pipelines, optimizing data architecture, and working with advanced big data tools and cloud platforms. This is a great opportunity to be a key player in transforming data into meaningful insights by leveraging modern data engineering practices on Azure, AWS, and Databricks.

You will be working with cross-functional teams including data scientists, analysts, and software engineers to deliver robust data solutions. The ideal candidate will be technically strong in Azure Data Factory, PySpark, Databricks, and AWS services, and will have experience in building end-to-end ETL workflows and driving business impact through data.

Key Responsibilities :

- Design, build, and maintain scalable and reliable data pipelines and ETL workflows

- Implement data ingestion and transformation using Azure Data Factory (ADF) and Azure Databricks (PySpark)

- Work across multiple data platforms including Azure, AWS, Snowflake, and Redshift

- Collaborate with data scientists and business teams to understand data needs and deliver solutions

- Optimize data storage, processing, and retrieval for performance and cost-effectiveness

- Develop data quality checks and monitoring frameworks for pipeline health

- Ensure data governance, security, and compliance with industry standards

- Lead code reviews, set data engineering standards, and mentor junior team members

- Propose and evaluate new tools and technologies for continuous improvement

Must-Have Skills :

- Strong programming skills in Python, SQL, or Scala

- Azure Data Factory, Azure Databricks, Synapse Analytics

- Hands-on with PySpark, Spark, Hadoop, Hive

- Experience with cloud platforms (Azure preferred; AWS/GCP acceptable)

- Data Warehousing : Snowflake, Redshift, BigQuery

- Strong ETL/ELT pipeline development experience

- Workflow orchestration tools such as Airflow, Prefect, or Luigi

- Excellent problem-solving, debugging, and communication skills

Nice to Have :

- Experience with real-time streaming tools (Kafka, Flink, Spark Streaming)

- Exposure to data governance tools and regulations (GDPR, HIPAA)

- Familiarity with ML model integration into data pipelines

- Containerization and CI/CD exposure : Docker, Git, Kubernetes (basic)

- Experience with Vector databases and unstructured data handling

Technical Environment :

- Programming : Python, Scala, SQL

- Big Data Tools : Spark, Hadoop, Hive

- Cloud Platforms : Azure (ADF, Databricks, Synapse), AWS (S3, Glue, Lambda), GCP

- Data Warehousing : Snowflake, Redshift, BigQuery

- Databases : PostgreSQL, MySQL, MongoDB, Cassandra

- Orchestration : Apache Airflow, Prefect, Luigi

- Tools : Git, Docker, Azure DevOps, CI/CD pipelines

Soft Skills :

- Strong analytical thinking and problem-solving abilities

- Excellent verbal and written communication

- Collaborative team player with leadership qualities

- Self-motivated, organized, and able to manage multiple projects

Education & Certifications

- Bachelor's or Master's Degree in Computer Science, IT, Engineering, or equivalent

- Cloud certifications (e.g., Microsoft Azure Data Engineer, AWS Big Data) are a plus

Key Result Areas (KRAs) :

- Timely delivery of high-performance data pipelines

- Quality of data integration and governance compliance

- Business team satisfaction and data readiness

- Proactive optimization of data processing workloads

Key Performance Indicators (KPIs) :

- Pipeline uptime and performance metrics

- Reduction in overall data latency

- Zero critical issues in production post-release

- Stakeholder satisfaction score

- Number of successful integrations and migrations