Senior Data Engineer
2 weeks ago
Skills Mandatory - Azure Fabric, Databricks, Data Modelling
Skills - Primary - PySpark, Databricks, SQL, Python
Skills - Good to have - Azure Data Factory,CICD, Azure Devops, Airflow
Qualification - Bachelors Degree
Total Experience 3-8 years
Relevant Experience 3 years
Work Location Cochin/TVM /Remote
Job Purpose
Own the end-to-end data lifecycle by collaborating with cross-functional teams in an Agile, sprint-based environment to achieve business goals.
Job Description / Duties & Responsibilities
Data Lifecycle Ownership:
• Design, build, and maintain scalable data pipelines to ingest raw data into our data lake (bronze layer).
• Develop and implement robust data models, transforming and enriching data through cleansing, validation (silver layer), and
aggregation to create business-ready datasets (gold layer).
• Establish and enforce data quality and governance standards across the entire data lifecycle.
• Optimize data platform performance for consumption by analytics, data science, and BI teams.
• Create and maintain comprehensive documentation for data pipelines, models, and architectures.
Agile Methodology & Collaboration:
• Actively participate in all Agile ceremonies, including sprint planning, daily stand-ups, reviews, and retrospectives.
• Work closely with Product Owners and stakeholders to understand business requirements and translate them into technical
specifications and data models.
• Collaborate with QA engineers to develop testing strategies, automate data quality checks, and resolve issues.
• Partner with software developers and platform engineers to ensure seamless integration of data sources and stability of the
data platform.
• Manage tasks, priorities, and timelines effectively within two-week sprint cycles, communicating progress and blockers
clearly.
• To adhere to the Information Security Management policies and procedures.
Job Specification / Skills and Competencies
Data Lifecycle Ownership (Fabric & Databricks):
• Design, build, and maintain scalable data ingestion pipelines using Azure Data Factory, Fabric Data Pipelines, and Databricks Notebooks (PySpark/SQL).
• Implement and manage a robust Medallion architecture (Bronze/Silver/Gold) across Azure Data Lake Storage (ADLS Gen2) and Fabric OneLake.
• Develop, optimize, and manage high-performance data models within Databricks (Delta Lake) and Fabric Lakehouses.
• Optimize Databricks cluster configurations and Fabric capacities for performance, scalability, and cost-efficiency.
Platform Automation & CI/CD:
• Implement and manage CI/CD pipelines for all data solutions using Azure DevOps (Repos, Pipelines).
• Automate the deployment of Databricks notebooks, Fabric artifacts (Lakehouses, pipelines, reports), and underlying Azure
resources (ARM/Bicep templates).
• Drive the data platform automation strategy, focusing on infrastructure as code (IaC), automated testing, and repeatable
release management.
• Monitor, troubleshoot, and optimize deployment pipelines for speed, reliability, and security.
Agile Methodology & Collaboration:
• Actively participate in all Agile ceremonies, including sprint planning, daily stand-ups, reviews, and retrospectives.
• Partner closely with Product Owners to understand business requirements and translate them into technical specifications
for Fabric and Databricks.
• Collaborate with QA engineers to integrate automated data quality testing into CI/CD pipelines.
• Work with Power BI developers and Data Scientists who consume the gold-layer data from Fabric, ensuring their deployment needs are met.
Required Qualifications
• 3+ years of proven experience as a Data Engineer with a strong focus on the Azure ecosystem.
• Deep, hands-on expertise with Azure Databricks (including PySpark, Spark SQL, Delta Lake).
• Demonstrable experience with Microsoft Fabric, including OneLake, Lakehouses, and Data Pipelines.
• Proven experience with Azure DevOps (Repos, Pipelines) to build and manage CI/CD pipelines for data workloads.
• Strong understanding of data platform automation and infrastructure as code (IaC) using ARM templates or Bicep.
• Strong proficiency in SQL and Python (PySpark).
• Solid understanding of data modelling principles and the Medallion architecture (bronze/silver/gold).
• Proven experience working in an Agile/Scrum development environment.
• Excellent problem-solving skills and a collaborative, team-first mindset.
Preferred Qualifications
• Previous domain experience in supply chain, logistics, or operations.
• Microsoft Certification: Azure Data Engineer (DP-203), Fabric Analytics Engineer (DP-600), or DevOps Engineer Expert (AZ-400).
• Experience with data streaming using Databricks Structured Streaming, Azure Event Hubs, or Fabric Real-Time Analytics
-
Senior Data Engineer
2 weeks ago
Thiruvananthapuram, India HireAlpha Full timeSenior Data Engineer (Informatica Platform Engineer / Administrator) – Bangalore We’re looking for a Senior Data Engineer with strong expertise in Informatica platform administration to join our Group IT Data Engineering team.This role is ideal for professionals who enjoy building and maintaining enterprise-grade data platforms that power large-scale...
-
Senior Data Engineer
4 weeks ago
Kochi, India Invokhr Full timePosition: Senior Data Engineer / Data Engineer Desired Experience: 3-8 years Salary: Best-in-industry You will act as a key member of the Data consulting team, working directly with the partners and senior stakeholders of the clients designing and implementing big data and analytics solutions. Communication and organisation skills are keys for this position,...
-
Senior Data Engineer
4 weeks ago
Kochi, India Invokhr Full timePosition: Senior Data Engineer / Data Engineer Desired Experience: 3-8 years Salary: Best-in-industry You will act as a key member of the Data consulting team, working directly with the partners and senior stakeholders of the clients designing and implementing big data and analytics solutions. Communication and organisation skills are keys for this position,...
-
Senior Data Engineer
4 weeks ago
Kochi, India Invokhr Full timeLOOKING FOR IMMEDIATE JOINERS OR 15 DAYS NOTICE PERIODS AND THIS IS WORK FROM HOME OPPORTUNITY Position: Senior Data Engineer / Data Engineer Desired Experience: 3-8 years Salary: Best-in-industry You will act as a key member of the Data consulting team, working directly with the partners and senior stakeholders of the clients designing and...
-
Senior Data Engineer
4 weeks ago
Kochi, India Invokhr Full timeLOOKING FOR IMMEDIATE JOINERS OR 15 DAYS NOTICE PERIODS AND THIS IS WORK FROM HOME OPPORTUNITY Position: Senior Data Engineer / Data Engineer Desired Experience: 3-8 years Salary: Best-in-industry You will act as a key member of the Data consulting team, working directly with the partners and senior stakeholders of the clients designing and...
-
Senior Data Scientist
1 week ago
Kochi, Thiruvananthapuram, India Pinnacle Software And Services Full time ₹ 20,00,000 - ₹ 25,00,000 per yearAre you passionate about building AI-driven solutions? We are looking for an experienced Senior Data Scientist / AI Engineer to join our team and play a key role in developing an AI-enabled ERP solution. You will work on designing scalable AI models, implementing ML pipelines, and exploring cutting-edge AI technologies to drive innovation. Additionally, this...
-
Senior Data Engineer
1 week ago
Kochi, India Velodata Global Pvt Ltd Full timeAbout the job :Role : Lead/ Senior Data Engineer - AWS, Python, PysparkMandatory Skills : AWS, Python, PysparkNotice Period : Immediate to 15 daysExperience Range : 7+ YearsLocation : Kerala (Local candidates only Kochi/Trivandrum)Job Overview : We are seeking an experienced Senior Data Engineer to lead the development of a scalable data ingestion framework...
-
Senior Data Engineer
6 days ago
Bengaluru, Kochi, Thiruvananthapuram, India Litmus7 Systems Consulting Full time ₹ 1,04,000 - ₹ 1,30,878 per yearWe are seeking an experienced Data Engineer to design and implement scalable data solutionsThe ideal candidate will have deep expertise in cloud data warehousing, ETL/ELT processes, data modeling, and business intelligence.Requirements:6+ years of experience in data engineering; Deep expertise with AWS Redshift, including data modeling, query optimization,...
-
Senior Data Engineer
2 weeks ago
Thiruvananthapuram, Kerala, India Quantiphi Full time ₹ 6,00,000 - ₹ 18,00,000 per yearDescription : While technology is the heart of our business, a global and diverse culture is the heart of our success. We love our people and we take pride in catering them to a culture built on transparency, diversity, integrity, learning and growth. If working in an environment that encourages you to innovate and excel, not just in professional...
-
Senior Data Integration Engineer
2 weeks ago
Kochi, India Indsafri Full timeJob Title - Senior Data Integration EngineerExperience - 6+YearsJob SummaryWe are seeking a highly experienced and strategic Senior Data Integration Engineer to architect, build, and manage the data pipelines that power our customer intelligence ecosystem. In this critical role, you will be the subject matter expert responsible for designing and implementing...