Senior Data Engineer
1 week ago
JOB DESCRIPTION : Data EngineerWe are seeking a highly skilled Data Engineer with deep expertise in Apache Kafka integration with Databricks, structured streaming, and large-scale data pipeline design using the Medallion Architecture. The ideal candidate will demonstrate strong hands-on experience in building and optimizing real-time and batch pipelines, and will be expected to solve real coding problems during the interview.Job Description : - Design, develop, and maintain real-time and batch data pipelines in Databricks.- Integrate Apache Kafka with Databricks using Structured Streaming.- Implement robust data ingestion frameworks using Databricks Autoloader.- Build and maintain Medallion Architecture pipelines across Bronze, Silver, and Gold layers.- Implement checkpointing, output modes, and appropriate processing modes in structured streaming jobs.- Design and implement Change Data Capture (CDC) workflows and Slowly Changing Dimensions (SCD) Type 1 and Type 2 logic.- Develop reusable components for merge/upsert operations and window function based transformations.- Handle large volumes of data efficiently through proper partitioning, caching, and cluster tuning techniques.- Collaborate with cross-functional teams to ensure data availability, reliability, and consistency.Must Have : - Apache Kafka : Integration, topic management, schema registry (Avro/JSON).- Databricks & Spark Structured Streaming : 1. Processing Modes: Append, Update, Complete 2. Output Modes: Memory, Console, File, Kafka, Delta 3. Checkpointing and fault tolerance - Databricks Autoloader : Schema inference, schema evolution, incremental loads.- Medallion Architecture implementation expertise.- Performance Optimization : i. Data partitioning strategies ii. Caching and persistence iii. Adaptive query execution and cluster configuration tuning - SQL & Spark SQL : Proficiency in writing efficient queries and transformations.- Data Governance : Schema enforcement, data quality checks, and monitoring.Good to Have : - Strong coding skills in Python and PySpark.- Experience working in CI/CD environments for data pipelines.- Exposure to cloud platforms (AWS/Azure/GCP).- Understanding of Delta Lake, time travel, and data versioning.- Familiarity with orchestration tools like Airflow or Azure Data Factory. (ref:hirist.tech)
-
Senior Data Engineer
2 weeks ago
Navi Mumbai, Maharashtra, India SoftPixel Solutions Pvt. Ltd. Full time ₹ 12,00,000 - ₹ 30,00,000 per yearJob Title: Senior Data Engineer – Commerce Platform | Data Architecture & IntegrationJob Summary:We are looking for an experienced Senior Data Engineer (12–15 years) to design, develop, and optimize data solutions that power insights and analytics across our commerce ecosystem. You will play a key role in building scalable data platforms, integrating...
-
Senior Data Engineer
4 days ago
Navi Mumbai, India Sulzer Full timeGF_IT-E660 Senior Data Engineer - Full Time - Navi Mumbai, India Sulzer is a leading engineering company with a proud heritage of innovation. Join our global team to grow your expertise and develop innovative solutions that enable a prosperous and more sustainable society. We are looking for a/an Senior Data Engineer to join our GF_IT team in India...
-
Senior Manager
2 weeks ago
Mumbai, India NTT DATA Full timeJob Description Make an impact with NTT DATA Join a company that is pushing the boundaries of what is possible. We are renowned for our technical excellence and leading innovations, and for making a difference to our clients and society. Our workplace embraces diversity and inclusion – it’s a place where you can grow, belong and thrive. Your day at NTT...
-
Senior Manager RD
2 weeks ago
Navi Mumbai, India Agency Data Full timeJob Description The Coca-Cola Company is seeking a highly skilled and experienced Senior Manager - Research & Development (R&D) to lead innovative projects and drive product development initiatives. This role requires a strategic thinker with strong leadership capabilities to oversee R&D teams, collaborate cross-functionally, and contribute to the...
-
Senior Data Engineer
3 weeks ago
Mumbai, India Fluent Health Full timeAbout Us: Fluent Health is a dynamic healthcare startup revolutionizing how you manage your healthcare and that of your family. The company will provide customers with high-quality, personalized options, credible information through trustworthy content, and absolute privacy. To assist us in our growth journey, we are seeking a highly motivated and...
-
Senior Data Engineer
3 weeks ago
Mumbai, India Fluent Health Full timeAbout Us: Fluent Health is a dynamic healthcare startup revolutionizing how you manage your healthcare and that of your family. The company will provide customers with high-quality, personalized options, credible information through trustworthy content, and absolute privacy. To assist us in our growth journey, we are seeking a highly motivated and...
-
Senior Data Engineer
4 weeks ago
Mumbai, India Fluent Health Full timeAbout Us:Fluent Health is a dynamic healthcare startup revolutionizing how you manage your healthcare and that of your family. The company will provide customers with high-quality, personalized options, credible information through trustworthy content, and absolute privacy. To assist us in our growth journey, we are seeking a highly motivated and experienced...
-
Lead / Senior Manager Controls Design SME
2 weeks ago
Navi Mumbai, India BDx Data Centers Full timeLead / Senior Manager Controls Design SMECompany OverviewBDx is Asia’s fastest-growing data center platform, delivering colocation, build-to-suit, managed services, and interconnection solutions for hyperscalers, large enterprises, and AI customers. Operating across high-growth markets—including Indonesia, Hong Kong, Singapore, and Taiwan—BDx currently...
-
Lead / Senior Manager Controls Design SME
2 weeks ago
Navi Mumbai, India BDx Data Centers Full timeLead / Senior Manager Controls Design SMECompany OverviewBDx is Asia’s fastest-growing data center platform, delivering colocation, build-to-suit, managed services, and interconnection solutions for hyperscalers, large enterprises, and AI customers. Operating across high-growth markets—including Indonesia, Hong Kong, Singapore, and Taiwan—BDx currently...
-
Lead / Senior Manager Controls Design SME
2 weeks ago
Navi Mumbai, India BDx Data Centers Full timeLead / Senior Manager Controls Design SMECompany OverviewBDx is Asia’s fastest-growing data center platform, delivering colocation, build-to-suit, managed services, and interconnection solutions for hyperscalers, large enterprises, and AI customers. Operating across high-growth markets—including Indonesia, Hong Kong, Singapore, and Taiwan—BDx currently...
