Senior Data Engineer

17 hours ago

Mumbai Maharashtra India, Maharashtra Celebal Technologies Full time

JOB DESCRIPTION Data Engineer Designation – Data Engineer Experience – 5+ Years Location Mumbai (onsite)


Job Summary: We are seeking a highly skilled Data Engineer with deep expertise in Apache Kafka integration with Databricks, structured streaming, and large-scale data pipeline design using the Medallion Architecture. The ideal candidate will demonstrate strong hands-on experience in building and optimizing real-time and batch pipelines, and will be expected to solve real coding problems during the interview. Job Description: • Design, develop, and maintain real-time and batch data pipelines in Databricks. • Integrate Apache Kafka with Databricks using Structured Streaming. • Implement robust data ingestion frameworks using Databricks Autoloader. • Build and maintain Medallion Architecture pipelines across Bronze, Silver, and Gold layers. • Implement checkpointing, output modes, and appropriate processing modes in structured streaming jobs. • Design and implement Change Data Capture (CDC) workflows and Slowly Changing Dimensions (SCD) Type 1 and Type 2 logic. • Develop reusable components for merge/upsert operations and window functionbased transformations. • Handle large volumes of data efficiently through proper partitioning, caching, and cluster tuning techniques. • Collaborate with cross-functional teams to ensure data availability, reliability, and consistency. Must Have: • Apache Kafka: Integration, topic management, schema registry (Avro/JSON). • Databricks & Spark Structured Streaming: o Processing Modes: Append, Update, Complete o Output Modes: Memory, Console, File, Kafka, Delta o Checkpointing and fault tolerance • Databricks Autoloader: Schema inference, schema evolution, incremental loads. • Medallion Architecture implementation expertise. • Performance Optimization: o Data partitioning strategies o Caching and persistence o Adaptive query execution and cluster configuration tuning • SQL & Spark SQL: Proficiency in writing efficient queries and transformations. • Data Governance: Schema enforcement, data quality checks, and monitoring. • Good to Have: • Strong coding skills in Python and PySpark. • Experience working in CI/CD environments for data pipelines. • Exposure to cloud platforms (AWS/Azure/GCP). • Understanding of Delta Lake, time travel, and data versioning. • Familiarity with orchestration tools like Airflow or Azure Data Factory.


Mandatory Hands-on Coding Assessment (During Interview): Candidates will be required to demonstrate hands-on proficiency in the following areas:

1. Window Functions: o Implement logic using ROW_NUMBER, RANK, and DENSE_RANK in Spark. o Use cases such as deduplication, ranking within groups.

2. Merge/Upsert Logic: o Write PySpark code to perform MERGE operations in Delta Lake.

3. SCD Implementation: o SCD Type 1: Overwriting existing records. o SCD Type 2: Versioning records with effective start/end dates or is_current flags.

4. CDC (Change Data Capture): o Capture and process changes using techniques such as: ▪ Comparison with previous snapshots ▪ Using audit columns or timestamps ▪ Kafka-based event-driven ingestion

  • Senior Data Engineer

    1 week ago

    Mumbai, Maharashtra, India, Maharashtra A.P. Moller - Maersk Full time

    Maersk Line Fleet Management and Technology is responsible for Ship Management of Maersk Line owned container vessels. As a part of the Energy Transition Transparency and Enablement team you will be responsible for developing the strong enterprise architecture, building effective data pipelines based on ADLS/MIDAS and providing end-to-end support on energy...

  • Senior Data Engineer

    17 hours ago

    Pune, Maharashtra, India, Maharashtra RiDiK Full time

    We’re looking for a Senior Data Engineer to support our team in PAN India. This role offers the opportunity to work on meaningful projects, collaborate with talented colleagues, and contribute to the success of a growing company. If you’re someone who takes initiative, values continuous learning, and thrives in a collaborative setting, we’d love to...

  • Senior Data Engineer

    1 week ago

    Pune, Maharashtra, India, Maharashtra Hireologist Full time

    Job Title: Big Data EngineerLocation: Pune (Hybrid)Experience - 5-7 YOESalary Range: ₹35–40 LPAAbout the RoleWe are looking for a Big Data Engineer to design, build, and scale high-performance data pipelines that power our products and insights. You’ll be working closely with product managers, architects, and engineering leads to define technical...

  • Senior Data Engineer

    1 week ago

    Mumbai, Maharashtra, India Oracle Financial Services Software Ltd Full time ₹ 15,00,000 - ₹ 25,00,000 per year

    Senior Consultant - Databricks Senior Data Engineer – Databricks (Azure/AWS) Role Overview: We are looking for a hands-on Senior Data Engineer experienced in migrating and building large-scale data pipelines on Databricks using Azure or AWS platforms. The role will focus on implementing batch and streaming pipelines, applying the bronze-silver-gold...

  • Senior Data Warehouse Architect

    17 hours ago

    Mumbai, Maharashtra, India, Maharashtra Talent Corner HR Services Pvt Ltd Full time

    Industry : HeathTechLocation: Mumbai.Designation: Senior Data ArchitectWORK IN OFFICE Working Days: Mon. to Fri.Working Time: 9 am to 6pmQualification: Bachelor's degree in Computer Science, Information Technology, or a related field (or equivalent work experience).Experience: min. 5 years with at least 2+ years in a senior or architectural role.MUST...

  • Senior Data Engineer( DWH)

    17 hours ago

    Pune, Maharashtra, India, Maharashtra Mastercard Full time

    Title and SummarySenior Data EngineerJob Description SummarySenior Data Engineer (DW)Mastercard is a global technology company in the payments industry. Our mission is to connect and power an inclusive, digital economy that benefits everyone, everywhere by making transactions safe, simple, smart, and accessible. Using secure data and networks, partnerships...

  • Senior Data Engineer

    2 weeks ago

    Mumbai, Maharashtra, India origin hr Full time ₹ 20,00,000 - ₹ 25,00,000 per year

    Job Description Senior Data Engineer Full time Locations: Bangalore, Delhi, Hyderabad, Mumbai, Pune Posted: 11 hours agoPosition Senior Data Engineer Experience: 8 years Job Location: Mumbai / Pune / Bangalore / Delhi NCR / HyderabadSkills Required:Developing, maintaining, and documenting reliable and scalable data infrastructure and data products in...

  • Senior Manager

    1 week ago

    Pune, Maharashtra, India, Maharashtra Opus Technologies Full time

    Job Title: Data Engineering LeaderLocation: PuneExperience: 15+ YearsEmployment Type: Full-TimeCompany Name: Opus TechnologiesAbout the Role:We are looking for a dynamic and visionary Data Engineering Leader to help build and scale our Data Practice. This leadership role is ideal for someone with deep technical expertise, strong delivery acumen, and a...

  • Senior Data Engineer

    2 weeks ago

    Mumbai, Maharashtra, India Uplers Full time ₹ 12,00,000 - ₹ 36,00,000 per year

    Experience: yearsSalary: Confidential (based on experience)Shift: (GMT+05:30) Asia/Kolkata (IST)Opportunity Type: RemotePlacement Type: Full time Permanent Position(*Note: This is a requirement for one of Uplers' client - Forbes Advisor)What do you need for this opportunity?Must have skills required: Airflow, dbt, Google CloudForbes Advisor is Looking...

  • Data Engineer

    2 weeks ago

    Mumbai, Maharashtra, India NTT DATA Global Delivery Services Ltd Full time ₹ 12,00,000 - ₹ 36,00,000 per year

    FID Federal Risk Applications Data Engineer Req ID: 341951 NTT DATA strives to hire exceptional, innovative and passionate individuals who want to grow with us. If you want to be part of an inclusive, adaptable, and forward-thinking organization, apply now. We are currently seeking a FID Federal Risk Applications Data Engineer to join our team in...