[15h Left] Senior Data Engineer

1 week ago

Pune, India Celebal Technologies Full time

Job Summary: We are seeking a highly skilled Data Engineer with deep expertise in Apache Kafka integration with Databricks, structured streaming, and large-scale data pipeline design using the Medallion Architecture. The ideal candidate will demonstrate strong hands-on experience in building and optimizing real-time and batch pipelines, and will be expected to solve real coding problems during the interview. Job Description: •Design, develop, and maintain real-time and batch data pipelines in Databricks. •Integrate Apache Kafka with Databricks using Structured Streaming. •Implement robust data ingestion frameworks using Databricks Autoloader. •Build and maintain Medallion Architecture pipelines across Bronze, Silver, and Gold layers. •Implement checkpointing, output modes, and appropriate processing modes in structured streaming jobs. •Design and implement Change Data Capture (CDC) workflows and Slowly Changing Dimensions (SCD) Type 1 and Type 2 logic. •Develop reusable components for merge/upsert operations and window function-based transformations. •Handle large volumes of data efficiently through proper partitioning, caching, and cluster tuning techniques. •Collaborate with cross-functional teams to ensure data availability, reliability, and consistency. Must Have: •Apache Kafka: Integration, topic management, schema registry (Avro/JSON). •Databricks & Spark Structured Streaming: oProcessing Modes: Append, Update, Complete oOutput Modes: Memory, Console, File, Kafka, Delta oCheckpointing and fault tolerance •Databricks Autoloader: Schema inference, schema evolution, incremental loads. •Medallion Architecture implementation expertise. •Performance Optimization: oData partitioning strategies oCaching and persistence oAdaptive query execution and cluster configuration tuning •SQL & Spark SQL: Proficiency in writing efficient queries and transformations. •Data Governance: Schema enforcement, data quality checks, and monitoring. Good to Have: •Strong coding skills in Python and PySpark. •Experience working in CI/CD environments for data pipelines. •Exposure to cloud platforms (AWS/Azure/GCP). •Understanding of Delta Lake, time travel, and data versioning. •Familiarity with orchestration tools like Airflow or Azure Data Factory. Please share your CVs Immediate Joiners Preferred

  • Solution Architect

    5 days ago

    Pune, Maharashtra, India Left Right Mind Full time

    We are looking for a Software Solution Architect to join us in helping our clients benefit from our expertise in customer experience, enterprise solutions & product development. As a technologist at Left Right Mind, you will do the following activities - Drive technical projects and take pride in building systems that are usable, functional & scalable. -...

  • Data Engineer

    2 days ago

    Pune, India Bits In Glass Full time

    Title: Senior Data Engineer / ETL Engineer Location: Pune / Hyderabad / Mohali Job Summary: As a Senior Data Engineer / ETL Engineer, you will be instrumental in designing, developing, and optimizing data processing systems that support our organization's data initiatives. Your expertise in Bigdata and Google Cloud Platform (GCP) will be essential in...

  • Left Right Mind

    3 weeks ago

    Pune, India Left Right Mind Full time

    Job Summary :We are seeking a skilled and motivated AI/ML Engineer to join our technology team.In this role, you will be responsible for designing, developing, deploying, and maintaining machine learning and artificial intelligence models that power intelligent applications.Youll work on problems involving prediction, automation, personalization, NLP,...

  • [15h Left] Data Engineer

    2 weeks ago

    Pune, India The Machine Learning Company Full time

    (Immediate Joiners only) Job Description – Data Engineer (3–5 Years Experience) Location: Onsite (Pune) Employment Type: Full-time Experience: 3 to 5 Years Department: Data Engineering About The Machine Learning Company (TMLC) At TMLC, we build enterprise-grade AI and Data solutions that drive measurable business impact. From predictive analytics to...

  • 15h Left! Data Engineer

    1 week ago

    Pune, India Atica Global Full time

    About Us: Atica is a leading, tech-first remote sales & revenue management company for Hotels and Hotel operators in the US, enabled by a unified tech-led solution. We are a startup, funded by the top investors in the country, like Titan Capital, DIBS. As innovators at the forefront of the industry, we strive to deliver the highest quality service to all our...

  • ▷ 15h Left! Senior Data Engineer

    1 week ago

    Pune, India Jobted IN C2 Full time

    Your Role - Design and develop scalable data solutions in Azure using Microsoft Fabric - Build and optimize data pipelines (ETL/ELT) with Azure Data Factory, Synapse, Dataflows, and Pipelines - Integrate data from various structured and unstructured sources - Collaborate with AI/ML teams to implement GenAI and NLP models - Create and manage data models and...

  • ▷ [15h Left] Data Engineer

    2 weeks ago

    Pune, India Enterprise Minds, Inc Full time

    Data Engineer No of Positions-2 Exp-4-8 Years Mode-Hybrid Location-Pune Your Position As a data engineer, you will be responsible for delivering data intelligence solutions to our customers all around the globe, based on an innovative product, which provides insights into the performance of their material handling systems. You will be working on implementing...

  • (15h Left) Senior Developer

    4 weeks ago

    Pune, India Veca Consulting Pvt Ltd Full time

    Position Name: Senior Developer - Computational Geometry Work Experience: 5-10 years Location : Pune Shift : 09:30 AM to 7:00PM Position Overview: We are seeking a highly skilled and experienced Senior Developer with expertise in Computational Geometry to join our team. The Senior Developer will play a key role in developing algorithms, data structures, and...

  • Data Engineer Senior Consultant

    4 days ago

    Pune, India NTT Data Full time

    Job Description NTT DATA strives to hire exceptional, innovative and passionate individuals who want to grow with us. If you want to be part of an inclusive, adaptable, and forward-thinking organization, apply now. We are currently seeking a Data Engineer Senior Consultant to join our team in Pune, Mahrshtra (IN-MH), India (IN). Location: | Experience: 5+...

  • 15h Left: Data Engineer

    11 hours ago

    Pune, India Persistent Systems Full time

    About Position:We are looking for Strong Data engineers to join the Data Integration team and become part of our successful group- Role: Data Engineer - Location: All Persistent Locations - Experience: 5+ Years - Job Type: Full Time EmploymentWhat You'll Do:- Design, code, test, deploy, maintain, monitor and execute production ETL processes - Investigate...