Senior Data Engineer

14 hours ago

Amravati, India Celebal Technologies Full time

Job Summary:We are seeking a highly skilled Data Engineer with deep expertise in Apache Kafka integration with Databricks, structured streaming, and large-scale data pipeline design using the Medallion Architecture. The ideal candidate will demonstrate strong hands-on experience in building and optimizing real-time and batch pipelines, and will be expected to solve real coding problems during the interview.Job Description:•Design, develop, and maintain real-time and batch data pipelines in Databricks.•Integrate Apache Kafka with Databricks using Structured Streaming.•Implement robust data ingestion frameworks using Databricks Autoloader.•Build and maintain Medallion Architecture pipelines across Bronze, Silver, and Gold layers.•Implement checkpointing, output modes, and appropriate processing modes in structured streaming jobs.•Design and implement Change Data Capture (CDC) workflows and Slowly Changing Dimensions (SCD) Type 1 and Type 2 logic.•Develop reusable components for merge/upsert operations and window function-based transformations.•Handle large volumes of data efficiently through proper partitioning, caching, and cluster tuning techniques.•Collaborate with cross-functional teams to ensure data availability, reliability, and consistency.Must Have:•Apache Kafka: Integration, topic management, schema registry (Avro/JSON).•Databricks & Spark Structured Streaming:oProcessing Modes: Append, Update, CompleteoOutput Modes: Memory, Console, File, Kafka, DeltaoCheckpointing and fault tolerance•Databricks Autoloader: Schema inference, schema evolution, incremental loads.•Medallion Architecture implementation expertise.•Performance Optimization:oData partitioning strategiesoCaching and persistenceoAdaptive query execution and cluster configuration tuning•SQL & Spark SQL: Proficiency in writing efficient queries and transformations.•Data Governance: Schema enforcement, data quality checks, and monitoring.Good to Have:•Strong coding skills in Python and PySpark.•Experience working in CI/CD environments for data pipelines.•Exposure to cloud platforms (AWS/Azure/GCP).•Understanding of Delta Lake, time travel, and data versioning.•Familiarity with orchestration tools like Airflow or Azure Data Factory.Please share your CVs shailendra.sharma@celebaltech.comImmediate Joiners Preferred

  • (Urgent Search) Senior Data Engineer

    3 weeks ago

    Amravati, India CareerXperts Consulting Full time

    We are looking for a highly skilled Senior Data Engineer to design, build, and optimize scalable data pipelines and infrastructure. The ideal candidate will have strong expertise in modern data engineering tools and frameworks, along with the ability to collaborate closely with data scientists, analysts, and product teams to deliver reliable and efficient...

  • Senior Data Engineer

    10 hours ago

    Amravati, India Koantek Full time

    We are seeking an experienced Databricks Full Stack Data Engineer with 5–6 years of industryexperience. The ideal candidate will have a proven track record of working on live projects,preferably within the manufacturing or energy sectors. He/she will play a key role in developingand maintaining scalable data solutions using Databricks and related...

  • Senior lead data engineer

    3 weeks ago

    Amravati, India TerraFinn Technologies Full time

    Company DescriptionTerra Finn Technologies outsources scholars from various top business schools in India and experienced employees to Fintech companies globally. The company focuses on providing expert employees to help fintech companies grow their business and achieve their goals efficiently. Terra Finn Technologies values long-term relationships with both...

  • Data Engineer

    4 days ago

    Amravati, India ValueLabs Full time

    Job Title: Lead Data EngineerCompany: ValueLabsLocation: HyderabadNotice Period : Immediate to 15 DaysJob Description:We are seeking a talented Data Engineer to join our dynamic team at ValueLabs. As a Data Engineer, you will be responsible for designing, developing, and maintaining scalable data pipelines that support our business operations. This role...

  • Data Engineer

    5 days ago

    Amravati, India ValueLabs Full time

    Job Title: Lead Data EngineerCompany: ValueLabsLocation: HyderabadNotice Period : Immediate to 15 DaysJob Description:We are seeking a talented Data Engineer to join our dynamic team at ValueLabs. As a Data Engineer, you will be responsible for designing, developing, and maintaining scalable data pipelines that support our business operations. This role...

  • Data engineer

    4 days ago

    Amravati, India GEMRAJ TECHNOLOGIES LIMITED Full time

    Job Title: Data Engineer Location: RemotePosition: Full-timeExp- Must Have 8+ YearsPosition OverviewWe are currently seeking a Data Engineer to join our technology team. In this role, you will work on building data-driven products that sit atop cloud-based data infrastructure. The ideal candidate is passionate about technology, software design, data...

  • Data Architect

    2 weeks ago

    Amravati, India Mastech Digital Full time

    Position: Staff, Data Engineer - Data ArchitectureLocation: Spire T110, Hyderabad Knowledge City, Madhapur, Hyderabad, Telangana, India, 500081Duration: Full timeComplete onsite (5 days/week)Job Description:Roles & Responsibilities:Shape and drive enterprise-wide data architecture strategy: Define and evolve the long-term technical vision for scalable,...

  • Data Engineer

    1 week ago

    Amravati, India Tata Consultancy Services Full time

    Proven expertise in Google Cloud Data Engineering services like BigQuery, Dataproc, and more.Strong programming skills in Python and PySpark for data manipulation and engineering.Solid experience with SQL and NoSQL databases.Deep understanding of data warehousing concepts and hands-on experience with modern data warehousing solutions.Proficiency in designing...

  • Senior Engineer – AI

    3 weeks ago

    Amravati, India IntraEdge Full time

    Seeking a hands-on Senior Engineer with proven expertise in Artificial Intelligence (AI/ML) and Java development. This role combines deep technical skills with the ability to design, build, and optimize AI-powered enterprise applications. The engineer will be responsible for developing scalable Java-based systems, integrating AI/ML models into production,...

  • Databricks Data Engineer

    2 weeks ago

    Amravati, India Insight Global Full time

    ** Immediate Joiner **Insight Global is seeking a Databricks Data Engineer in India with with 3–5 years of experience to support data engineering initiatives in the pharmaceutical domain. The ideal candidate will have hands-on expertise in Databricks, SQL, and Python, and a strong understanding of pharma/life sciences data. This role involves building and...