Senior Data Engineer

3 weeks ago

Amravati, India Celebal Technologies Full time

Job Summary:We are seeking a highly skilled Data Engineer with deep expertise in Apache Kafka integration with Databricks, structured streaming, and large-scale data pipeline design using the Medallion Architecture. The ideal candidate will demonstrate strong hands-on experience in building and optimizing real-time and batch pipelines, and will be expected to solve real coding problems during the interview.Job Description:•Design, develop, and maintain real-time and batch data pipelines in Databricks.•Integrate Apache Kafka with Databricks using Structured Streaming.•Implement robust data ingestion frameworks using Databricks Autoloader.•Build and maintain Medallion Architecture pipelines across Bronze, Silver, and Gold layers.•Implement checkpointing, output modes, and appropriate processing modes in structured streaming jobs.•Design and implement Change Data Capture (CDC) workflows and Slowly Changing Dimensions (SCD) Type 1 and Type 2 logic.•Develop reusable components for merge/upsert operations and window function-based transformations.•Handle large volumes of data efficiently through proper partitioning, caching, and cluster tuning techniques.•Collaborate with cross-functional teams to ensure data availability, reliability, and consistency.Must Have:•Apache Kafka: Integration, topic management, schema registry (Avro/JSON).•Databricks & Spark Structured Streaming:oProcessing Modes: Append, Update, CompleteoOutput Modes: Memory, Console, File, Kafka, DeltaoCheckpointing and fault tolerance•Databricks Autoloader: Schema inference, schema evolution, incremental loads.•Medallion Architecture implementation expertise.•Performance Optimization:oData partitioning strategiesoCaching and persistenceoAdaptive query execution and cluster configuration tuning•SQL & Spark SQL: Proficiency in writing efficient queries and transformations.•Data Governance: Schema enforcement, data quality checks, and monitoring.Good to Have:•Strong coding skills in Python and PySpark.•Experience working in CI/CD environments for data pipelines.•Exposure to cloud platforms (AWS/Azure/GCP).•Understanding of Delta Lake, time travel, and data versioning.•Familiarity with orchestration tools like Airflow or Azure Data Factory.Please share your CVs shailendra.sharma@celebaltech.comImmediate Joiners Preferred

  • Senior Data Engineer

    3 weeks ago

    Amravati, India Koantek Full time

    We are seeking an experienced Databricks Full Stack Data Engineer with 5–6 years of industryexperience. The ideal candidate will have a proven track record of working on live projects,preferably within the manufacturing or energy sectors. He/she will play a key role in developingand maintaining scalable data solutions using Databricks and related...

  • Senior data scientist

    2 weeks ago

    Amravati, India ValueMatrix.AI Full time

    A Behavioural sciences Deep Tech company. Role Description This is a full-time remote role for a Senior Data Scientist. The Senior Data Scientist will be responsible for performing data analysis, data visualization, and developing data science models. The role involves designing and implementing statistical models, providing insights through data analytics,...

  • Principal Data Engineer

    2 days ago

    amravati, India CodeMyMobile Full time

    Experience Required - 7 to 10 YearsHow to Apply: Are you a Data Engineer who cares about clean engineering, autonomy, and solving real data challenges? If this sounds like you, we'd love to connect Email your application to careers@codemymobile.com with:Your GitHub links or other professional works.Your resume (PDF or online profile).A Data Engineering...

  • Data engineer

    2 weeks ago

    Amravati, India Recro Full time

    Key Responsibilities include: Data Pipeline Engineering: Design, build, and maintain ingestion, transformation, and storage pipelines using Azure Data Factory, Synapse Analytics, and Data Lake. AI Data Enablement: Collaborate with AI Consultants to prepare and manage datasets for supervised, unsupervised, and synthetic learning workflows. POC Support:...

  • Data Engineer

    4 weeks ago

    Amravati, India ValueLabs Full time

    Job Title: Lead Data EngineerCompany: ValueLabsLocation: HyderabadNotice Period : Immediate to 15 DaysJob Description:We are seeking a talented Data Engineer to join our dynamic team at ValueLabs. As a Data Engineer, you will be responsible for designing, developing, and maintaining scalable data pipelines that support our business operations. This role...

  • Data Engineer

    4 weeks ago

    Amravati, India ValueLabs Full time

    Job Title: Lead Data EngineerCompany: ValueLabsLocation: HyderabadNotice Period : Immediate to 15 DaysJob Description:We are seeking a talented Data Engineer to join our dynamic team at ValueLabs. As a Data Engineer, you will be responsible for designing, developing, and maintaining scalable data pipelines that support our business operations. This role...

  • Data engineer

    4 weeks ago

    Amravati, India GEMRAJ TECHNOLOGIES LIMITED Full time

    Job Title: Data Engineer Location: RemotePosition: Full-timeExp- Must Have 8+ YearsPosition OverviewWe are currently seeking a Data Engineer to join our technology team. In this role, you will work on building data-driven products that sit atop cloud-based data infrastructure. The ideal candidate is passionate about technology, software design, data...

  • Data Engineer

    4 weeks ago

    Amravati, India Tata Consultancy Services Full time

    Proven expertise in Google Cloud Data Engineering services like BigQuery, Dataproc, and more.Strong programming skills in Python and PySpark for data manipulation and engineering.Solid experience with SQL and NoSQL databases.Deep understanding of data warehousing concepts and hands-on experience with modern data warehousing solutions.Proficiency in designing...

  • Manager, data scientist and machine learning engineer

    2 weeks ago

    Amravati, India Ocwen Financial Solutions Pvt. Ltd. - APAC Full time

    Position Summary: We’re seeking a Data Scientist to support the development and deployment of machine learning models and analytics solutions that improve decision-making across the mortgage lifecycle—from acquisition to servicing. In this hands-on role, you’ll contribute to building predictive models, customer segmentation tools, and automation...

  • Senior Manager, Data Scientist and Machine Learning Engineer

    2 weeks ago

    Amravati, India Ocwen Financial Solutions Pvt. Ltd. - APAC Full time

    POSITION SUMMARY:We are seeking a highly skilled and motivated individual contributor to join our Data Science and Analytics team as a Senior Manager based in India. This hands-on role requires deep expertise in SAS programming, with a strong focus on statistical modeling to support our mortgage and consumer lending businesses. The ideal candidate will be...