Senior Data Engineer

4 weeks ago

Konkan Division, India Celebal Technologies Full time

Job Summary: We are seeking a highly skilled Data Engineer with deep expertise in Apache Kafka integration with Databricks, structured streaming, and large-scale data pipeline design using the Medallion Architecture. The ideal candidate will demonstrate strong hands-on experience in building and optimizing real-time and batch pipelines, and will be expected to solve real coding problems during the interview. Job Description: •Design, develop, and maintain real-time and batch data pipelines in Databricks. •Integrate Apache Kafka with Databricks using Structured Streaming. •Implement robust data ingestion frameworks using Databricks Autoloader. •Build and maintain Medallion Architecture pipelines across Bronze, Silver, and Gold layers. •Implement checkpointing, output modes, and appropriate processing modes in structured streaming jobs. •Design and implement Change Data Capture (CDC) workflows and Slowly Changing Dimensions (SCD) Type 1 and Type 2 logic. •Develop reusable components for merge/upsert operations and window function-based transformations. •Handle large volumes of data efficiently through proper partitioning, caching, and cluster tuning techniques. •Collaborate with cross-functional teams to ensure data availability, reliability, and consistency. Must Have: •Apache Kafka: Integration, topic management, schema registry (Avro/JSON). •Databricks & Spark Structured Streaming: oProcessing Modes: Append, Update, Complete oOutput Modes: Memory, Console, File, Kafka, Delta oCheckpointing and fault tolerance •Databricks Autoloader: Schema inference, schema evolution, incremental loads. •Medallion Architecture implementation expertise. •Performance Optimization: oData partitioning strategies oCaching and persistence oAdaptive query execution and cluster configuration tuning •SQL & Spark SQL: Proficiency in writing efficient queries and transformations. •Data Governance: Schema enforcement, data quality checks, and monitoring. Good to Have: •Strong coding skills in Python and PySpark. •Experience working in CI/CD environments for data pipelines. •Exposure to cloud platforms (AWS/Azure/GCP). •Understanding of Delta Lake, time travel, and data versioning. •Familiarity with orchestration tools like Airflow or Azure Data Factory. Please share your CVs shailendra.sharma@celebaltech.com Immediate Joiners Preferred

  • Senior Data Scientist

    2 weeks ago

    Konkan Division, India CarTrade.com Full time

    Key Responsibilities Data Science Design, develop, and deploy sophisticated machine learning models to solve complex business challenges. Build and optimize end-to-end ML pipelines, including data preprocessing, feature engineering, model training, and deployment. Drive Generative AI initiatives, creating innovative products and solutions. Conduct in-depth...

  • Senior Electrical Design Engineer

    4 weeks ago

    Konkan Division, India Integrated Personnel Services Limited Full time

    skills : - Senior Electrical Engineer Location : - Navi Mumbai Prepare Basis of Design, Load list, Single Line Diagram, Schematics. Power distribution from 33kV / 22kV and downstream based on power ratings of equipment to various voltage levels 11kV / 6.6kV / 3.3kV & 415V. Other national standard voltage levels for GID project. Calculations for Transformer...

  • Senior Piping Engineer

    1 week ago

    Konkan Division, India MCE Consulting Engineers Pvt Ltd Full time

    Preparation of Plot Plan, Preparation of Piping Layout, equipment Layout. Preparation of Valve Data sheet Preparation of piping items/ component data sheet Preparation of pipe support and pipe load calculation for civil. Checking of piping isometrics. Preparation of Nozzle orientation drawings & Vessel trims. Preparation and creating Piping BOM & Support...

  • Senior Data Engineer

    1 week ago

    Bangalore Division, India Elements HR Services Pvt Ltd Full time

    Senior Data Engineer Title Senior Data Engineer Department ISS Location Bengaluru Reports To Data Engineers – Lead Level 4,5,6 We share a commitment to making things better for clients and each other. We continually explore new technology and different ways of working to put our clients first. So, bring your boldest ideas to our (name of team/business...

  • Senior Data Engineer

    5 days ago

    Bangalore Division, India Veraxion Full time

    We’re Hiring: Senior Data Engineer (Azure) We are looking for a Senior Data Engineer who can design, build, and scale modern data platforms that power analytics, decision-making, and AI across the organization. If you love solving complex data problems, architecting scalable pipelines, and working with cutting-edge Azure technologies — this role is for...

  • Senior Data Engineer

    3 days ago

    Pune Division, India SPIRO Full time

    Position: Senior Data Engineer Location: Pune Experience: 5+ years Job Summary: We are seeking an experienced Senior Data Engineer with a strong background in PySpark, Spark, and Big Data technologies. The ideal candidate will have over five years of hands-on experience designing, building, and optimizing large-scale data processing systems and pipelines....

  • Senior Sales Executive

    2 weeks ago

    Konkan Division, India Aurionpro Full time

    The Senior Sales Executive will be responsible for driving business growth through strategic sales of IT solutions and services, focusing on Security, Cloud, and Data, to large and mid-sized enterprises in the BFSI sector. The role requires strong solution-selling skills, deep domain knowledge of BFSI IT environments, and the ability to engage with C-level...

  • Senior Data Engineer

    1 day ago

    Gurgaon Division, India Atomic North Full time

    Position: Senior Data Engineer (Azure) Experience: 6+ years Skills: Azure Databricks, PySpark, Python, SQL, Azure Data Factory, Data Modelling, Data Migration, Data Warehousing Location: India (Remote / Hybrid options available) Interview Mode: Virtual Notice Period: Immediate to 30 Days 💼 Key Responsibilities Design and develop data pipelines and ETL...

  • Lead - Data Science

    4 weeks ago

    Konkan Division, India ICICIDirect Full time

    Experience: 12- 16 Years About the Role: We are looking for a seasoned professional with deep expertise in analytics and AI, particularly within the BFSI domain. The ideal candidate will bring strong hands-on experience in Python-based statistical and machine learning model development, along with exposure to AWS AI stack and GenAI technologies. This role...

  • Senior Data Engineer

    4 weeks ago

    Bangalore Division, India USEReady Full time

    Job Title: Senior Databricks Engineer Experience Level: 5-8 Years Job Summary As a Senior Databricks Engineer, you will be responsible for designing, developing, and optimizing our data architecture and pipelines on the Databricks Lakehouse Platform. You will leverage your deep expertise in Spark, Delta Lake, and cloud technologies to build scalable and...