Azure Data Engineer

3 weeks ago

Nagpur, Maharashtra, India Celebal Technologies Full time
Job Title: Azure Data Engineer

Duration: Fulltime

Positions: Multiple

Experience Required: 3-5 Years & 6-8 Years

Job Summary: We are looking for a highly skilled Azure Data Engineer with a strong background in real-time and batch data ingestion and big data processing, particularly using Kafka and Databricks. The ideal candidate will have a deep understanding of streaming architectures, Medallion data models, and performance optimization techniques in cloud environments. This role requires hands-on technical expertise, including live coding during the interview process.

Key Responsibilities

• Design and implement streaming data pipelines integrating Kafka with Databricks using Structured Streaming.

• Architect and maintain Medallion Architecture with well-defined Bronze, Silver, and Gold layers.

• Implement efficient ingestion using Databricks Autoloader for high-throughput data loads.

• Work with large volumes of structured and unstructured data, ensuring high availability and performance.

• Apply performance tuning techniques such as partitioning, caching, and cluster resource optimization.

• Collaborate with cross-functional teams (data scientists, analysts, business users) to build robust data solutions.

• Establish best practices for code versioning, deployment automation, and data governance.

Required Technical Skills:

• Strong expertise in Azure Databricks and Spark Structured Streaming

• 3-8 Years' experience in Data Engineering

• Processing modes (append, update, complete)

• Output modes (append, complete, update)

• Checkpointing and state management

• Experience with Kafka integration for real-time data pipelines

• Deep understanding of Medallion Architecture

• Proficiency with Databricks Autoloader and schema evolution

• Deep understanding of Unity Catalog and Foreign catalog

• Strong knowledge of Spark SQL, Delta Lake, and DataFrames

• Expertise in performance tuning (query optimization, cluster configuration, caching strategies)

• Must have Data management strategies

• Excellent with Governance and Access management

• Strong with Data modelling, Data warehousing concepts, Databricks as a platform

• Solid understanding of Window functions

  • Data Engineer

    3 weeks ago

    Nagpur, Maharashtra, India Live Connections Full time

    Position: Data Engineer – Big Data & Azure Location: Bangalore, IndiaRole Purpose: We are seeking a highly skilled Data Engineer with expertise in Big Data, Python, and PySpark to design, develop, and optimize large-scale data solutions on Azure. The role involves working with complex structured and unstructured datasets to deliver high-quality analytics...

  • Senior Data Engineer

    1 day ago

    Nagpur, Maharashtra, India beBeeData Full time ₹ 1,50,00,000 - ₹ 2,10,00,000

    Senior Data Engineer - Azure CloudJob Title: Senior Data Engineer - Azure CloudLocation: RemoteNotice Period: Immediate Joiners OnlyExperience: 4–6 YearsWe are seeking a skilled Senior Data Engineer to work in our hybrid development + support team.About the Role:We require an experienced professional with expertise in building, maintaining, and supporting...

  • Senior Data Engineering Professional

    2 days ago

    Nagpur, Maharashtra, India beBeeDataEngineering Full time ₹ 20,00,000 - ₹ 30,00,000

    Job Overview:We are seeking an experienced Data Engineering Specialist to work on a short-term contract. The ideal candidate will have 7+ years of experience in designing and implementing data pipelines using Azure Data Factory, developing optimized Databricks notebooks with SQL & Python, and working with various file formats such as Parquet, Delta, and...

  • Chief Data Engineering Specialist

    5 hours ago

    Nagpur, Maharashtra, India beBeeData Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

    Azure Data Engineer PositionOur company is seeking a skilled Azure Data Engineer to spearhead data pipeline development and deliver scalable data solutions on the Azure platform.Key Responsibilities:Develop and implement comprehensive end-to-end data pipelines that integrate with various data sources.Ensure accurate data validation, transformation, and...

  • Data Engineering Expert

    5 days ago

    Nagpur, Maharashtra, India beBeeDataEngineering Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

    Job Title: Data Engineering ExpertAbout the RoleWe are seeking an experienced Data Engineer to join our team. This is a key position that requires a strong understanding of data engineering principles and practices.The ideal candidate will have hands-on experience with Informatica DEI, as well as a solid foundation in Azure Data Engineering stack, including...

  • Cloud Data Engineer

    1 day ago

    Nagpur, Maharashtra, India beBeeDataEngineer Full time ₹ 80,00,000 - ₹ 1,50,00,000

    Job Description:As a Data Engineer, you will be responsible for designing, building, and maintaining large-scale data processing systems on Azure Databricks. You will work with big data technologies such as PySpark, Delta Lake, and ADF to ensure the smooth flow of data from various sources to our data warehouses.We are looking for someone with hands-on...

  • Chief Cloud Architect: Data Engineering

    24 hours ago

    Nagpur, Maharashtra, India beBeeEngineer Full time ₹ 1,80,00,000 - ₹ 2,40,00,000

    Senior Cloud Data EngineerWe are seeking a seasoned Senior Cloud Data Engineer to join our team. The ideal candidate will have expertise in designing, developing, and optimizing cloud-native data pipelines on Microsoft Azure.Key Responsibilities:Data Ingestion and Processing: Design and implement scalable, robust, and secure data ingestion and processing...

  • Data Engineering Specialist

    2 days ago

    Nagpur, Maharashtra, India beBeeDataEngineer Full time ₹ 15,00,000 - ₹ 30,00,000

    Job Title: Data EngineerWe are seeking a skilled professional to fill this role.This position involves developing and optimizing data pipelines, working with streaming data, and collaborating with cross-functional teams to deliver high-quality solutions.Design, build, and refine data pipelines and ETL workflows.Analyze and process large datasets using Spark,...

  • Senior Data Systems Engineer

    7 hours ago

    Nagpur, Maharashtra, India beBeeDatabase Full time ₹ 1,80,00,000 - ₹ 2,10,00,000

    Job Position Overview:We are looking for a highly skilled data developer with extensive experience in Microsoft-based Database Engineering Technologies.The ideal candidate will have expertise in Transact-SQL Development, Data Warehousing, and ETL processes.Additional skills in Azure Data Lake Storage, Enterprise Job Scheduler, Eventhub / Stream Analytics,...

  • Master Data Architect

    5 hours ago

    Nagpur, Maharashtra, India beBeeDataEngineer Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

    Unlock your potential and unleash your career growth with a Data Engineer role.We are seeking a skilled and detail-oriented professional to design, build, and manage scalable data pipelines and enterprise data solutions.Design, develop, and manage ETL/ELT pipelines using Azure Data Factory (ADF) and Databricks for batch and real-time data...