Azure Data Engineer

2 weeks ago

Navi Mumbai, Maharashtra, India Celebal Technologies Full time

Job Title: Azure Data Engineer

Duration: Fulltime

Positions: Multiple

Experience Required: 3-5 Years & 6-8 Years

Job Summary: We are looking for a highly skilled Azure Data Engineer with a strong background in real-time and batch data ingestion and big data processing, particularly using Kafka and Databricks. The ideal candidate will have a deep understanding of streaming architectures, Medallion data models, and performance optimization techniques in cloud environments. This role requires hands-on technical expertise, including live coding during the interview process.

Key Responsibilities


• Design and implement streaming data pipelines integrating Kafka with Databricks using Structured Streaming.


• Architect and maintain Medallion Architecture with well-defined Bronze, Silver, and Gold layers.


• Implement efficient ingestion using Databricks Autoloader for high-throughput data loads.


• Work with large volumes of structured and unstructured data, ensuring high availability and performance.


• Apply performance tuning techniques such as partitioning, caching, and cluster resource optimization.


• Collaborate with cross-functional teams (data scientists, analysts, business users) to build robust data solutions.


• Establish best practices for code versioning, deployment automation, and data governance.

Required Technical Skills:


• Strong expertise in Azure Databricks and Spark Structured Streaming


• 3-8 Years' experience in Data Engineering


• Processing modes (append, update, complete)


• Output modes (append, complete, update)


• Checkpointing and state management


• Experience with Kafka integration for real-time data pipelines


• Deep understanding of Medallion Architecture


• Proficiency with Databricks Autoloader and schema evolution


• Deep understanding of Unity Catalog and Foreign catalog


• Strong knowledge of Spark SQL, Delta Lake, and DataFrames


• Expertise in performance tuning (query optimization, cluster configuration, caching strategies)


• Must have Data management strategies


• Excellent with Governance and Access management


• Strong with Data modelling, Data warehousing concepts, Databricks as a platform


• Solid understanding of Window functions

  • Azure Data Engineer

    2 weeks ago

    Navi Mumbai, Maharashtra, India Celebal Technologies Full time

    Job Title: Azure Data Engineer Duration: Fulltime Positions: Multiple Experience Required: 3-5 Years & 6-8 Years Job Summary: We are looking for a highly skilled Azure Data Engineer with a strong background in real-time and batch data ingestion and big data processing, particularly using Kafka and Databricks. The ideal candidate will have a deep...

  • Azure Data Engineer

    22 hours ago

    Mumbai, Maharashtra, India LTIMindtree Full time

    Skill: Azure Data EngineerPrimary skill: ADB,ADF,SQLSecondary skill: Py spark or PythonExperience: 5-12YRSLocation: Hyderabad, Pune, Mumbai, Kolkata, Nagpur, Bangalore,ChennaiNotice period: Imm to 30 DaysJD: Proactive Q2 Azure Data Factory ADFP3 Job Description Azure Databricks DeveloperLocation India ISTOverview Data Engineer having good experience on Azure...

  • Azure data engineer

    2 weeks ago

    Mumbai, Maharashtra, India M&G plc Full time ₹ 6,00,000 - ₹ 12,00,000 per year

    We are M&G Global Services Private Limited (formerly known as 10FA India Private Limited, and prior to that Prudential Global Services Private Limited). We are a fully owned subsidiary of the M&G plc group of companies, operating as a Global Capability Centre providing a range of value adding services to the Group since 2003. At M&G our purpose is to give...

  • Azure Data Specialist

    14 hours ago

    Mumbai, Maharashtra, India beBeeData Full time ₹ 1,20,00,000 - ₹ 2,50,00,000

    Job Title: Azure Data ArchitectOverview:We are seeking a proactive data architect to design and implement innovative data solutions using Azure Databricks and Python. The primary focus will be on providing cost-effective solutions that optimize resource utilization.Main Responsibilities:Developing data pipelines using Azure Data Factory (ADF) and...

  • Azure Data Solutions Engineer

    14 hours ago

    Mumbai, Maharashtra, India beBeeData Full time ₹ 1,04,000 - ₹ 1,30,878

    Job Title: Data ArchitectWe are seeking a skilled Data Architect to join our team.As a Data Architect, you will be responsible for designing and implementing data solutions that meet the needs of our organization. This includes defining data models, developing data pipelines, and ensuring data quality and integrity.The ideal candidate will have experience...

  • Azure Data and AI Expert

    4 days ago

    Mumbai, Maharashtra, India beBeeDataScientist Full time ₹ 2,00,00,000 - ₹ 2,50,00,000

    Unlocking Expertise in Azure Data and AIWe are seeking a Subject Matter Expert (SME) in Azure Data & AI to drive the technical depth, accuracy, and instructional value of our training content.Key Responsibilities:Define and validate learning content aligned to DP-203, AI-102, and DP-100 certification pathsProvide technical guidance on Azure data pipelines,...

  • Chief Azure Data Architect

    7 days ago

    Mumbai, Maharashtra, India beBeeDataEngineering Full time

    Senior Data Engineer PositionWe are seeking a highly experienced and skilled Senior Data Engineer to join our team. The ideal candidate will have a strong background in Azure technologies, including ADB, ADF, SQL, PySpark, Python, Synapse, Delta Tables, and Unity Catalog.Key Responsibilities:Develop and maintain large-scale data engineering solutions using...

  • Azure Data Solutions Developer

    4 days ago

    Mumbai, Maharashtra, India beBeeDataEngineer Full time ₹ 8,00,000 - ₹ 15,00,000

    Job Title:Azure Data EngineerAbout the Role:This role involves working with data platforms to design, develop, and implement data solutions using Azure Databricks.The ideal candidate will have experience with data engineering, data warehousing, and data analytics using Python and SQL.We are looking for someone who is passionate about data and has a strong...

  • Azure Data Solutions Specialist

    2 days ago

    Mumbai, Maharashtra, India beBeeDatabase Full time ₹ 19,32,000 - ₹ 25,15,000

    Job Title:Azure Database Engineer RoleWe are seeking a skilled professional to join our team as an Azure Database Engineer.The role requires proficiency with Azure PostgreSQL, SQL Server, and Application DBA tasks for robust database management and optimization.The engineer will design, build, develop, and manage pipelines using Database Technologies,...

  • AI & Data Architect - Azure

    4 weeks ago

    Mumbai, Maharashtra, India CloudFronts - AI First Microsoft Solutions Partner Full time

    AI & Data Architect - Azure Job Summary: AI & Data Architect will lead all aspects of AI, Business Intelligence, and Data Architecture with expertise in Azure services, including AI/ML solutions, data engineering, and analytics. The role involves designing, configuring, and managing BI services, aggregating data from multiple sources into a data warehouse,...