Lead/Azure Data Engineer

3 weeks ago

Bengaluru, Karnataka, India On Time Solutions Full time

Designation : Lead Azure Data Engineer

Experience : 5+

Location : Bangalore/Hyderabad

Skills : Azure Data Engiineer, ADF, ADB, ADLS, Python, Pypark

Key technical skills :

- Hands-on experience in designing and building scalable data pipelines using Databricks with PySpark, supporting batch and near-real-time ingestion, transformation, and processing.

- Ability to optimize Spark jobs and manage large-scale data processing using RDD/DataFrameAPIs.

- Demonstrated expertise in partitioning strategies, file format optimization (Parquet/Delta), and Spark SQL tuning.

- Skilled in governing and manage data access for Azure Data lakehouse with Unity Catalog. Experience in configuring data permissions, object lineage, and access policies with Unity Catalog.

- Understanding of integrating Unity Catalog with Azure AD, external metastores, and audit trails.

- Experience in building efficient orchestration solutions using Azure data factory, Databricks Workflows.

- Ability to design modular, reusable workflows using tasks, triggers, and dependencies.

- Skilled in using dynamic expressions, parameterized pipelines, custom activities, and triggers.

- Familiarity with integration runtime configurations, pipeline performance tuning, and error handling strategies.

- Good experience in implementing secure, hierarchical namespace-based data lake storage for structured/semi-structured data, aligned to bronze-silver-gold layers with ADLS Gen2.

- Hands-on experience with lifecycle policies, access control (RBAC/ACLs), and folder-level security.

- Understanding of best practices in file partitioning, retention management, and storage performance optimization.

- Capable of developing T-SQL queries, stored procedures, and managing metadata layers on Azure SQL

- Comprehensive experience working across the Azure Cloud ecosystem, including networking, security, monitoring, and cost management relevant to data engineering workloads.

- Experience in writing modular, testable Python code used in data transformations, utility functions, and packaging reusable components.

- Familiarity with Python environments, dependency management (pip/Poetry/Conda), and packaging libraries.

- Ability to write unit tests using PyTest/unittest and integrate with CI/CD pipelines.

- Able to prepare Design documents for developmentadopting code analyzers and unit testing frameworks.

- Write clear and concise documentation leveraging industry standard practices & guidelines.

(ref:hirist.tech)

  • Azure Data Engineering Lead

    1 day ago

    Bengaluru, Karnataka, India beBeeDataEngineering Full time ₹ 1,04,000 - ₹ 1,30,878

    Senior Data Engineer - Azure ExpertThe ideal candidate will have hands-on experience with Microsoft Azure cloud services, data engineering, and a strong background in designing and implementing scalable data solutions. Key tasks & accountabilities include:Design, develop, and maintain scalable data pipelines and workflows using Azure Data Factory, Azure...

  • Azure Data Engineer

    10 hours ago

    Bengaluru, Karnataka, India Infogain Full time

    Posted on October 24 2024ROLES RESPONSIBILITIESCore Skills- Mastery of Azure data services such as Azure SQL Database Azure Cosmos DB and Azure Data Lake Storage - Deep expertise in ETL Extract Transform Load processes and data integration - Advanced SQL and database querying optimization skills - Experience in leading and managing data...

  • Azure Data Engineer

    4 days ago

    Bengaluru, Karnataka, India Talent Worx Full time ₹ 9,00,000 - ₹ 12,00,000 per year

    Global consulting firm seeking an Azure Data Engineer to join our dynamic team. As an Azure Data Engineer, you will be responsible for designing and implementing robust data pipelines and workflows using Azure cloud technologies. The ideal candidate will possess a deep understanding of data engineering principles and hands-on experience with Azure data...

  • Azure Data Engineer

    4 days ago

    Bengaluru, Karnataka, India Talent Worx Full time ₹ 15,00,000 - ₹ 20,00,000 per year

    We are looking for an experienced Azure Data Engineer - Senior Consultant .This role requires expertise in designing, implementing, and managing data solutions on Microsoft Azure. The ideal candidate will be well-versed in data architecture, ETL processes, and the Azure ecosystem.Experience: 7 to 8 years Location: Bangalore RequirementsKey...

  • Azure Data Engineer

    3 weeks ago

    Bengaluru, Karnataka, India Talent Worx Full time

    We are looking for an experienced Azure Data Engineer Senior Consultant . This role requires expertise in designing, implementing, and managing data solutions on Microsoft Azure. The ideal candidate will be well-versed in data architecture, ETL processes, and the Azure ecosystem. Experience : 7 to 8 years. Location : Bangalore.Requirements : Key...

  • Lead Azure Data Engineering Specialist

    3 days ago

    Bengaluru, Karnataka, India beBeeDataEngineering Full time ₹ 15,00,000 - ₹ 20,00,000

    About the RoleWe're seeking a seasoned data engineering expert with a strong background in data engineering, ELT, data modeling, and integration. The ideal candidate will have extensive experience working with Azure cloud services, modern data platforms, and Big Data technologies, along with expertise in DevOps practices using GitHub and Azure DevOps.Key...

  • Azure Data Engineer

    3 weeks ago

    Bengaluru, Karnataka, India GALAXY I TECHNOLOGIES INC Full time

    Role Description :We are seeking a highly skilled Azure Data Engineer to join our team. The ideal candidate will be responsible for designing, building, and maintaining robust and scalable data pipelines on the Microsoft Azure platform. This role involves integrating, transforming, and consolidating data from various sources to enable business intelligence...

  • Azure Data Engineer

    4 weeks ago

    Bengaluru, Karnataka, India Teamware Solutions Full time

    As a Sr. Azure Cloud Data Engineer, you will play a crucial role in Trane Technologies' organization, in processing and analysing IoT data from our connected products to drive value for our customers. You will be responsible for delivering analytical data solutions for both internal and external customers. The Cloud Data Engineer plays an active role in...

  • Data Engineering Lead

    10 hours ago

    Bengaluru, Karnataka, India beBeeAzure Full time ₹ 15,00,000 - ₹ 25,00,000

    Data Engineering LeadWe are seeking a seasoned professional with expertise in Azure Data Factory and ETL processes to lead our data engineering efforts.Responsibilities:Design, develop, and implement scalable data pipelines using Azure Data Factory.Create efficient ETL/ELT processes leveraging SSIS and/or Azure Data Factory.Collaborate with cross-functional...

  • Azure Data Engineer

    3 days ago

    Bengaluru, Karnataka, India Capgemini Full time US$ 90,000 - US$ 1,20,000 per year

    Choosing Capgemini means choosing a company where you will be empowered to shape your career in the way you'd like, where you'll be supported and inspired by a collaborative community of colleagues around the world, and where you'll be able to reimagine what's possible. Join us and help the world's leading organizations unlock the value of technology and...