Natobotics - Senior Data Engineer - Azure Databricks

23 hours ago

Anywhere in IndiaMultiple Locations Natobotics Technologies Pvt Limited Full time ₹ 20,00,000 - ₹ 25,00,000 per year

Description : Senior Data Engineer - Azure ETL/ELT & PySpark


Location : Mumbai, MH / Hyderabad, TG / Chennai, TN / Bhubaneswar, OR / Bangalore, KA


Experience : 6-8 Years

About the Role :

We are seeking an experienced Senior Data Engineer with 6-8 years in data engineering to join our team across multiple locations. This role is focused on designing, developing, and maintaining scalable and fault-tolerant data pipelines using the Microsoft Azure ecosystem. The ideal candidate will have deep expertise in Azure Data Factory, Databricks, PySpark, and advanced SQL to drive complex data transformations and implement robust CI/CD practices.

Responsibilities :

- ETL/ELT Pipeline Development : Design, develop, and maintain complex ETL/ELT pipelines for large-scale data ingestion and transformation using Azure Data Factory (ADF) and Azure Databricks.

- Data Ingestion Flows : Implement robust data ingestion flows to integrate data from diverse sources, including Azure Blob Storage, Azure Data Lake (ADLS), On-Prem SQL, and SFTP, ensuring reliability and security.

- Data Transformation & Optimization : Design and optimize dimensional and relational data models, and perform complex data transformations utilizing Spark SQL, PySpark, Oracle, SQL Server, and Progress DB SQL.

- Orchestration Workflow : Build sophisticated orchestration workflows in ADF using a range of activities such as Lookup, ForEach, Execute Pipeline, and Set Variable, to manage complex, multi-stage data jobs.

- Troubleshooting and Maintenance : Perform root cause analysis and quickly resolve critical production issues affecting data pipelines, Databricks notebooks, and data flow stability.

- CICD and DevOps : Collaborate with DevOps teams on the creation and enhancement of CI/CD pipelines using tools like Azure DevOps and Jenkins to enable automated testing and deployment of data assets.

- Performance Tuning : Apply specialized performance tuning techniques to optimize query execution and resource utilization within Azure Synapse Analytics (SQL DW) environments.

- Documentation and Quality : Maintain comprehensive technical documentation, including detailed runbooks, technical design specifications, and QA test cases, ensuring knowledge transfer and operational efficiency.

Required Skill Set (Mandatory Skills) :

- Cloud Platforms : 6-8 years of hands-on experience utilizing key Microsoft Azure services for data engineering, including Azure Data Factory (ADF), Azure Databricks (ADB), Azure Data Lake Storage (ADLS), Azure SQL, and Azure Synapse Analytics.

- Programming & Big Data : Expert proficiency in Python for Data Science, and deep experience with PySpark and Spark SQL for large-scale distributed data processing.

- ETL Tools : Mandatory expertise in building and maintaining enterprise-level data pipelines using Azure Data Factory.

- Database & SQL : Expert-level proficiency in SQL for complex querying, performance tuning, and data model design.

Preferred Skills :

- Practical experience applying performance tuning specifically to Azure Synapse Analytics (SQL DW).

- Experience with Azure DevOps, Git, and CICD pipeline creation using either Azure DevOps or Jenkins.

- Familiarity with non-relational databases such as Cosmos DB or other NoSQL platforms.

- Experience ingesting data from older enterprise systems like DB2 or diverse formats like NoSQL sources.

  • Senior Data Engineer

    3 days ago

    Anywhere in India/Multiple Locations Timus consulting Services Full time ₹ 20,00,000 - ₹ 25,00,000 per year

    Description : About the Role : We are seeking an experienced and highly skilled Senior Data Engineer Azure Databricks to join our data engineering team. In this role, you will be responsible for designing, building, and maintaining scalable data pipelines and solutions on Azure using Databricks, with a strong focus on performance, reliability, and...

  • Azure Data Engineer/Data Engineer

    6 days ago

    Anywhere in India/Multiple Locations TESTQ Technologies Limited Full time ₹ 20,00,000 - ₹ 25,00,000 per year

    About the Opportunity : We are looking for a skilled Azure Data Engineer / Data Engineer with hands-on experience in Azure Data Factory, Snowflake, Databricks, and DBT to architect and implement large-scale data integration and transformation pipelines. The ideal candidate will bring deep technical expertise in ETL/ELT design, data modeling, and big...

  • Azure Data Engineer

    6 days ago

    Anywhere in India/Multiple Locations Digivance Solution Full time ₹ 15,00,000 - ₹ 25,00,000 per year

    Description :Position : Azure Data EngineerLocation : India (Remote / Hybrid)Experience : 7 Years (5-7 years relevant)Mode of Interview : Video InterviewBackground Verification : After onboardingRoles & Responsibilities : - Work in an Agile development environment to design, build, and optimize data solutions. - Develop and manage scalable ETL pipelines...

  • Azure Data Engineer

    6 days ago

    Multiple Locations, India CROSSDEV TECHNOLOGIES PRIVATE LIMITED Full time ₹ 10,00,000 - ₹ 25,00,000 per year

    Description :Job Summary : We are looking for an experienced Data Engineer with around 3 years of hands-on experience, strong expertise in Azure Databricks and Azure Database services. The ideal candidate will be responsible for building and maintaining scalable data pipelines, ensuring data quality, and supporting analytical and operational use cases....

  • Azure Data Engineer

    1 week ago

    Anywhere in India/Multiple Locations AATRAL HR CONSULTING LLP Full time ₹ 15,00,000 - ₹ 25,00,000 per year

    Description : We are looking for a skilled Azure Data Engineer to design, build, and maintain scalable data pipelines and architectures in Microsoft Azure. You will play a crucial role in enabling data-driven decision-making by transforming raw data into accessible, clean, and reliable datasets. If you have hands-on experience with Azure data...

  • Azure DevOps Engineer

    3 days ago

    Anywhere in India/Multiple Locations Digihelic Solutions Private Limited Full time ₹ 20,00,000 - ₹ 25,00,000 per year

    Description : Job description.Title : Azure DevOps Engineer Databricks.Experience : 6 Years.Location : Trivandrum and Bangalore.Notice Period : Immediate. We are seeking a skilled Cloud Engineer to ensure the stability, performance, and scalability of our Data Transformation Platform hosted on Microsoft Azure. This role combines operational ownership...

  • Data Engineer

    5 days ago

    Anywhere in India/Multiple Locations Vikgol Full time ₹ 15,00,000 - ₹ 25,00,000 per year

    Job Title : Remote Data Engineer - Azure Databricks Location : Remote Job Type : Full-time Hiring 3 Data Engineers (Remote) Vikgol is looking for experienced Data Engineers to design, build, and maintain our cloud data infrastructure. This is a 100% remote, full-time position where you will work with cutting-edge technologies like Azure...

  • Databricks Developer

    1 week ago

    Anywhere in India/Multiple Locations Jigya Software Services (P) Ltd Full time ₹ 15,00,000 - ₹ 25,00,000 per year

    Description : Job Title : Senior Databricks Developer. Location : Remote. About Us : The Role : We are looking for a passionate Databricks Developer to architect and implement our next-generation data platform. You will be instrumental in building robust, scalable data pipelines, optimizing our cloud data warehouse, and creating powerful...

  • Data Engineer

    3 days ago

    Anywhere in India/Multiple Locations Codesmith Full time ₹ 12,00,000 - ₹ 36,00,000 per year

    Description : We are seeking a strong Data Engineer with advanced expertise in Databricks and PySpark. The successful candidate will be a key contributor to critical projects, including migrating Palantir data transformation pipelines to Databricks Notebooks, designing and implementing incremental data pipelines, and orchestrating workflows in Azure...

  • Azure Data Engineer

    7 days ago

    Multiple Locations, India AVENSYS CONSULTING PRIVATE LIMITED Full time ₹ 15,00,000 - ₹ 25,00,000 per year

    Description : Avensys is a reputed global IT professional services company headquartered in Singapore. Our service spectrum includes enterprise solution consulting, business intelligence, business process automation and managed services. Given our decade of success we have evolved to become one of the top trusted providers in Singapore and service a client...