Azure Data Engineer

2 weeks ago

Bengaluru Chennai Hyderabad, India Randomtrees Full time US$ 1,20,000 - US$ 1,50,000 per year

Position : Senior Data Engineer Databricks Expert

We are seeking a Senior Data Engineer with in-depth knowledge of Databricks and Unity Catalog to serve as the subject matter expert for all things Databricks within our organization. This role requires deep expertise in Databricks, including CI/CD setup for data bricks, data lineage through Unity Catalog, and strong proficiency in ETL, SQL, and modern data engineering practices. You will be the go-to person for designing, implementing, and optimizing data solutions with Databricks.

Key Responsibilities:

  • Serve as the point of contact and subject matter expert for all Databricks-related activities, including architecture, development, and operational best practices.
  • Design, develop, and manage ETL/ELT pipelines in Databricks using Python (PySpark), integrating various data sources to support business operations.
  • Leverage Unity Catalog to ensure data lineage, security, and governance are properly managed across the Databricks environment.
  • Implement and maintain CI/CD pipelines for Databricks, ensuring smooth deployments, version control, and automation using Git and other DevOps tools.
  • Build scalable data architectures, including Data Lakes, Lakehouses, and Data Warehouses, ensuring efficient data management and accessibility.
  • Configure and optimize Databricks clusters, jobs, and workflows for both batch and streaming data processing to handle large-scale datasets.
  • Stay up to date with the latest Databricks features and advancements, continuously enhancing our data engineering practices.
  • Collaborate with cross-functional teams to implement data governance and ensure compliance with security and industry regulations.
  • Monitor and tune Databricks workloads to ensure high performance and scalability, adapting to business needs as required.
  • Provide guidance and mentorship to junior engineers, ensuring adherence to best practices and fostering a collaborative environment.
  • Maintain technical documentation and provide training to internal stakeholders on Databricks tools and processes.

Qualifications:

  • 5+ years of experience in data engineering with significant expertise in Databricks and Apache Spark.

    • Proficient in Unity Catalog for managing data lineage, security, and governance within the Databricks ecosystem.

    • Experience building and optimizing ETL pipelines using tools like Azure Data Factory, Informatica, or similar.

    • Strong understanding of CI/CD practices with experience in Git for version control and integration with Databricks.

    • Expertise in SQL development and performance tuning for large-scale datasets.

    • Knowledge of the Azure ecosystem, including data services like Azure Data Lake and Azure Storage.

    • Ability to work with both batch and streaming data processing pipelines.

    • Experience with data modelling and dimensional design (e.g., star schema).

    • Good understanding of data governance, compliance, and security best practices.

    • Excellent communication and problem-solving skills, with the ability to manage multiple priorities.

    • Ability to stay current on Databricks innovations and proactively introduce new features and capabilities to the team.Role & responsibilities

Preferred candidate profile

  • Azure Data Engineer

    22 hours ago

    Hyderabad, Telangana, India Enable Data Full time ₹ 15,00,000 - ₹ 25,00,000 per year

    Enable Data Incorporated is currently seeking a skilled and experienced Azure Data Engineer to join our dynamic team. As a leading provider of advanced application, data, and cloud engineering services, Enable Data has developed deep expertise across various industries. We work closely with our customers to leverage modern solutions and technologies to drive...

  • Azure Data Engineer

    6 days ago

    Hyderabad, India Enable Data Incorporated Full time

    Enable Data Incorporated is currently seeking a skilled and experienced Azure Data Engineer to join our dynamic team. As a leading provider of advanced application, data, and cloud engineering services, Enable Data has developed deep expertise across various industries. We work closely with our customers to leverage modern solutions and technologies to drive...

  • Azure Data Engineer With Snowflake

    1 day ago

    Hyderabad, Telangana, India Enable Data Full time ₹ 15,00,000 - ₹ 25,00,000 per year

    Enable Data Incorporated is currently seeking a skilled and experienced Azure Data Engineer to join our dynamic team. As a leading provider of advanced application, data, and cloud engineering services, Enable Data has developed deep expertise across various industries. We work closely with our customers to leverage modern solutions and technologies to drive...

  • Azure Data Engineer With Snowflake

    2 days ago

    Hyderabad, India Enable Data Incorporated Full time

    Enable Data Incorporated is currently seeking a skilled and experienced Azure Data Engineer to join our dynamic team. As a leading provider of advanced application, data, and cloud engineering services, Enable Data has developed deep expertise across various industries. We work closely with our customers to leverage modern solutions and technologies to drive...

  • Azure Data Engineer With Snowflake

    1 day ago

    Hyderabad, Telangana, India Enable Data Incorporated Full time ₹ 20,00,000 - ₹ 25,00,000 per year

    Enable Data Incorporated is currently seeking a skilled and experienced Azure Data Engineer to join our dynamic team. As a leading provider of advanced application, data, and cloud engineering services, Enable Data has developed deep expertise across various industries. We work closely with our customers to leverage modern solutions and technologies to drive...

  • Data Engineer for Azure DevOps

    6 days ago

    Chennai, India NTT DATA Full time

    Req ID: NTT DATA strives to hire exceptional, innovative and passionate individuals who want to grow with us. If you want to be part of an inclusive, adaptable, and forward-thinking organization, apply now. We are currently seeking a Data Engineer for Azure DevOps to join our team in Chennai, Tamil Nādu (IN-TN), India (IN). "Job Duties: 7+ years of...

  • Azure Cloud Engineer

    23 hours ago

    Hyderabad, Telangana, India NTT DATA Full time ₹ 10,00,000 - ₹ 30,00,000 per year

    NTT DATA strives to hire exceptional, innovative and passionate individuals who want to grow with us. If you want to be part of an inclusive, adaptable, and forward-thinking organization, apply now. We are currently seeking a Azure Cloud Engineer to join our team in Hyderabad, Telangana (IN-TG), India (IN). Role: - Azure Cloud Engineer - PaaS, SaaS,...

  • Azure Data Architect

    1 week ago

    Bengaluru, India NTT Data Full time

    Job Description NTT DATA strives to hire exceptional, innovative and passionate individuals who want to grow with us. If you want to be part of an inclusive, adaptable, and forward-thinking organization, apply now. We are currently seeking a Azure Data Architect to join our team in Bangalore, Karntaka (IN-KA), India (IN). Azure - Architect Job...

  • Azure Data Engineer

    2 weeks ago

    Hyderabad, Telangana, India Apex One Full time ₹ 15,00,000 - ₹ 28,00,000 per year

    Job SummaryWe are seeking a skilled Azure Data Engineer with 4 years of overall experience, including at least 2 years of hands-on experience with Azure Databricks (Must). The ideal candidate will have strong expertise in building and maintaining scalable data pipelines and working across cloud-based data platforms. Key ResponsibilitiesDesign, develop, and...

  • Azure Data Engineer

    6 days ago

    Hyderabad, India Apex One Full time

    Job SummaryWe are seeking a skilled Azure Data Engineer with 4 years of overall experience, including at least 2 years of hands-on experience with Azure Databricks (Must). The ideal candidate will have strong expertise in building and maintaining scalable data pipelines and working across cloud-based data platforms. Key Responsibilities Design, develop,...