Data Engineering Azure databricks

1 day ago

India EXL Full time

Data Engineer (DE) Consultant is responsible for designing, developing, and maintaining data assets and data related products by liaising with multiple stakeholders.

Responsibilities:

  • Work with stakeholders to understand the data requirements to design, develop, and maintain complex ETL processes.
  • Create the data integration and data diagram documentation.
  • Lead the data validation, UAT and regression test for new data asset creation.
  • Create and maintain data models, including schema design and optimization.
  • Create and manage data pipelines that automate the flow of data, ensuring data quality and consistency.

Qualifications and Skills:

  • Strong knowledge on Python and Pyspark
  • Expectation is to have ability to write Pyspark scripts for developing data workflows.
  • Strong knowledge on SQL, Hadoop, Hive, Azure, Databricks and Greenplum
  • Expectation is to write SQL to query metadata and tables from different data management system such as, Oracle, Hive, Databricks and Greenplum.
  • Familiarity with big data technologies like Hadoop, Spark, and distributed computing frameworks.
  • Expectation is to use Hue and run Hive SQL queries, schedule Apache Oozie jobs to automate the data workflows.
  • Good working experience of communicating with the stakeholders and collaborate effectively with the business team for data testing.
  • Expectation is to have strong problem-solving and troubleshooting skills.
  • Expectation is to establish comprehensive data quality test cases, procedures and implement automated data validation processes.
  • Degree in Data Science, Statistics, Computer Science or other related fields or an equivalent combination of education and experience.
  • 4-7 years of experience in Data Engineer.
  • Proficiency in programming languages commonly used in data engineering, such as Python, Pyspark, SQL.
  • Experience in Azure cloud computing platform, such as developing ETL processes using Azure Data Factory, big data processing and analytics with Azure Databricks.
  • Strong communication, problem solving and analytical skills with the ability to do time management and multi-tasking with attention to detail and accuracy.

  • Data Engineering Azure databricks

    3 days ago

    India EXL Full time

    Data Engineer (DE) Consultant is responsible for designing, developing, and maintaining data assets and data related products by liaising with multiple stakeholders.Responsibilities:Work with stakeholders to understand the data requirements to design, develop, and maintain complex ETL processes.Create the data integration and data diagram documentation.Lead...

  • Data Engineering Azure databricks

    9 hours ago

    India EXL Full time

    Data Engineer (DE) Consultant is responsible for designing, developing, and maintaining data assets and data related products by liaising with multiple stakeholders. Responsibilities: Work with stakeholders to understand the data requirements to design, develop, and maintain complex ETL processes. Create the data integration and data diagram documentation....

  • Azure Databricks Developer

    2 weeks ago

    India Adastra Corp Full time

    Role & responsibilities As an Azure Databricks Developer with Terraform, your responsibilities include: Designing and optimizing data pipelines using Azure Databricks (Spark, Delta Lake, notebooks, jobs) Automating infrastructure provisioning on Azure through Terraform Collaborating with data engineers, analysts, and cloud architects to integrate Databricks...

  • Data Engineer – Databricks

    4 weeks ago

    India Epergne Solutions Full time

    Project Overview We are seeking skilled Data Engineers with strong hands-on experience in Databricks for a high-impact data platform project The project involves building scalable data pipelines integrating with cloud platforms and managing data governance using Unity Catalog Key Requirements Primary Skills Expertise in Databricks working on...

  • Databricks Data Engineer

    2 weeks ago

    India Adastra Corp Full time

    Job Description As a Databricks Data Engineer, you will be responsible for: Technical Requirements Gathering and Development of Functional Specifications. Design, develop, and maintain scalable data pipelines and ETL processes using Azure Databricks, Data Factory, and other Azure services. Implement and optimize Spark jobs, data transformations, and data...

  • Databricks Data Engineer

    3 days ago

    India Adastra Corp Full time ₹ 12,00,000 - ₹ 36,00,000 per year

    Job DescriptionAs a Databricks Data Engineer, you will be responsible for:Technical Requirements Gathering and Development of Functional Specifications.Design, develop, and maintain scalable data pipelines and ETL processes using Azure Databricks, Data Factory, and other Azure services.Implement and optimize Spark jobs, data transformations, and data...

  • Azure Data Engineer

    10 hours ago

    Hyderabad, India Fragma Data Systems Full time

    Job Description Must-Have Skills - Good experience in Pyspark - Including Dataframe core functions and Spark SQL - Good experience in SQL DBs - Be able to write queries including fair complexity. - Should have excellent experience in Big Data programming for data transformation and aggregations - Good at ELT architecture. Business rules processing and data...

  • Engineering Manager

    1 week ago

    India DataE2E Technologies Full time

    Job Title: Engineering Manager – Big Data, Azure & Databricks Location: Remote Employment Type: Full-time Department: Data Engineering / Technology Experience: 8+ years (3+ years in a managerial role) Salary - 30 LPA About the Role: We are seeking a highly experienced and strategic Engineering Manager with deep expertise in ...

  • Engineering Manager – Big Data, Azure

    1 week ago

    India DataE2E Technologies Full time

    Job Title: Engineering Manager – Big Data, Azure & Databricks Location: Remote Employment Type: Full-time Department: Data Engineering / Technology Experience: 8+ years (3+ years in a managerial role) Salary - 30 LPA About the Role: We are seeking a highly experienced and strategic Engineering Manager with deep expertise in Big Data platforms,...

  • Azure data engineer

    1 week ago

    India RapidBrains Full time

    Position – Senior Azure Data Engineer Location: PAN India Employment Type: Full-time Notice Period - Immediate Joiners Only About the Role We are seeking an experienced Senior Data Engineer to join our IT division and play a pivotal role in supporting the company’s global operations across diverse sectors. The ideal candidate will have strong...