Data Engineering Azure databricks

10 hours ago

india, IN EXL Full time

Data Engineer (DE) Consultant is responsible for designing, developing, and maintaining data assets and data related products by liaising with multiple stakeholders.

Responsibilities:

  • Work with stakeholders to understand the data requirements to design, develop, and maintain complex ETL processes.
  • Create the data integration and data diagram documentation.
  • Lead the data validation, UAT and regression test for new data asset creation.
  • Create and maintain data models, including schema design and optimization.
  • Create and manage data pipelines that automate the flow of data, ensuring data quality and consistency.

Qualifications and Skills:

  • Strong knowledge on Python and Pyspark
  • Expectation is to have ability to write Pyspark scripts for developing data workflows.
  • Strong knowledge on SQL, Hadoop, Hive, Azure, Databricks and Greenplum
  • Expectation is to write SQL to query metadata and tables from different data management system such as, Oracle, Hive, Databricks and Greenplum.
  • Familiarity with big data technologies like Hadoop, Spark, and distributed computing frameworks.
  • Expectation is to use Hue and run Hive SQL queries, schedule Apache Oozie jobs to automate the data workflows.
  • Good working experience of communicating with the stakeholders and collaborate effectively with the business team for data testing.
  • Expectation is to have strong problem-solving and troubleshooting skills.
  • Expectation is to establish comprehensive data quality test cases, procedures and implement automated data validation processes.
  • Degree in Data Science, Statistics, Computer Science or other related fields or an equivalent combination of education and experience.
  • 4-7 years of experience in Data Engineer.
  • Proficiency in programming languages commonly used in data engineering, such as Python, Pyspark, SQL.
  • Experience in Azure cloud computing platform, such as developing ETL processes using Azure Data Factory, big data processing and analytics with Azure Databricks.
  • Strong communication, problem solving and analytical skills with the ability to do time management and multi-tasking with attention to detail and accuracy.

  • Engineering Manager – Big Data

    10 hours ago

    india, IN DataE2E Technologies Full time

    Job Title: Engineering Manager – Big Data, Azure & DatabricksLocation: RemoteEmployment Type: Full-timeDepartment: Data Engineering / TechnologyExperience: 8+ years (3+ years in a managerial role)Salary - 30 LPAAbout the Role:We are seeking a highly experienced and strategic Engineering Manager with deep expertise in Big Data platforms, Microsoft Azure,...

  • Azure data Engineer

    10 hours ago

    Bhubaneswar, Odisha, India, IN Tata Consultancy Services Full time

    Role: Azure data EngineerExperience: 4Location: BhubaneswarMust-Have4+ years of hands-on experience in development.Azure – cloud technology – Function Apps, API, Azure DevOps, Databricks, Data Factory, SynapseStrong knowledge Incident, problem, Change management, service management, service request.Must have hands on experience code versioning tool...

  • Data Engineer

    10 hours ago

    india, IN Vriba Solutions Full time

    Role: Data Engineering Remote Looking for 5-10 years of Exp• Design, develop & maintain ETL/ELT pipelines• Ingest & transform data from APIs, DBs, files, streams• Build real-time & batch processing solutions• Data validation, quality & cleansing• Translate business needs into data models• Ensure data security, access control & compliance•...

  • Senior Data Engineer

    10 hours ago

    Bangalore Urban, Karnataka, India, IN USEReady Full time

    Job Title: Senior Databricks EngineerExperience Level: 5-8 YearsJob SummaryAs a Senior Databricks Engineer, you will be responsible for designing, developing, and optimizing our data architecture and pipelines on the Databricks Lakehouse Platform. You will leverage your deep expertise in Spark, Delta Lake, and cloud technologies to build scalable and...

  • Data Engineer

    10 hours ago

    Bengaluru North, Karnataka, India, IN CentricaSoft Full time

    Title: Data Engineer - PySpark About CentricaSoft: CentricaSoft is a data-driven technology partner delivering end-to-end data solutions for our clients. We design, build, and scale modern data platforms that empower business decisions. We’re growing our data engineering team and seeking a hands-on PySpark Developer who thrives in a fast-paced,...

  • Cloud Data Engineer

    10 hours ago

    india, IN Lemongrass Full time

    About LemongrassLemongrass is a software-enabled services provider, synonymous with SAP on Cloud, focused on delivering superior, highly automated Managed Services to Enterprise customers. Our customers span multiple verticals and geographies across the Americas, EMEA and APAC. We partner with AWS, SAP, Microsoft and other global technology leaders.We are...

  • Data Engineer

    10 hours ago

    india, IN Data-Hat AI Full time

    Department: Data Engineering & AI Solutions Reports To: Lead Data Solutions Architect Travel: International travel required (up to 30–40%) Position Summary: We are hiring a senior-level Data Engineer to lead the design, development, and optimization of high-performance data infrastructure that underpins mission-critical AI systems. With 12+ years of...

  • Senior Data Engineer

    10 hours ago

    india, IN Sovereign IT Solutions Pvt Ltd Full time

    We are seeking a Senior Data Engineer with 8+ years of real-world experience to act as a Subject Matter Expert (SME) and train students specifically for Data Engineering, Big Data, and AI/ML Data Platform interviews.The ideal candidate will not only be hands-on with the latest data + AI/ML technologies but also excel at coaching, mentoring, and conducting...

  • Data Scientist

    10 hours ago

    india, IN vueverse. Full time

    About the RoleWe are looking for an experienced Data Scientist with expertise in Machine Learning, Generative AI, and Large Language Models (LLMs) to design, build, and deploy AI-driven solutions using the Azure ecosystem.ResponsibilitiesBuild end-to-end ML pipelines (data prep, training, validation, deployment).Apply LLMs & GenAI for NLP, predictive...

  • Senior Data Engineer

    10 hours ago

    india, IN Infogain Full time

    Job Summary:Big Data Engineer (Lead): As a Big Data Engineer (Lead), you will be responsible for leading a team of big data engineers. You will work closely with clients and team members to understand their requirements and develop architectures that meet their needs. You will also be responsible for providing technical leadership and guidance to your...