Senior Data Pipeline Engineer

15 hours ago

Mumbai, Maharashtra, India beBeeDataEngineer Full time ₹ 18,00,000 - ₹ 20,00,000

Job Overview

We are seeking a highly skilled professional to build and maintain secure, scalable data pipelines using Databricks and Azure.

This role requires expertise in handling ingestion from diverse sources (files, APIs, streaming), data transformation, and quality validation.

The ideal candidate will collaborate with subsystem data science and product teams for ML experiences.

Key Responsibilities

  • Data Ingestion: Develop efficient data ingestion processes from various sources, including files, APIs, and streaming platforms.
  • Data Transformation: Design and implement data transformation workflows using Notebooks (SQL, Python) and Delta Lake.
  • Data Quality: Establish robust data quality validation processes to ensure accuracy and reliability.
  • Collaboration: Work closely with data science and product teams to develop and deploy machine learning models.

Required Skills

  • Notebooks and Databases: Proficient in Notebooks (SQL, Python), Delta Lake, Unity Catalog, ADLS/S3, job orchestration, APIs, structured logging, and IaC (Terraform).
  • Development Methodologies: Familiarity with Trunk-based development, TDD, Git, CI/CD for notebooks and pipelines.
  • Data Formats: Knowledge of JSON, CSV, XML, Parquet, SQL/NoSQL/graph databases.

What You Get

  • Opportunity: Opportunity to work with a world-class team.
  • Learning: Chance to learn something new every day.
  • Culture: Collaborative environment with designers, engineers, and product geeks.

  • Senior Data Pipeline Engineer

    3 hours ago

    Mumbai, Maharashtra, India beBeeDataScientist Full time ₹ 18,00,000 - ₹ 25,00,000

    Key ResponsibilitiesOur team is seeking a skilled Python Developer to design and implement automated data pipelines using Google Cloud Platform. The ideal candidate will have experience in developing analytical algorithms to drive business insights, creating REST APIs to facilitate data exchange, and working with SQL databases and Neo4j graph databases to...

  • Lead Data Pipeline Specialist

    9 hours ago

    Mumbai, Maharashtra, India beBeeData Full time ₹ 1,20,00,000 - ₹ 2,40,00,000

    Senior Data EngineerWe are seeking a Senior Data Engineer to join our team. As a member of this team, you will be responsible for building data pipelines to ingest various source data from enterprise technologies and PitchBook Platform data.You will work with a range of data and reporting technologies (e.g. Python, Docker, Tableau, Power BI) to build upon a...

  • Data Pipeline Migration Specialist

    3 days ago

    Mumbai, Maharashtra, India beBeeDataGovernance Full time ₹ 1,75,00,000 - ₹ 2,50,00,000

    Technical Leadership OpportunityThis is a senior-level position that oversees the technical aspects of implementing data pipeline migration requirements, ensuring our platform is used to its full potential through designing and building applications around business stakeholder needs.Main ResponsibilitiesCollaborate directly with stakeholders to gather...

  • Senior Data Pipeline Architect

    2 days ago

    Mumbai, Maharashtra, India beBeeDataEngineer Full time ₹ 20,00,000 - ₹ 25,00,000

    Transforming marketing data into actionable insights requires a unique blend of technical expertise and analytical thinking. As a key member of our data infrastructure team, you will design, develop, and maintain robust data pipelines, warehouses, and analytics platforms.Design and implement scalable data pipelines for marketing data ingestion and...

  • Senior Data Engineer

    4 days ago

    Mumbai, Maharashtra, India HealthyMe Ai Full time

    About the RoleWe are seeking a Senior Data Engineer with deep expertise in building data pipelines and integrating healthcare data. You will report to the Head of AI and Engineering and be responsible for designing, building, and operating the backbone of our AI-powered applications. This is a critical role where you will create scalable pipelines that...

  • Data Pipeline Specialist

    5 days ago

    Mumbai, Maharashtra, India beBeeDataEngineer Full time US$ 1,00,000 - US$ 1,50,000

    Data Engineer OpportunityWe are seeking a skilled Data Engineer to design and develop scalable data pipelines.This role involves working with streaming data, collaborating with data scientists, and delivering high-quality solutions.

  • Senior Big Data Engineer

    7 hours ago

    Mumbai, Maharashtra, India beBeeBigDataEngineer Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

    Job Title: Senior Big Data EngineerJob Summary:Data Engineers design, develop and maintain large-scale data processing systems. In this role, you will work with our team to build scalable data pipelines using PySpark and ensure the smooth operation of our data infrastructure.About the Role:The ideal candidate will have strong experience in building data...

  • Pipeline Design Engineer

    3 weeks ago

    Mumbai, Maharashtra, India Petro Manpower Engineering Consultant Full time

    We have an urgent requirement for the Pipeline Design Engineer - for OMAN Location OMAN based Client Duration - Client Payroll Long Term Experience 15 to 25 yrs in - Pipeline Design Engineering to Oil and Gas - Pipeline Qualification - Mechanical Engineering BE Job Profile - Hands-on Experience in the the following - 1 Pipeline Route Selection ...

  • Senior Data Engineer

    2 hours ago

    Mumbai, Maharashtra, India beBeeData Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

    Job Title: Senior Data EngineerThis is a senior-level position that requires extensive experience in designing, developing, and maintaining large-scale data infrastructure.The ideal candidate will have a strong background in data engineering, analytics, and software development, with expertise in cloud platforms, database technologies, and data visualization...

  • Leading Data Pipeline Expert

    8 hours ago

    Mumbai, Maharashtra, India beBeeData Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

    Cloud Data EngineerWe are seeking a seasoned Cloud Data Engineer with expertise in designing, building and maintaining large-scale data pipelines. The ideal candidate has experience with AWS cloud services such as S3, EC2, RDS, DynamoDB, Athena and Redshift.Design, build and maintain production-level data pipelines that handle terabytes of structured and...