Data Pipeline Specialist

1 day ago

Varanasi, Uttar Pradesh, India beBeeDataEngineer Full time US$ 1,20,000 - US$ 1,50,000

Job Description:

As a skilled data engineer, you will be responsible for designing and implementing robust data pipelines using Python and SQL. You will work with modern DevOps/configuration management tools to manage multi-environment deployments.

  • Design, implement, and manage CI/CD pipelines using Github Actions or equivalent tools.
  • Configure and maintain Github branch protection rules and workflows to ensure smooth code integration and deployment.
  • Build, optimize, and debug Spark jobs in Databricks for large-scale data processing.
  • Automate Databricks deployments using Databricks Asset Bundles (DABs) or similar frameworks.
  • Develop, test, and maintain robust data pipelines using Python and SQL.
  • Implement unit testing, validation, and quality checks for data workflows.
  • Manage multi-environment deployments (Dev, Stage, Prod) ensuring reliability and consistency.
  • Write and maintain configuration files (YAML, JSON) for workflows, pipelines, and automation.
  • Troubleshoot and resolve configuration or syntax issues in YAML and related files.
  • Keep critical Databricks notebooks running reliably to ensure business continuity.

Required Skills and Qualifications:

  • Strong skills in SQL, Databricks (Spark), and Python/R.
  • Strong knowledge of Github and CI/CD practices (branch protection, Github Actions workflows).
  • Experience in implementing unit tests and quality checks in data workflows.
  • Knowledge of Databricks Asset Bundles (DABs) or similar automation tools.
  • Familiarity with YAML configuration management, debugging, and error resolution.
  • Strong problem-solving and debugging skills in data engineering workflows.
  • Experience with Snowflake or another cloud data warehouse.
  • Hands-on experience with Tableau/Power BI dashboards and a good sense for usability.

  • Principal Data Pipeline Specialist

    2 days ago

    Varanasi, Uttar Pradesh, India beBeeDataEngineer Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

    Job Overview:We are seeking an experienced Data Engineer to design, develop, and maintain large-scale data pipelines using Azure Databricks and PySpark. This role involves collaboration with cross-functional teams to integrate data engineering solutions with business requirements, ensuring data quality, security, and compliance throughout the data...

  • Senior Data Pipeline Specialist

    1 day ago

    Varanasi, Uttar Pradesh, India beBeeDataEngineer Full time ₹ 18,00,000 - ₹ 27,00,000

    Meet the requirements for a senior data engineer position to create and maintain ETL/ELT pipelines in our organization.Key ResponsibilitiesPipeline Development: Design, build, and manage efficient ETL/ELT pipelines using AWS services like Lambda, DMS, and Glue.Data Transformation: Implement complex business logic and data transformations utilizing Snowflake...

  • Cloud Data Pipelines Specialist

    2 hours ago

    Varanasi, Uttar Pradesh, India beBeeDataEngineer Full time US$ 1,80,000 - US$ 2,00,000

    Job Title:">Data Engineer - Cloud Integration">Description:">We are seeking a skilled professional to design, build, and maintain scalable data pipelines using Azure Data Factory, Azure Synapse Analytics, and Azure Data Lake.">This role requires expertise in data governance frameworks, ensuring data quality across all pipelines.">The ideal candidate will...

  • Data Pipelines Expert

    13 hours ago

    Varanasi, Uttar Pradesh, India beBeedataEngineer Full time ₹ 2,00,00,000 - ₹ 2,50,00,000

    Job TitleA Data Engineer is needed to design, build and maintain high-quality, secure and scalable data pipelines using various technologies.

  • AI Pipeline Specialist

    6 days ago

    Varanasi, Uttar Pradesh, India beBeeMachine Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

    AI Pipeline SpecialistAs a pivotal member of our team, you will play a critical role in designing, implementing, and deploying cutting-edge Artificial Intelligence (AI) and Machine Learning (ML) pipelines across various cloud environments.Key ResponsibilitiesDesign and develop AI/ML pipelines using AWS services like SageMaker, Fargate, and Bedrock.Apply...

  • Senior Data Engineer

    2 hours ago

    Varanasi, Uttar Pradesh, India beBeeData Full time ₹ 15,00,000 - ₹ 21,00,000

    Job OpportunityWe are seeking a skilled data expert to take on the challenge of designing, building, and maintaining robust data pipelines for financial data.

  • Data Architect Specialist

    17 hours ago

    Varanasi, Uttar Pradesh, India beBeeDataArchitect Full time US$ 80,000 - US$ 1,40,000

    Data Architect SpecialistAre you a data expert looking for a challenging role in the field of data engineering? Do you have experience with designing and developing scalable data pipelines using Python and AWS services such as S3, Redshift, Lambda, and Glue? If so, we want to hear from you!Key Responsibilities:Data Pipeline Development: Develop and maintain...

  • Data Pipeline Specialist

    8 hours ago

    Varanasi, Uttar Pradesh, India beBeeDataEngineer Full time ₹ 20,00,000 - ₹ 35,00,000

    Oil and Gas Data Engineer OpportunityJob Description:We are seeking a seasoned Oil and Gas Data Engineer with expertise in designing, building, and optimizing scalable data pipelines and cloud solutions. The ideal candidate will have hands-on experience with AWS cloud services, including S3, Redshift, Glue, Athena, EMR, Lake Formation, and Lambda.Key...

  • Data Scientist Specialist

    1 day ago

    Varanasi, Uttar Pradesh, India beBeeMachine Full time ₹ 37,75,255 - ₹ 50,31,850

    We're looking for a skilled Machine Learning Specialist to join our team.Key Responsibilities:Proficient in Python language and familiar with Tensorflow/Keras frameworks.Experience in building ML Pipeline and setting up big data processing pipelines.Ability to evaluate models and benchmark various model versions on different data and report...

  • Cloud Data Specialist

    1 day ago

    Varanasi, Uttar Pradesh, India beBeeData Full time ₹ 15,00,000 - ₹ 20,00,000

    Cloud Data SpecialistWe are seeking a skilled Data Engineer to join our organization. The successful candidate will work on designing, building and maintaining large-scale data systems using Azure technologies.Key Responsibilities:Design and develop scalable data pipelines using cloud-based services such as Azure Data Factory (ADF) and Azure Databricks...