Senior Data Pipeline Specialist

4 days ago

Gurgaon, Haryana, India beBeeDataEngineer Full time ₹ 20,00,000 - ₹ 25,00,000
Job Overview

A Data Engineer is responsible for designing, building, and maintaining scalable and secure data pipelines and infrastructure on GCP. This includes working with various GCP services like BigQuery, Dataflow, Cloud Storage, and Pub/Sub to create efficient data solutions.

Key responsibilities include developing ETL processes, implementing data governance practices, and ensuring data quality and accessibility.

Responsibilities
  1. Designing and Implementing Data Pipelines:
    1. Build and optimize data pipelines using GCP services like Dataflow, Dataproc, BigQuery, and Cloud Storage.
    2. Develop and implement Extract, Transform, Load (ETL) processes using tools like Python and PySpark.
  2. Data Modeling and Storage:
  • Design and maintain data models, schemas, and documentation for efficient data storage and retrieval.
Data Governance and Security:
  • Implement data governance policies, security best practices, and access control within GCP.
Data Quality and Monitoring:
  • Ensure data quality through testing, validation, and monitoring of data pipelines.
Collaboration and Communication:
  • Work closely with data scientists, analysts, and other stakeholders to understand data requirements and deliver solutions.
Troubleshooting and Optimization:
  • Identify and resolve issues related to data pipelines and infrastructure, and optimize performance.
Staying Updated:
  • Keep abreast of the latest trends and advancements in data engineering and GCP technologies.

Essential Skills
  1. Programming Languages:
  • Strong proficiency in Python, SQL, and potentially Java.
GCP Services:
  • Hands-on experience with BigQuery, Dataflow, Cloud Storage, and other relevant GCP services.
Big Data Technologies:
  • Familiarity with Hadoop, Spark, or Kafka is often preferred.
Data Modeling and Warehousing:
  • Experience with data warehousing and data modeling techniques.
ETL and Data Pipelines:
  • Strong understanding of ETL processes and data pipeline development.
Cloud Infrastructure:
  • Experience with Infrastructure as Code (IaC) using tools like Terraform is a plus.
CI/CD:
  • Experience with Continuous Integration and Continuous Deployment (CI/CD) pipelines.
Problem-Solving:
  • Excellent analytical and problem-solving skills.
Communication:
  • Ability to communicate effectively with technical and non-technical audiences.

Working at [Company Name]

  • Senior Data Pipeline Engineer

    4 hours ago

    Gurgaon, Haryana, India beBeeData Full time ₹ 1,50,000 - ₹ 28,00,000

    Job Description:A Data Integration Specialist designs, develops and maintains complex data pipelines to extract, transform and load data into target systems. Collaborating with cross-functional teams, this role identifies business requirements and creates solutions that meet those needs.Develops ETL processes using Informatica PowerCenter to optimize...

  • Data Pipeline Architect

    2 days ago

    Gurgaon, Haryana, India beBeeDataEngineer Full time ₹ 15,00,000 - ₹ 28,00,000

    Job Overview:We are seeking a skilled professional to fill the role of Senior Data Engineer, responsible for designing and implementing data pipelines and extracting information from various sources.Key Responsibilities:Data Extraction and Parsing: Develop automated systems to extract relevant data from websites, ensuring accuracy and uptime.Pipeline...

  • Senior Cloud Engineer for Data Pipelines

    3 days ago

    Gurgaon, Haryana, India beBeeDataEngineer Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

    Job Opportunity:We are seeking a seasoned Senior Cloud Engineer to join our team. As a key member, you will be responsible for designing, developing, and maintaining scalable data pipelines.The ideal candidate will have a deep understanding of cloud design patterns, experience working with Snowflake as a data warehouse solution, and hands-on expertise with...

  • Senior Data Pipeline Engineer

    5 days ago

    Gurgaon, Haryana, India beBeeKafkaIntegration Full time US$ 1,04,000 - US$ 1,30,878

    We are seeking a highly skilled Kafka Integration Specialist with extensive experience in designing, developing and integrating Apache Kafka solutions.This role requires a strong understanding of distributed systems and data streaming concepts to deliver robust real-time integration solutions. We are looking for an expert in creating and maintaining...

  • Data Pipeline Specialist

    4 days ago

    Gurgaon, Haryana, India beBeeDataEngineer Full time ₹ 15,00,000 - ₹ 20,10,000

    About the Role:We are seeking a skilled professional to design, build, and optimize data pipelines while ensuring security, scalability, and high performance.This role will involve working closely with global teams on large-scale data projects in a collaborative and innovation-driven environment.Location: Gurgaon (Hybrid)Experience: 7+ yearsEmployment Type:...

  • Data Pipeline Specialist

    4 days ago

    Gurgaon, Haryana, India beBeeBusiness Full time ₹ 15,00,000 - ₹ 19,00,000

    Job DescriptionAn experienced Business Analyst is required to design and develop scalable data pipelines using PySpark and Delta Lake for efficient processing of large, structured, and semi-structured datasets.The ideal candidate will apply analytical skills to identify trends, patterns, and opportunities for process improvement by analyzing complex data...

  • Senior Data Quality Specialist

    1 week ago

    Gurgaon, Haryana, India beBeeDataQuality Full time ₹ 15,00,000 - ₹ 20,00,000

    As a senior data quality specialist, you will be responsible for designing and conducting advanced training programs.Key ResponsibilitiesDesigning and delivering interactive practical training sessions.Enabling teams to leverage data quality governance and AI-driven analytics solutions.RequirementsAt least 10 years of experience in data quality training and...

  • Visionary Data Architect

    4 days ago

    Gurgaon, Haryana, India beBeeDataEngineer Full time ₹ 20,00,000 - ₹ 25,00,000

    Senior Data EngineerThe ideal candidate will have a strong background in data engineering, with expertise in ETL/data warehousing and cloud-based technologies.Key Responsibilities:Design scalable data pipelines using PySpark, Python, and cloud-based technologies.Maintain and optimize existing data systems, ensuring high performance and...

  • Senior Data Pipeline Specialist

    3 days ago

    Gurgaon, Haryana, India beBeeDataEngineer Full time ₹ 80,00,000 - ₹ 1,50,00,000

    Key Data Engineer RoleAs a key member of our team, we are seeking an experienced Data Engineer to join us in driving innovation and delivering high-quality solutions. The ideal candidate will have a strong background in data engineering, with expertise in designing, developing, and maintaining robust data pipelines using Python, PySpark, and SQL.Essential...

  • AI/ML Engineer

    4 days ago

    Gurgaon, Haryana, India beBeeDataPipeline Full time ₹ 40,00,000 - ₹ 50,00,000

    Job Title: AI/ML Engineer - Data Pipeline Specialist We are seeking a skilled AI/ML engineer to join our team. The successful candidate will be responsible for building, optimizing and scaling AI/ML pipelines for regulatory compliance solutions. Key Responsibilities:Build and optimize data pipelines to transform and clean data for RAG (Retrieval Augmented...