Senior Data Pipeline Engineer

2 days ago

Coimbatore, Tamil Nadu, India beBeeDataPipeline Full time ₹ 15,00,000 - ₹ 20,00,000
About This Role

We're seeking an experienced Data Pipeline Specialist to join our data engineering team.

As a Data Pipeline Specialist, you will design and develop scalable and efficient data pipelines using IBM DataStage (on Cloud Pak for Data), AWS Glue, and Snowflake.

Main Responsibilities:

  • Design, build, and maintain ETL pipelines using IBM DataStage (CP4D) and AWS Glue/Lambda for ingestion from various sources such as flat files, APIs, Oracle, DB2, etc.
  • Create and optimize data flows for loading curated datasets into Snowflake, leveraging best practices for schema design, partitioning, and transformation logic.
  • Participate in code reviews, performance tuning, and defect triage sessions.
  • Collaborate with data governance teams to ensure lineage, privacy tagging, and quality controls are embedded within pipelines.
  • Contribute to the integration of ETL components using Git, Jenkins, and parameterized job configurations.
  • Troubleshoot and resolve issues in QA/UAT/Production environments as needed.
  • Adhere to agile delivery practices, sprint planning, and documentation requirements.
Required Skills and Experience
  • 4+ years of experience in ETL development with at least 1–2 years in IBM DataStage (preferably CP4D version).
  • Hands-on experience with AWS Glue (PySpark or Spark) and AWS Lambda for event-based processing.
  • Experience working with Snowflake: loading strategies, stream-task, zero-copy cloning, and performance tuning.
  • Proficiency in SQL, Unix scripting, and basic Python for data handling or automation.
  • Familiarity with S3, version control systems (Git), and job orchestration tools.
  • Experience with data profiling, cleansing, and quality validation routines.
  • Understanding of data lake/data warehouse architectures and DevOps practices.
Key Qualifications

To be successful in this role, you should have strong skills and knowledge in the following areas:

  • Data pipeline design and development.
  • ETL process optimization.
  • Cloud-based data storage and processing.
  • Agile software development methodologies.
  • Version control systems.
  • Automated testing and deployment.

  • Data Pipeline Testing Expert

    6 days ago

    Coimbatore, Tamil Nadu, India beBeeTestEngineer Full time ₹ 18,00,000 - ₹ 21,00,000

    About UsWe are a leading engineering and digital transformation consultancy, serving top Life Sciences, MedTech, and Healthcare clients across the US, Ireland, and Europe.Job Summary:We are seeking an experienced Senior Test Engineer to drive the testing strategy and execution for a project leveraging Databricks, Data Factory pipelines, and Power BI.Key...

  • Senior Data Pipeline Developer

    4 days ago

    Coimbatore, Tamil Nadu, India beBeeDataEngineer Full time US$ 1,00,000 - US$ 1,20,000

    Data EngineerWe are seeking a highly skilled Data Engineer to join our dynamic data team. As a key member of the infrastructure team, you will design, develop, and maintain robust data pipelines, data warehouses, and analytics platforms that serve as the backbone of our digital marketing product development.You will collaborate with cross-functional teams...

  • Data Pipeline Specialist

    2 days ago

    Coimbatore, Tamil Nadu, India beBeeDataPipeline Full time ₹ 30,00,000 - ₹ 40,00,000

    Key Data Pipeline SpecialistThe role entails designing, developing, and maintaining high-capacity data pipelines that efficiently handle structured and unstructured data.ResponsibilitiesDesign and Build Complex Pipelines: You will be responsible for designing and building complex production-level data pipelines utilizing AWS services such as S3, EC2, RDS,...

  • Unlock Your Potential as a Senior Data Pipeline Specialist

    5 days ago

    Coimbatore, Tamil Nadu, India beBeeDataEngineer Full time ₹ 12,00,000 - ₹ 20,00,000

    Senior Data Pipeline Specialist PositionAt our organization, we are seeking an experienced Senior Data Pipeline Specialist with a strong background in data engineering practices. The ideal candidate will have hands-on expertise in building and automating large-scale data pipelines, managing multi-environment deployments, and working with modern...

  • Senior Data Engineer

    1 day ago

    Coimbatore, Tamil Nadu, India ShellKode Full time US$ 1,50,000 - US$ 2,00,000 per year

    About The RoleWe are seeking a highly skilledSenior Data Engineerwith expertise inSQL, Python, and Sparkto join our team. The ideal candidate will have extensive experience in designing, building, and optimizing data pipelines onAWS,Databricks,Snowflake, andAirflow, enabling robust analytics and machine learning use cases. You will work closely with data...

  • Cloud Data Engineer

    3 days ago

    Coimbatore, Tamil Nadu, India beBeeData Full time ₹ 2,00,00,000 - ₹ 2,50,00,000

    Job OverviewThis role is focused on designing, building, and optimizing data pipelines using cloud-based technologies. As a senior data engineer, you will be responsible for creating scalable ingestion and transformation workflows across multiple cloud platforms.">Key ResponsibilitiesDevelop and maintain ETL/ELT data pipelines using Apache Spark, AWS Glue,...

  • Data Pipeline Architect

    4 days ago

    Coimbatore, Tamil Nadu, India beBeeETLDeveloper Full time ₹ 15,00,000 - ₹ 25,00,000

    Job Title: ETL DeveloperWe are seeking a skilled and motivated ETL professional to join our data engineering team.You will work on designing, developing, and maintaining scalable and efficient data pipelines using IBM DataStage (on Cloud Pak for Data), AWS Glue, and Snowflake.Main Responsibilities:Design, develop, and maintain ETL pipelines using IBM...

  • Senior Data Pipeline Specialist

    6 days ago

    Coimbatore, Tamil Nadu, India beBeeData Full time ₹ 20,00,000 - ₹ 25,00,000

    Database Engineer OpportunityWe are seeking a skilled ETL/Database Tester to join our team with 5+ years of experience in testing, monitoring, and improving data pipelines.Key Responsibilities:Design, implement, and execute comprehensive test plans for ETL processes and databases.Collaborate with cross-functional teams to identify and prioritize testing...

  • Chief Data Engineering Expert

    2 days ago

    Coimbatore, Tamil Nadu, India beBeeDataEngineer Full time ₹ 20,00,000 - ₹ 25,00,000

    Senior Data Engineer RoleWe are seeking an experienced Senior Data Engineer to develop and manage ETL pipelines using Azure Data Factory and Fabric toolbox.Design and implement data models to deliver high-quality datasets for analytics across business areas.Collaborate with cross-functional teams to integrate data engineering solutions with existing...

  • Data Pipeline Engineer

    3 days ago

    Coimbatore, Tamil Nadu, India beBeeDataPipeline Full time ₹ 15,00,000 - ₹ 25,00,000

    Job Description:We are seeking a skilled Data Pipeline Specialist to design, build, and optimize scalable data pipelines using Azure Data Factory.The ideal candidate will have hands-on experience with ADF pipelines, data integration, and transformations for smooth, secure, and efficient data flows.Design and implement Azure Data Factory pipelines for ETL and...