Senior Data Pipeline Engineer

4 days ago

Palakkad, Kerala, India beBeeETL Full time ₹ 18,00,000 - ₹ 25,40,000
Job Opportunity

Seeking a highly skilled ETL Developer to design, develop, and maintain data pipelines using IBM DataStage (CP4D) and AWS Glue/Lambda for ingestion from varied sources.

As an experienced ETL professional, you will be responsible for building and optimizing data flows for loading curated datasets into Snowflake, leveraging best practices for schema design, partitioning, and transformation logic.

Key Responsibilities:

  • Design, develop, and maintain ETL pipelines using IBM DataStage (CP4D) and AWS Glue/Lambda for ingestion from varied sources.
  • Build and optimize data flows for loading curated datasets into Snowflake, leveraging best practices for schema design, partitioning, and transformation logic.
  • Participate in code reviews, performance tuning, and defect triage sessions.
  • Work closely with data governance teams to ensure lineage, privacy tagging, and quality controls are embedded within pipelines.
  • Contribute to CI/CD integration of ETL components using Git, Jenkins, and parameterized job configurations.
  • Troubleshoot and resolve issues in QA/UAT/Production environments as needed.
Required Skills and Experience:
  • 4+ years of experience in ETL development with at least 1–2 years in IBM DataStage (preferably CP4D version).
  • Hands-on experience with AWS Glue (PySpark or Spark) and AWS Lambda for event-based processing.
  • Experience working with Snowflake: loading strategies, stream-task, zero-copy cloning, and performance tuning.
  • Proficiency in SQL, Unix scripting, and basic Python for data handling or automation.
  • Familiarity with S3, version control systems (Git), and job orchestration tools.
  • Experience with data profiling, cleansing, and quality validation routines.
  • Understanding of data lake/data warehouse architectures and DevOps practices.
Benefits and Opportunities:
  • Be part of a dynamic team focused on delivering high-quality data solutions.
  • Collaborate with cross-functional teams to drive business growth through data-driven insights.
  • Develop and maintain expertise in emerging technologies and tools.
  • Grow your career in a supportive and inclusive environment.

  • Senior Data Engineer

    4 days ago

    Palakkad, Kerala, India beBeeData Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

    Job DescriptionWe are seeking a highly skilled professional to join our team as a Big Data Engineer. In this role, you will be responsible for designing, building, and maintaining robust data pipelines that handle high-volume financial data.The ideal candidate will have experience with data pipeline tools, cloud platforms, and data warehousing. They will...

  • Data Engineer for Scalable Data Pipelines

    18 hours ago

    Palakkad, Kerala, India beBeeDataEngineer Full time ₹ 15,00,000 - ₹ 19,00,000

    We are seeking a highly skilled Data Engineer to lead the design and development of scalable data pipelines. The ideal candidate will have expertise in data engineering principles, with a strong understanding of cloud-based data platforms.Key Responsibilities:Design and develop efficient data pipelines using Airflow and SnowflakeCollaborate with...

  • Senior Data Engineer

    6 days ago

    Palakkad, Kerala, India beBeeDataEngineering Full time ₹ 1,00,00,000 - ₹ 1,50,00,000

    Job DescriptionAs a senior data engineering professional, you will design and implement large-scale data pipelines using Azure Databricks and Azure Data Lake. Your primary focus will be on building production-ready workflows for data ingestion, transformation, quality checks, lineage capture, access auditing, cost usage analysis, retention tracking, and...

  • Data Pipeline Developer

    7 days ago

    Palakkad, Kerala, India beBeeIngestion Full time ₹ 15,00,000 - ₹ 25,00,000

    Job OpportunityWe are seeking an Ingestion Specialist with expertise in data pipeline development and Kubernetes to fill a key role in our organization.This position requires a strong understanding of data ingestion processes, focusing on integrating various data sources into Databricks.The ideal candidate will have hands-on experience with data pipelines,...

  • Data Pipeline Specialist

    21 hours ago

    Palakkad, Kerala, India beBeeDataEngineer Full time ₹ 15,00,000 - ₹ 20,00,000

    Data Engineering ProfessionalWe are seeking a skilled data engineer to join our analytics division.You will be part of our data analytics and business intelligence vertical, responsible for conceptualizing and owning the build out of problem-solving data marts for consumption by data science and business intelligence teams. You will evaluate design and...

  • Senior Data Pipeline Specialist

    22 hours ago

    Palakkad, Kerala, India beBeeDataEngineer Full time ₹ 20,00,000 - ₹ 25,00,000

    Cloud Data Engineer PositionLemongrass is seeking a skilled Cloud Data Engineer to design, develop, and maintain scalable data pipelines using cloud-native tools. The ideal candidate will have extensive experience with AWS, Azure, and GCP, as well as knowledge of data governance, security, and compliance best practices.">Key Responsibilities:Design, develop,...

  • Chief Data Pipeline Engineer

    5 days ago

    Palakkad, Kerala, India beBeeData Full time ₹ 1,20,00,000 - ₹ 2,00,00,000

    Key Responsibilities:We are seeking a skilled Data Architect to develop and maintain data pipelines using Python and AWS (S3, Redshift, Lambda, Glue).Develop scalable data architectures using AWS services.Implement and optimize ETL processes for large datasets using Python and AWS Glue.Collaborate with cross-functional teams to design and implement data...

  • Senior Data Engineer

    4 days ago

    Palakkad, Kerala, India beBeeData Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

    Job OverviewAchieve success as a Senior Data Engineer with expertise in large-scale data processing and system design.Key Responsibilities:• Develop high-performance ETL workflows using Python and Pandas to handle complex supply chain operations and personalized customer experiences.• Implement data ingestion and transformation processes using Hive SQL...

  • Expert Data Pipeline Architect

    12 hours ago

    Palakkad, Kerala, India beBeeDataEngineer Full time ₹ 20,00,000 - ₹ 30,00,000

    Job Title:A Data Engineer is needed to design, build, and maintain large-scale data pipelines using Google Cloud Platform.Responsibilities:Develop, construct, test, and maintain data acquisition pipelines for structured and unstructured data in GCP.Create complex datasets based on business requirements.Construct 'big data' pipeline architecture.Identify...

  • Lead Data Pipeline Developer

    5 days ago

    Palakkad, Kerala, India beBeeDataEngineer Full time ₹ 1,80,00,000 - ₹ 2,50,00,000

    Job DescriptionWe are seeking a skilled Data Engineer to join our team in Mumbai. The ideal candidate will have strong experience in PySpark and hands-on expertise in building scalable data pipelines.The successful candidate will work on designing, developing, and maintaining batch and streaming data pipelines using Spark SQL, DataFrame, and RDD APIs. They...