Efficient Data Pipeline Architect

1 week ago

Vellore, Tamil Nadu, India beBeeETLEngineer Full time ₹ 1,50,00,000 - ₹ 2,00,00,000
ETL Developer - DataStage, AWS, Snowflake

Job Summary:

We seek an ETL developer with experience in IBM DataStage and AWS Glue to join our data engineering team. The ideal candidate will have a strong background in designing and developing efficient ETL pipelines using IBM DataStage (CP4D) and AWS Glue/Lambda.

  • The successful candidate will collaborate with architects, business analysts, and data modelers to ensure timely and accurate delivery of critical data assets supporting analytics and AI/ML use cases.
  • They will be responsible for designing, developing, and maintaining ETL pipelines using IBM DataStage (CP4D) and AWS Glue/Lambda for ingestion from varied sources like flat files, APIs, Oracle, DB2, etc.
  • Furthermore, the candidate will build and optimize data flows for loading curated datasets into Snowflake, leveraging best practices for schema design, partitioning, and transformation logic.

Key Responsibilities:

  • Participate in code reviews, performance tuning, and defect triage sessions.
  • Work closely with data governance teams to ensure lineage, privacy tagging, and quality controls are embedded within pipelines.
  • Contribute to CI/CD integration of ETL components using Git, Jenkins, and parameterized job configurations.
  • Troubleshoot and resolve issues in QA/UAT/Production environments as needed.
  • Adhere to agile delivery practices, sprint planning, and documentation requirements.

Required Skills and Qualifications:

  • 4+ years of experience in ETL development with at least 1-2 years in IBM DataStage (preferably CP4D version).
  • Hands-on experience with AWS Glue (PySpark or Spark) and AWS Lambda for event-based processing.
  • Experience working with Snowflake: loading strategies, stream-task, zero-copy cloning, and performance tuning.
  • Proficiency in SQL, Unix scripting, and basic Python for data handling or automation.
  • Familiarity with S3, version control systems (Git), and job orchestration tools.
  • Experience with data profiling, cleansing, and quality validation routines.
  • Understanding of data lake/data warehouse architectures and DevOps practices.

Good to Have:

  • Experience with Collibra, BigID, or other metadata/governance tools
  • Exposure to Data Mesh/Data Domain models
  • Experience with agile/Scrum delivery and Jira/Confluence tools
  • AWS or Snowflake certification is a plus

  • Data Pipeline Architect

    6 days ago

    Vellore, Tamil Nadu, India beBeeDataPipeline Full time ₹ 2,00,00,000 - ₹ 2,50,00,000

    Data Pipeline ArchitectDesign and develop scalable, reliable data platforms to support business objectives.We are looking for a seasoned Data Architect to design, build, and maintain complex data pipelines. You will collaborate with stakeholders to gather requirements and translate business needs into robust technical solutions.Develop data pipelines using...

  • Senior Data Architect

    6 days ago

    Vellore, Tamil Nadu, India beBeeData Full time ₹ 15,00,000 - ₹ 25,00,000

    A top-tier organization is seeking a seasoned Data Architect to spearhead the development of next-generation data pipelines. The ideal candidate will have extensive experience in designing high-throughput, low-latency data pipelines using Python and cloud infrastructure.Key ResponsibilitiesData Pipeline Development: Design and build scalable data pipelines...

  • Data Pipeline Architect

    1 week ago

    Vellore, Tamil Nadu, India beBeeDataEngineer Full time ₹ 9,00,000 - ₹ 12,00,000

    Key Roles and ResponsibilitiesWe are seeking a highly skilled professional to build and maintain scalable data pipelines, connecting Aurora MySQL to Redshift Serverless and S3/QuickSight. Key tasks include designing data models, materialized views, and APIs to enable fast reporting and analytics.Must-Have Skills and QualificationsExpertise in building and...

  • Chief Data Architect

    2 weeks ago

    Vellore, Tamil Nadu, India beBeeData Full time ₹ 1,80,00,000 - ₹ 2,30,00,000

    We are seeking a Senior Data Engineer to join our team of experts in designing, developing, and maintaining large-scale data processing systems.The ideal candidate will have 5+ years of experience in data engineering or related fields, with a focus on big data technologies and cloud platforms. In this role, you will be responsible for architecting and...

  • Senior Data Pipeline Specialist

    2 weeks ago

    Vellore, Tamil Nadu, India beBeeDataEngineer Full time ₹ 2,00,00,000 - ₹ 2,09,00,000

    About Us:We're a leading provider of telenutrition and foodcare solutions. Our platform empowers healthier food choices, drives lasting behavior change, and delivers long-term health outcomes.Our highly personalized digital platform guides users on a tailored journey to eating well while saving time and money.We integrate dietary assessments and nutrition...

  • Senior Enterprise Architect

    7 days ago

    Vellore, Tamil Nadu, India beBeeCloud Architect Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

    Key Position:We are seeking a highly skilled professional to lead the design and implementation of complex, scalable workflows and solutions that align with business needs and technology goals.The ideal candidate will have expertise in cloud-native architectures, data integration strategies, and DevOps best practices. Develop cloud-native architectures,...

  • Chief Data Pipeline Architect

    2 weeks ago

    Vellore, Tamil Nadu, India beBeeData Full time ₹ 2,00,00,000 - ₹ 2,50,00,000

    Azure Data EngineerWe are seeking a skilled Azure Data Engineer to fill this key role in our organization.Key Responsibilities:Design and implement data pipelines using PySpark, SQL, and Azure Data Factory (ADF)Collaborate with stakeholders to deliver scalable data-driven outcomesWork in Agile teams to handle data validation, transformation, and...

  • Cloud Data Architect

    6 days ago

    Vellore, Tamil Nadu, India beBeeData Full time ₹ 15,00,000 - ₹ 20,00,000

    Job Title: Cloud Data Architect">About the Role:We are seeking an experienced Cloud Data Architect to design, build, and maintain scalable data processing systems on cloud platforms. As a key member of our team, you will be responsible for developing efficient data models and schemas for structured and unstructured data.Key Responsibilities:Data Pipeline...

  • Senior Data Architect

    2 weeks ago

    Vellore, Tamil Nadu, India beBeeData Full time ₹ 10,00,000 - ₹ 20,20,000

    Our organization is seeking an experienced data engineer with a strong background in ETL pipeline design, data modeling, and data infrastructure to work on our modern data platform.This role involves designing, building, and maintaining efficient, scalable data pipelines using various tools and technologies such as dbt, Airflow, and Apache Spark.The ideal...

  • Data Solutions Architect

    2 weeks ago

    Vellore, Tamil Nadu, India beBeeData Full time ₹ 10,00,000 - ₹ 20,00,000

    **Job Title: Data Solutions Architect**We are seeking a skilled Data Solutions Architect to join our team. The ideal candidate will have expertise in designing, building and maintaining data systems.The role involves cross-functional collaborations and ensuring highest standards of data quality and performance by extending expertise in data engineering, data...