Data Pipeline Specialist

2 days ago

Mysore, Karnataka, India beBeeetldeveloper Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

Are you a data pipeline specialist looking for a new challenge?

We have an exciting opportunity for an ETL Developer to join our team of talented professionals.

In this role, you will be responsible for designing, developing, and maintaining efficient data pipelines using IBM DataStage and AWS Glue/Lambda. You will work closely with architects, business analysts, and data modelers to ensure timely and accurate delivery of critical data assets supporting analytics and AI/ML use cases.

Key Responsibilities

  • Design and Develop ETL Pipelines: Create scalable and efficient data pipelines using IBM DataStage and AWS Glue/Lambda for ingestion from varied sources like flat files, APIs, Oracle, DB2, etc.
  • Data Flow Optimization: Build and optimize data flows for loading curated datasets into Snowflake, leveraging best practices for schema design, partitioning, and transformation logic.
  • Collaboration and Review: Participate in code reviews, performance tuning, and defect triage sessions to ensure high-quality deliverables.
  • Data Governance: Work closely with data governance teams to ensure lineage, privacy tagging, and quality controls are embedded within pipelines.
  • CI/CD Integration: Contribute to the integration of ETL components using Git, Jenkins, and parameterized job configurations.
  • Troubleshooting: Troubleshoot and resolve issues in QA/UAT/Production environments as needed.

Required Skills and Qualifications

  • ETL Development Experience: 4+ years of experience in ETL development with at least 1–2 years in IBM DataStage (preferably CP4D version).
  • AWS and Snowflake Experience: Hands-on experience with AWS Glue (PySpark or Spark) and AWS Lambda for event-based processing, as well as experience working with Snowflake: loading strategies, stream-task, zero-copy cloning, and performance tuning.
  • Technical Proficiency: Proficiency in SQL, Unix scripting, and basic Python for data handling or automation.
  • Software Knowledge: Familiarity with S3, version control systems (Git), and job orchestration tools.
  • Data Quality: Experience with data profiling, cleansing, and quality validation routines.
  • Data Architecture: Understanding of data lake/data warehouse architectures and DevOps practices.

Good to Have

  • Metadata Tools: Experience with Collibra, BigID, or other metadata/governance tools.
  • Data Mesh: Exposure to Data Mesh/Data Domain models.
  • Agile Delivery: Experience with agile/Scrum delivery and Jira/Confluence tools.
  • Certification: AWS or Snowflake certification is a plus.

  • Data Pipeline Architect

    5 days ago

    Mysore, Karnataka, India beBeeDataEngineer Full time ₹ 20,00,000 - ₹ 25,00,000

    Job Title: Senior Data EngineerAbout the RoleWe are seeking a seasoned professional to build data pipelines and integrate healthcare data.This role is critical in creating scalable pipelines that process clinical encounter data, retrieve associated medical documents, and deliver validated information to machine learning systems for inference.You will work...

  • Data Pipeline Architect

    6 days ago

    Mysore, Karnataka, India beBeeDataPipelineDeveloper Full time ₹ 37,50,000 - ₹ 52,50,000

    Senior Data Pipeline DeveloperJob OverviewAs a Senior Data Pipeline Developer, you will be responsible for designing and implementing high-performance data pipelines using advanced ETL concepts and methodologies.Key ResponsibilitiesDesign and implement complex data pipelines utilizing ETL frameworks and tools such as SSIS and AWS GlueDevelop and maintain...

  • Azure Data Pipeline Architect

    5 days ago

    Mysore, Karnataka, India beBeeDataEngineer Full time ₹ 15,00,000 - ₹ 21,00,000

    Job Overview:We are seeking a skilled professional to design and deploy robust data pipelines that support data governance and analytics initiatives. This role will focus on building production-ready workflows for ingestion, transformation, quality checks, lineage capture, access auditing, cost usage analysis, retention tracking, and metadata integration,...

  • Chief Data Pipeline Architect

    2 hours ago

    Mysore, Karnataka, India beBeeDataEngineer Full time ₹ 15,00,000 - ₹ 25,00,000

    Key Data Engineer RoleCraft efficient data pipelines and collaborate with cross-functional teams to empower decision-making through data-driven insights.

  • Advanced Data Pipeline Architect

    4 days ago

    Mysore, Karnataka, India beBeeDataEngineer Full time US$ 1,00,000 - US$ 1,40,000

    **Job Summary**As a skilled data engineer, you will be responsible for designing and implementing efficient data pipelines using Python and AWS technologies. Your primary goal will be to ensure seamless data flow across various systems.",

  • Pipeline Development Specialist

    1 hour ago

    Mysore, Karnataka, India beBeeBusiness Full time ₹ 40,00,000 - ₹ 1,00,00,000

    Job DescriptionWe're looking for a highly skilled Pipeline Development Specialist to join our tight-knit team working directly with senior leadership. This role offers the opportunity to drive business growth across key markets while developing critical skills in pre-sales workflows and sales intelligence.The ideal candidate will be responsible for building...

  • ETL Specialist with Advanced Data Pipelining Expertise

    10 hours ago

    Mysore, Karnataka, India beBeeDatastage Full time ₹ 18,00,000 - ₹ 30,00,000

    ETL Developer with IBM DataStage ExperienceWe are seeking an experienced and skilled ETL Developer to join our data engineering team.Design, develop, and maintain scalable and efficient data pipelines using IBM DataStage, AWS Glue, and Snowflake.Build and optimize data flows for loading curated datasets into Snowflake, leveraging best practices for schema...

  • Data Architecture Specialist

    3 days ago

    Mysore, Karnataka, India beBeeData Full time ₹ 15,00,000 - ₹ 25,00,000

    Job Title: Data Architecture SpecialistWe are seeking a skilled Data Architecture Specialist to design, develop and maintain large-scale data processing systems.Key Responsibilities:Design and implement robust data pipelines using PySpark and HiveCollaborate with cross-functional teams to achieve business objectivesImplement data quality tools and ensure...

  • Data Insight Specialist

    5 days ago

    Mysore, Karnataka, India beBeeDataInsight Full time ₹ 18,00,000 - ₹ 25,00,000

    Unlock Data Insights">The role of Data Insight Specialist involves leveraging data science and data pipelining tools to drive business outcomes. The ideal candidate will have 5-7 years of experience in data science and a strong understanding of data pipelines.">Proficiency in Python for data pipelines and basic knowledge of GenAI are essential skills....

  • Cloud Data Specialist

    2 days ago

    Mysore, Karnataka, India beBeeData Full time ₹ 15,00,000 - ₹ 25,00,000

    Job Title: Cloud Data SpecialistAbout the Role:We are seeking a highly skilled data professional to join our team. The ideal candidate should have expertise in cloud-based data solutions and experience with Microsoft Azure services.The successful candidate will be responsible for designing, implementing, and maintaining robust data systems that support...