Data Pipeline Engineer

1 week ago

Hyderabad, Telangana, India beBeeETLDeveloper Full time ₹ 15,00,000 - ₹ 25,00,000

Job Title: ETL Developer – DataStage, AWS, Snowflake

Experience: 5–7 Years

Location: Remote

Job Type: Full-time

About the Role

We are seeking a skilled and motivated ETL developer to join our data engineering team. You will work on building scalable and efficient data pipelines using IBM DataStage (on Cloud Pak for Data), AWS Glue, and Snowflake.

Key Responsibilities

  • Design, develop, and maintain ETL pipelines using IBM DataStage (CP4D) and AWS Glue/Lambda for ingestion from varied sources like flat files, APIs, Oracle, DB2, etc.
  • Build and optimize data flows for loading curated datasets into Snowflake, leveraging best practices for schema design, partitioning, and transformation logic.
  • Participate in code reviews, performance tuning, and defect triage sessions.
  • Work closely with data governance teams to ensure lineage, privacy tagging, and quality controls are embedded within pipelines.
  • Contribute to CI/CD integration of ETL components using Git, Jenkins, and parameterized job configurations.
  • Troubleshoot and resolve issues in QA/UAT/Production environments as needed.
  • Adhere to agile delivery practices, sprint planning, and documentation requirements.

Required Skills and Experience

  • 4+ years of experience in ETL development with at least 1–2 years in IBM DataStage (preferably CP4D version).
  • Hands-on experience with AWS Glue (PySpark or Spark) and AWS Lambda for event-based processing.
  • Experience working with Snowflake: loading strategies, stream-task, zero-copy cloning, and performance tuning.
  • Proficiency in SQL, Unix scripting, and basic Python for data handling or automation.
  • Familiarity with S3, version control systems (Git), and job orchestration tools.
  • Experience with data profiling, cleansing, and quality validation routines.
  • Understanding of data lake/data warehouse architectures and DevOps practices.

Good to Have

  • Experience with Collibra, BigID, or other metadata/governance tools
  • Exposure to Data Mesh/Data Domain models
  • Experience with agile/Scrum delivery and Jira/Confluence tools
  • AWS or Snowflake certification is a plus

The ideal candidate will have strong technical skills, excellent problem-solving abilities, and effective communication skills. If you are passionate about data engineering and want to be part of a dynamic team, please submit your application.

  • Senior Data Engineer

    7 days ago

    Hyderabad, Telangana, India Enable Data Full time ₹ 15,00,000 - ₹ 20,00,000 per year

    Experience Required: 8+YearsMode of work: RemoteSkills Required: Azure DataBricks, Eventhub, Kafka, Architecture,Azure Data Factory, Pyspark, Python, SQL, SparkNotice Period : Immediate Joiners/ Permanent/Contract role (Can join within September 15th 2025)Design, develop, and maintain scalable and robust data solutions in the cloud using Apache Spark and...

  • Azure Data Engineer ll

    7 days ago

    Hyderabad, Telangana, India Enable Data Full time US$ 90,000 - US$ 1,20,000 per year

    Enable Data Incorporated is currently seeking a skilled and experienced Azure Data Engineer to join our dynamic team. As a leading provider of advanced application, data, and cloud engineering services, Enable Data has developed deep expertise across various industries. We work closely with our customers to leverage modern solutions and technologies to drive...

  • Data Engineer

    7 days ago

    Hyderabad, Telangana, India ICE Data Services Full time US$ 1,25,000 - US$ 1,75,000 per year

    Job PurposeAt Intercontinental Exchange (ICE), we engineer technology, exchanges and clearing houses that connect companies around the world to global capital and derivative markets. With a leading-edge approach to developing technology platforms, we have built market infrastructure in all major trading centers, offering customers the ability to manage risk...

  • Senior Data Engineer

    7 days ago

    Hyderabad, Telangana, India Enable Data Incorporated Full time ₹ 15,00,000 - ₹ 20,00,000 per year

    Experience Required: 8+YearsMode of work: RemoteSkills Required: Azure DataBricks, Eventhub, Kafka, Architecture,Azure Data Factory, Pyspark, Python, SQL, SparkNotice Period : Immediate Joiners/ Permanent/Contract role (Can join within September 15th 2025)Design, develop, and maintain scalable and robust data solutions in the cloud using Apache Spark and...

  • Lead/Senior Data Engineer

    7 days ago

    Hyderabad, Telangana, India Data Economy Full time US$ 1,50,000 - US$ 2,00,000 per year

    We are seeking a Lead/Senior Data Engineer with 7-12 years of experience to architect, develop, and optimize data solutions in a cloud-native environment. The role requires strong expertise in AWS Glue, PySpark, and Python with a proven ability to design scalable data pipelines and frameworks for large-scale enterprise systems. Prior exposure to financial...

  • Data Engineer for Large-Scale Data Pipelines

    7 days ago

    Hyderabad, Telangana, India beBeeDataEngineering Full time ₹ 18,00,000 - ₹ 21,00,000

    We are seeking a seasoned Data Engineer to join our team.This role involves collaborating with clients in the Asia-Pacific region to provide technical resources.The ideal candidate will have a solid understanding of software engineering principles and experience working with distributed data technologies.A strong background in cloud computing and experience...

  • Senior Data Engineer

    7 days ago

    Hyderabad, Telangana, India Enable Data Incorporated Full time ₹ 15,00,000 - ₹ 20,00,000 per year

    Design, develop, and maintain scalable and robust data solutions in the cloud using Apache Spark and Databricks. Gather and analyse data requirements from business stakeholders and identify opportunities for data-driven insights. Build and optimize data pipelines for data ingestion, processing, and integration using Spark and Databricks. Ensure data...

  • Data Pipeline Specialist

    3 days ago

    Hyderabad, Telangana, India beBeeDataEngineer Full time ₹ 8,00,000 - ₹ 12,00,000

    Data Engineer PositionWe are seeking a skilled data engineer to join our team. The ideal candidate will be responsible for designing, implementing, and optimizing data pipelines on our platform.Data Pipeline Development: Design, develop, and maintain scalable data pipelines using Azure Synapse, Databricks & Apache Spark (PySpark).Data Integration: Integrate...

  • Data Engineer

    1 day ago

    Hyderabad, Telangana, India beBeeData Full time ₹ 8,00,000 - ₹ 15,00,000

    Job DescriptionOur team is seeking a talented Data Engineer to play a key role in designing, building and maintaining scalable data pipelines using Python and PySpark. The ideal candidate will have hands-on experience with distributed data systems and a solid foundation in AWS.This is an exciting opportunity to work closely with senior engineers and product...

  • Mastering Data Pipelines

    2 weeks ago

    Hyderabad, Telangana, India beBeeDataEngineer Full time ₹ 20,00,000 - ₹ 25,00,000

    Job Title: Data EngineerWe are seeking a highly skilled data professional to design, build, and optimize data pipelines.Key Responsibilities:Develop scalable, high-performance data architectures using Spark and SQL.Process large datasets using PySpark and drive business insights.Create real-time streaming applications with Spark Streaming/Kafka.Collaborate...