Data Pipeline Specialist

2 days ago

Bengaluru, Karnataka, India beBeeData Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

About the Job

Are you ready for a new career challenge?

We are currently seeking an ETL Developer to join our team.

The successful candidate will have at least 5 years of experience in designing and implementing data pipelines using Databricks (PySpark/Scala/SQL).

The role involves assessing current-state Informatica ETL jobs, mappings, workflows, and dependencies, and redesigning them for optimal performance and scalability in the Databricks environment.

You will also work closely with data architects, DevOps, QA, and business stakeholders to validate functionality and ensure successful migration.

Develop automated testing and validation scripts to verify data integrity post-migration.

Evaluate design patterns, migration strategy, and data lineage for audit and compliance.

This is a fantastic opportunity to join our organization and contribute your skills and expertise.

Below are the key responsibilities:

Key Responsibilities:

  • Assess and understand current-state Informatica ETL jobs, mappings, workflows, and dependencies.
  • Redesign and implement equivalent data pipelines using Databricks (PySpark/Scala/SQL).
  • Optimize performance and scalability of the migrated workflows in the Databricks environment.
  • Collaborate with data architects, DevOps, QA, and business stakeholders to validate functionality and ensure successful migration.
  • Develop automated testing and validation scripts to verify data integrity post-migration.
  • Evaluate design patterns, migration strategy, and data lineage for audit and compliance.

Requirements:

To be successful in this role, you will need:

Skills and Qualifications:

  • At least 5 years of experience in designing and implementing data pipelines using Databricks (PySpark/Scala/SQL).
  • Strong understanding of Informatica ETL jobs, mappings, workflows, and dependencies.
  • Ability to collaborate with cross-functional teams to validate functionality and ensure successful migration.
  • Familiarity with automated testing and validation scripts.
  • Excellent problem-solving skills.

What We Offer:

We offer a competitive salary and benefits package, as well as opportunities for growth and development within the company.

This is a great opportunity to join our organization and contribute your skills and expertise.

We look forward to hearing from you

  • Data Pipeline Specialist

    1 hour ago

    Bengaluru, Karnataka, India beBeeDataPipeline Full time US$ 1,00,000 - US$ 1,50,000

    Job Overview- We are seeking a highly skilled Data Pipeline Specialist to join our team. The ideal candidate will have experience building and productionizing data pipelines using industry-leading technologies.- A strong background in working with complex SQL datasets is required, as well as expertise in building efficient SQL scripts using BigQuery Syntax-...

  • Data Pipeline Engineer

    3 hours ago

    Bengaluru, Karnataka, India beBeeData Full time ₹ 1,00,00,000 - ₹ 2,00,00,000

    The Role of Senior Data Integration SpecialistJob Overview:We are seeking an experienced Senior Data Integration Specialist to join our team. This is a key position that will play a vital role in the smooth operation of our enterprise data integration platform.As a Senior Data Integration Specialist, you will be responsible for designing, implementing, and...

  • Data Pipeline Specialist

    5 days ago

    Bengaluru, Karnataka, India beBeeDataEngineer Full time

    Job Title: Azure Databricks EngineerAs a skilled Data Engineer, you will play a pivotal role in our dynamic team.High Impact Required: Immediate joiners only. Please note that candidates with less than 12 months of experience will not be considered.Key Responsibilities:Our ideal candidate will have strong expertise in building and optimizing data pipelines...

  • Cloud Data Pipeline Specialist

    1 hour ago

    Bengaluru, Karnataka, India beBeeDataEngineer Full time ₹ 2,00,00,000 - ₹ 2,50,00,000

    We are seeking a seasoned expert to spearhead the design and implementation of large-scale data pipelines on Amazon Web Services (AWS) cloud infrastructure.The ideal candidate will possess strong technical expertise in AWS services such as Amazon S3, AWS Glue, AWS Lambda, and Amazon Redshift, as well as hands-on experience with Apache Spark for data...

  • Principal Data Pipeline Architect

    1 day ago

    Bengaluru, Karnataka, India beBeeDataEngineer Full time US$ 1,25,000 - US$ 1,75,000

    About the RoleWe are seeking a Senior Data Engineer with expertise in building scalable data pipelines and integrating healthcare data. Reporting to the Head of AI and Engineering, you will design, build, and operate the backbone of our AI-powered applications.You will work closely with software engineers, ML specialists, and clinical partners to bring our...

  • Principal Data Pipeline Specialist

    8 hours ago

    Bengaluru, Karnataka, India beBeeDataEngineer Full time ₹ 1,50,00,000 - ₹ 2,00,00,000

    Azure Data Engineer Position">We are seeking an experienced Azure Data Engineer to join our team. The successful candidate will design, build, and maintain robust data pipelines to curate and ingest data efficiently.Key Responsibilities:Design, build, and maintain data pipelines to ensure efficient data ingestion.Work with Azure cloud services to deploy and...

  • Lead Data Pipeline Specialist

    1 day ago

    Bengaluru, Karnataka, India beBeeDataEngineer Full time US$ 1,04,000 - US$ 1,30,878

    Job Title: Senior Data EngineerAbout the Role:This is an exceptional opportunity to join our organization as a Senior Data Engineer. As a key member of our data team, you will be responsible for designing, building, and optimizing scalable data pipelines using cloud-native technologies.Key Responsibilities:Design and architect high-throughput ETL pipelines...

  • Data Pipeline Specialist

    2 days ago

    Bengaluru, Karnataka, India beBeeDataEngineer Full time ₹ 20,00,000 - ₹ 25,00,000

    We're seeking a skilled Data Engineer to support our team.Key Responsibilities:Design, build, and maintain robust, scalable data pipelines for large volume data from various sources.Integrate data from multiple sources, ensuring consistency and reliability.Develop and maintain efficient data models and schemas.Manage and optimize databases for performance,...

  • Senior Data Pipeline Specialist

    5 days ago

    Bengaluru, Karnataka, India beBeeAutomatedTesting Full time

    Job Opportunity:The role of Senior Consultant involves designing, developing and implementing automated test cases for complex data pipelines.Ability to design, develop, and implement automated test cases for complex data pipelines is a mustAWS experience is essentialPharma expertise is also requiredProficiency in programming languages such as Python, R, or...

  • Data Pipeline Specialist

    4 days ago

    Bengaluru, Karnataka, India beBeeDataEngineer Full time ₹ 9,00,000 - ₹ 12,00,000

    Job TitleSenior Data EngineerJob DescriptionAs a Senior Data Engineer, you will design, implement, and optimize data pipelines for large-scale data processing. You will work with data scientists, analysts, and engineers to ensure data availability and quality.Key responsibilities include developing and maintaining ETL/ELT workflows using Spark, Hadoop, Hive,...