Data Pipeline Specialist

5 days ago

Bharatpur, Rajasthan, India beBeeCloudMigrationEngineer Full time ₹ 20,00,000 - ₹ 25,00,000

About this Role

Next Pathway is seeking a highly skilled and experienced Senior Engineer to lead large-scale cloud migration projects. This role requires hands-on professionals who can design, develop, and implement complex data pipelines working independently and as key contributors within a project team.

Main Responsibilities:

  • Development and Implementation: Design, develop, and deploy robust and scalable data pipelines for migrating data from various on-premise sources to cloud platforms.
  • Technical Expertise: Serve as a subject matter expert in the project's core technologies, including Datastage, Azure Data Factory (ADF), Databricks, and PySpark.
  • Solution Design: Collaborate with architects and technical leads to design efficient and effective cloud migration strategies and data transformation solutions.
  • Code Quality: Ensure delivery of high-quality, well-documented, and maintainable code. Participate in code reviews and provide constructive feedback to peers.
  • Performance Optimization: Analyze and optimize the performance of data pipelines and applications to ensure efficiency and scalability.
  • Problem-Solving: Troubleshoot and resolve complex technical issues related to data migration, data quality, and system performance.
  • Collaboration: Work closely with cross-functional teams, including business analysts, data scientists, and other engineers, to understand requirements and deliver successful outcomes.
  • Mentorship: Provide guidance and technical support to junior engineers, helping them grow their skills and contribute effectively to the team.

Required Skills and Qualifications:

  • Minimum of 10 years of experience in the IT industry, with a focus on data engineering and ETL development.
  • Extensive hands-on experience with Datastage for building and managing ETL jobs.
  • Proven experience in Cloud Migration projects, with a strong understanding of the challenges and best practices involved in moving data to the cloud.
  • In-depth knowledge and hands-on experience with Azure Data Factory (ADF) for building and orchestrating data workflows.
  • Strong proficiency in Databricks and its ecosystem, including notebooks, jobs, and Delta Lake.
  • Expertise in PySpark for developing data processing applications on a distributed computing framework.
  • Solid understanding of cloud platforms, preferably Microsoft Azure.
  • Experience with SQL and database concepts.
  • Excellent analytical, problem-solving, and communication skills.
  • Bachelor's or Master's degree in Computer Science, Engineering, or a related field.

  • Data Pipeline Engineer

    22 hours ago

    Bharatpur, Rajasthan, India beBeeData Full time ₹ 20,00,000 - ₹ 25,00,000

    Job Title:ETL Developer - Data Pipeline SpecialistKey Responsibilities:Design and develop ETL pipelines using IBM DataStage (CP4D) and AWS Glue/Lambda for data ingestion from diverse sources.Create and optimize data flows for loading curated datasets into Snowflake, incorporating best practices for schema design, partitioning, and transformation...

  • Data Pipeline Manager

    4 days ago

    Bharatpur, Rajasthan, India beBeeData Full time ₹ 18,00,000 - ₹ 25,00,000

    Job Position: Technical SpecialistPosition Overview:Our organization is seeking a highly skilled Technical Specialist to fill a key role in our team. As a Technical Specialist, you will be responsible for designing, implementing, and managing data pipelines using Azure Data Factory.Key Responsibilities:Design and implement scalable data processing solutions...

  • Data Pipelines Specialist

    4 days ago

    Bharatpur, Rajasthan, India beBeeAzure Full time ₹ 15,00,000 - ₹ 20,00,000

    Job Title: Freelance Azure Data FactoryJob Description:We are seeking a skilled professional to support our data integration and ETL needs. The ideal candidate will have hands-on expertise in designing, building, and maintaining ADF pipelines for scalable and secure data flows.The role involves several key responsibilities, including:Designing and optimizing...

  • Data Pipeline Architect

    8 hours ago

    Bharatpur, Rajasthan, India beBeePipeline Full time ₹ 1,00,00,000 - ₹ 2,00,00,000

    Job Title: Data Pipeline ArchitectAbout the Role:We are seeking a highly skilled Data Pipeline Architect to join our team. In this role, you will design and develop data pipelines using Oracle Data Flow or Apache Spark (PySpark) with a solid understanding of data patterns, Python, and SQL.Key Responsibilities:Develop data pipelines that ensure seamless...

  • Lead Data Pipeline Specialist

    3 days ago

    Bharatpur, Rajasthan, India beBeeDataEngineer Full time ₹ 18,00,000 - ₹ 22,00,000

    Job DescriptionAs a skilled data engineer, you will be responsible for building and managing ETL pipelines using Azure Data Factory and Fabric toolbox. You will work on integration, development, and support within a newly implemented Fabric Data & Analytics Platform.The role involves delivering high-quality datasets to support analytics across business...

  • Building Robust Data Pipelines

    5 days ago

    Bharatpur, Rajasthan, India beBeeDataEngineering Full time ₹ 18,00,000 - ₹ 25,00,000

    Job Title: Azure Data EngineerWe are seeking a skilled Azure Data Engineer to design, build, and maintain robust data pipelines.Key Responsibilities:Design and implement data solutions using Azure cloud services.Maintain CI/CD pipelines for seamless data workflows.Collaborate with cross-functional teams to ensure data integrity, security, and...

  • Data Transformation Specialist

    8 hours ago

    Bharatpur, Rajasthan, India beBeeTransformation Full time ₹ 20,00,000 - ₹ 25,00,000

    We are seeking a Data Transformation Specialist to strengthen our team of experts in data processing and integration.Job Overview:This role involves developing and maintaining scalable data pipelines using Azure Functions (.NET) for efficient data processing. The successful candidate will design and build robust ELT pipelines with ADF / Azure Data Services,...

  • Senior Pipeline Hydraulic Specialist

    8 hours ago

    Bharatpur, Rajasthan, India beBeeHydraulic Full time ₹ 15,00,000 - ₹ 20,00,000

    Job Title: Senior Pipeline Hydraulic SpecialistWe are seeking an experienced professional to lead our water supply and irrigation projects.The ideal candidate will have a strong background in pipeline hydraulics, software used for same, and experience as design engineer for water supply pipeline project or Irrigation projects.Key Responsibilities:Determine...

  • Data Processing Pipeline Lead

    5 days ago

    Bharatpur, Rajasthan, India beBeeEngineering Full time ₹ 2,00,00,000 - ₹ 2,50,00,000

    Job OpportunityWe are seeking an experienced professional to lead the development and optimization of our data processing pipelines.Key Responsibilities:In this role, you will be responsible for designing and building efficient data processing pipelines using Apache Beam and Databricks. This includes creating reusable components, standardizing configuration...

  • Data Specialist

    7 days ago

    Bharatpur, Rajasthan, India beBeeDataSpecialist Full time ₹ 1,50,00,000 - ₹ 2,00,00,000

    Job DescriptionWe are seeking a Data Specialist to design, develop and maintain large-scale data pipelines. This role involves working with streaming data, collaborating with cross-functional teams and delivering high-quality solutions.The ideal candidate will have experience with Spark Core API, Spark SQL, Structured Streaming and Kafka...