Data Pipeline Specialist

6 days ago

Mumbai, Maharashtra, India beBeeSpark Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

Big Data Expert Wanted

 

We are seeking a skilled Big Data expert to join our team. As a Big Data Engineer, you will design and develop scalable data pipelines using Apache Spark and Scala.

 

Responsibilities :

  • Design and develop scalable data pipelines using Apache Spark and Scala.
  • Optimize and troubleshoot Spark jobs for performance (e.g. memory management, shuffles, skew).
  • Work with massive datasets in on-prem Hadoop clusters or cloud platforms like AWS/GCP/Azure.
  • Write clean, modular Scala code using functional programming principles.
  • Collaborate with data teams to integrate with platforms like Snowflake, Databricks, or data lakes.
  • Ensure code quality, documentation, and CI/CD practices are followed.

 

Required Skills :

  • 3+ years of experience with Apache Spark in Scala.
  • Deep understanding of Spark internalsDAG, stages, tasks, caching, joins, partitioning.
  • Hands-on experience with performance tuning in production Spark jobs.
  • Proficiency in Scala functional programming (e.g. immutability, higher-order functions, Option/Either).
  • Proficiency in SQL.
  • Experience with any major cloud platform: AWS, Azure, or GCP.

 

Nice-to-Have :

  • Worked with Databricks, Snowflake, or Delta Lake.
  • Exposure to data pipeline tools like Airflow, Kafka, Glue, or BigQuery.
  • Familiarity with CI/CD pipelines and Git-based workflows.
  • Comfortable with SQL optimization and schema design in distributed environments.

  • Master Data Pipeline Specialist

    1 week ago

    Mumbai, Maharashtra, India beBeeDataEngineer Full time US$ 1,00,000 - US$ 1,40,000

    Job Title: Master Data Pipeline SpecialistAt diverse-lynx, we are seeking a highly skilled Data Engineer to design, develop, and maintain robust, scalable, and efficient data pipelines. Your primary responsibility will be to collect, process, and store structured and unstructured data from various sources.

  • Data Pipelines Expert

    3 days ago

    Mumbai, Maharashtra, India beBeePipeline Full time ₹ 15,00,000 - ₹ 28,00,000

    Job Title: Data Pipeline SpecialistWe are seeking a highly skilled and experienced professional to join our team as a Data Pipeline Specialist. In this role, you will be responsible for designing, developing, and maintaining large-scale data pipelines that enable us to extract insights from complex datasets.Required Skills:Proficiency in cloud-based...

  • Pipeline Data Specialist

    1 day ago

    Mumbai, Maharashtra, India beBeePipeline Full time ₹ 15,00,000 - ₹ 28,00,000

    Job OverviewWe are seeking a skilled GIS professional to support pipeline integrity projects.The ideal candidate will be responsible for extracting, processing, and validating GIS data while ensuring compliance with industry standards.Key Responsibilities:Extract GIS data from enterprise geodatabases and convert it into file geodatabases for further...

  • Data Pipeline Specialist

    1 week ago

    Mumbai, Maharashtra, India beBeeDataEngineer Full time US$ 1,00,000 - US$ 1,50,000

    Data Engineer OpportunityWe are seeking a skilled Data Engineer to design and develop scalable data pipelines.This role involves working with streaming data, collaborating with data scientists, and delivering high-quality solutions.

  • Pipeline Infrastructure Specialist

    4 days ago

    Mumbai, Maharashtra, India beBeePipelineInfrastructure Full time ₹ 1,04,000 - ₹ 1,30,878

    Job Title: Pipeline Infrastructure SpecialistWe are seeking an experienced Pipeline Infrastructure Specialist to join our team. As a key member of our design team, you will be responsible for preparing high-quality pipeline design drawings, including alignment sheets, plan and profile sheets, crossing drawings, and HDD drawings.The ideal candidate should...

  • Senior Data Pipeline Architect

    1 week ago

    Mumbai, Maharashtra, India beBeeDataEngineer Full time ₹ 20,00,000 - ₹ 25,00,000

    Transforming marketing data into actionable insights requires a unique blend of technical expertise and analytical thinking. As a key member of our data infrastructure team, you will design, develop, and maintain robust data pipelines, warehouses, and analytics platforms.Design and implement scalable data pipelines for marketing data ingestion and...

  • Data Pipeline Migration Specialist

    1 week ago

    Mumbai, Maharashtra, India beBeeDataGovernance Full time ₹ 1,75,00,000 - ₹ 2,50,00,000

    Technical Leadership OpportunityThis is a senior-level position that oversees the technical aspects of implementing data pipeline migration requirements, ensuring our platform is used to its full potential through designing and building applications around business stakeholder needs.Main ResponsibilitiesCollaborate directly with stakeholders to gather...

  • Data Architect and Pipeline Specialist

    4 days ago

    Mumbai, Maharashtra, India beBeeDataEngineer Full time ₹ 1,80,00,000 - ₹ 2,50,00,000

    Job SummaryWe are seeking a seasoned Senior Data Engineer to spearhead data engineering efforts.The successful candidate will design and develop scalable data pipelines, oversee junior data engineers, and collaborate with cross-functional teams to drive business outcomes.Key Responsibilities:Create and maintain optimal data pipeline architecture for...

  • Data Pipeline Architect

    4 days ago

    Mumbai, Maharashtra, India beBeePipeline Full time ₹ 2,00,00,000 - ₹ 2,50,00,000

    Job Title: Data Pipeline ArchitectWe are seeking an experienced professional to design, develop and manage scalable data pipelines.Key Responsibilities:Design and Develop Pipelines: Utilize expertise in designing and developing data pipelines using Oracle Data Flow or Apache Spark (PySpark) to create efficient and reliable data solutions.Ensure Seamless...

  • Lead Data Pipeline Specialist

    6 days ago

    Mumbai, Maharashtra, India beBeeData Full time ₹ 1,20,00,000 - ₹ 2,40,00,000

    Senior Data EngineerWe are seeking a Senior Data Engineer to join our team. As a member of this team, you will be responsible for building data pipelines to ingest various source data from enterprise technologies and PitchBook Platform data.You will work with a range of data and reporting technologies (e.g. Python, Docker, Tableau, Power BI) to build upon a...