Big Data Engineer: Optimizing Scalable Data Pipelines

5 days ago


Thāne, Maharashtra, India beBeeData Full time ₹ 15,00,000 - ₹ 30,00,000
Big Data Engineer Position

The role is for a highly skilled Big Data Engineer with strong expertise in Scala, Spark, Python, NiFi, and Apache Kafka. The ideal candidate will have hands-on experience in building, scaling, and optimizing big data pipelines.

We are seeking a talented individual who can design, develop, and optimize large-scale data pipelines and distributed data processing systems. The successful candidate will work extensively with Scala, Spark (PySpark), and Python for data processing and transformation.

The ideal candidate will also have hands-on experience with Apache Kafka and orchestration tools like NiFi / Airflow. They should be proficient in writing efficient queries and perform data analysis using Jupyter Notebooks and SQL.

The job requires collaboration with cross-functional teams to design scalable cloud-based data architectures. The selected candidate will ensure delivery of high-quality code through code reviews, performance tuning, and best practices.

Mandatory Skills:

  • Scala - Minimum 3+ years of hands-on experience.
  • Strong Expertise in Spark (PySpark) and Python.
  • Hands-on Experience with Apache Kafka.
  • Knowledge of NiFi / Airflow for Orchestration.
  • Strong Experience in Distributed Data Systems (5+ years).
  • Proficiency in SQL and Query Optimization.
  • Good Understanding of Cloud Architecture.

Preferred Skills:

  • Exposure to Messaging Technologies like Apache Kafka or Equivalent.
  • Experience in Designing Intuitive, Responsive UIs for Data Analytics Visualization.
  • Familiarity with Splunk or Other Monitoring/Alerting Solutions.
  • Hands-on Experience with CI/CD Tools (Git, Jenkins).
  • Strong Grasp of Software Engineering Concepts, Data Modeling, and Optimization Techniques.


  • Thāne, Maharashtra, India beBeeData Full time ₹ 10,00,000 - ₹ 16,00,000

    Job DescriptionWe are seeking a skilled Big Data Engineer to join our team. As a key member of our Professional Services, you will play a vital role in designing, building, and maintaining high-performance data pipelines that process massive amounts of data.Key Responsibilities:Design and implement scalable batch processing systems using Python and big data...

  • Big Data Specialist

    6 days ago


    Thāne, Maharashtra, India beBeeDataEngineering Full time ₹ 2,00,00,000 - ₹ 2,50,00,000

    Databricks Data Engineer OpportunityKey Responsibilities:Develop and maintain scalable data pipelines and ETL processes on Databricks.Lead the architecture and implementation of high-performance data solutions.Collaborate with stakeholders to deliver actionable data insights.Optimize data workflows for performance, cost-efficiency, and reliability.Integrate...


  • Thāne, Maharashtra, India beBeeDataEngineer Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

    We are looking for a skilled Big Data professional to join our team as a Big Data Engineer"> Big Data Engineer ", we require a candidate with strong skills in data engineering and analytics. The ideal candidate will have hands-on experience with Big Data technologies, including data ingestion, processing, and storage.">Key Responsibilities:Data Ingestion:...


  • Thāne, Maharashtra, India beBeeData Full time US$ 80,000 - US$ 1,50,000

    Job SummaryThe ideal candidate will have expertise in developing and maintaining scalable data pipelines using big data technologies such as PySpark, AWS, and Databricks.Responsibilities include designing, building, and optimizing ETL/ELT pipelines, performing data profiling and validation, and collaborating with analytics teams to surface trustworthy KPIs...

  • Big Data Leader

    6 hours ago


    Thāne, Maharashtra, India beBeeData Full time ₹ 1,80,00,000 - ₹ 2,40,00,000

    ">Senior Data Architect">This role is responsible for leading data engineering teams globally and collaborating with cross-functional teams to drive business outcomes. ">Lead large-scale data engineering projects using distributed systems like Apache Spark, HDFS, and Kubernetes.Design and implement scalable data pipelines for big data processing using Scala,...

  • AWS Big Data Engineer

    6 hours ago


    Thāne, Maharashtra, India beBeeDataEngineering Full time ₹ 18,00,000 - ₹ 24,00,000

    Job Description:As a data engineer, you play a vital role in crafting and implementing robust data pipelines to process channel activity data from diverse sources. This encompasses digital channels, Hadoop environments, and Terradata.Key Responsibilities:Design and maintain scalable data architectures that integrate multiple sources of data for effective...

  • Data Engineer

    6 hours ago


    Thāne, Maharashtra, India beBeeDataEngineer Full time ₹ 18,00,000 - ₹ 24,00,000

    Unlock a career in data engineering and join the movement towards informed decision-making.About this roleWe are seeking an experienced data engineer to design and build efficient data pipelines that drive business outcomes. This is a hands-on role that requires problem-solving skills, collaboration with cross-functional teams, and a passion for building...


  • Thāne, Maharashtra, India beBeeAzure Full time ₹ 10,00,000 - ₹ 15,00,000

    Job OpportunityThreats to businesses and organizations are a growing concern in the digital age.A proactive approach to cybersecurity involves continuous monitoring and testing of digital environments to identify vulnerabilities.Job DescriptionSeeking an experienced Freelance Azure Data Factory Specialist to support data integration and ETL needs. The ideal...

  • Data Pipeline Lead

    4 days ago


    Thāne, Maharashtra, India beBeeDataEngineer Full time ₹ 1,80,00,000 - ₹ 2,20,00,000

    Job OverviewWe are seeking a seasoned data engineer to lead our team in designing and developing scalable data pipelines. This role requires expertise in cloud computing, data engineering, and ETL/ELT processes.Key ResponsibilitiesDesign, develop, and optimize complex data pipelines using Apache Spark, AWS Glue, and Azure Data Factory.Collaborate with...


  • Thāne, Maharashtra, India beBeeDataIngestion Full time ₹ 15,00,000 - ₹ 25,00,000

    Job Title: Data Ingestion SpecialistWe are seeking a seasoned professional to design, develop, and optimize data ingestion pipelines that integrate multiple data sources into Databricks.This role requires a strong understanding of data ingestion processes, focusing on integrating various data sources into Databricks for efficient workflow management.The...