Big Data Pipeline Developer

7 days ago

Malappuram, Kerala, India beBeeDataEngineering Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

Job Title: Data Engineering Specialist

Location: Bangalore, Chennai, Gurgaon, Pune, Mumbai.

Experience: 7–10 Years (Minimum 3+ years in Scala)

Notice Period: Immediate to 30 Days

Mode of Work: Hybrid

Role Overview

We are seeking a highly skilled Data Engineering Specialist with strong expertise in Scala, Spark, Python, NiFi, and Apache Kafka to join our data engineering team. The ideal candidate will have a proven track record in building, scaling, and optimizing big data pipelines, and hands-on experience in distributed data systems and cloud-based solutions.

Key Responsibilities

  • Design, develop, and optimize large-scale data pipelines and distributed data processing systems.
  • Work extensively with Scala, Spark (PySpark), and Python for data processing and transformation.
  • Develop and integrate streaming solutions using Apache Kafka and orchestration tools like NiFi / Airflow.
  • Write efficient queries and perform data analysis using Jupyter Notebooks and SQL.
  • Collaborate with cross-functional teams to design scalable cloud-based data architectures.
  • Ensure delivery of high-quality code through code reviews, performance tuning, and best practices.
  • Build monitoring and alerting systems leveraging Splunk or equivalent tools.
  • Participate in CI/CD workflows using Git, Jenkins, and other DevOps tools.
  • Contribute to product development with a focus on scalability, maintainability, and performance.

Mandatory Skills

  • Scala – Minimum 3+ years of hands-on experience.
  • Strong expertise in Spark (PySpark) and Python.
  • Hands-on experience with Apache Kafka.
  • Knowledge of NiFi / Airflow for orchestration.
  • Strong experience in Distributed Data Systems (5+ years).
  • Proficiency in SQL and query optimization.
  • Good understanding of Cloud Architecture.

Preferred Skills

  • Exposure to messaging technologies like Apache Kafka or equivalent.
  • Experience in designing intuitive, responsive UIs for data analytics visualization.
  • Familiarity with Splunk or other monitoring/alerting solutions.
  • Hands-on experience with CI/CD tools (Git, Jenkins).
  • Strong grasp of software engineering concepts, data modeling, and optimization techniques.

  • Big Data Engineering Leader

    1 week ago

    Malappuram, Kerala, India beBeeDataEngineer Full time ₹ 24,00,000 - ₹ 36,00,000

    Job OverviewWe are seeking a skilled Big Data Engineering leader to join our team.The successful candidate will be responsible for designing, implementing, and maintaining large-scale big data systems. This includes leading the data platform and pipeline along with risk evaluation, providing technical leadership on data platform design, rollout, and...

  • Data Pipeline Architect

    2 weeks ago

    Malappuram, Kerala, India beBeeDataEngineer Full time ₹ 15,00,000 - ₹ 20,25,000

    Data Engineer PositionWe are seeking a skilled professional to design and implement scalable data pipelines using modern cloud and big data platforms. Key Responsibilities: Design, develop, and maintain large-scale data processing systems Implement efficient data ingestion, storage, and retrieval solutions Collaborate with cross-functional teams to ensure...

  • Big Data Specialist

    1 week ago

    Malappuram, Kerala, India beBeeData Full time ₹ 2,00,00,000 - ₹ 2,50,00,000

    Reimagining Data Infrastructure for Enhanced PerformanceWe are seeking a seasoned Data Engineer to join our team and contribute to the development of high-performance data pipelines that process massive amounts of data. The ideal candidate will have experience in designing, building, and maintaining scalable batch processing systems using Python and big data...

  • Chief Big Data Architect

    1 week ago

    Malappuram, Kerala, India beBeeBigDataEngineer Full time ₹ 20,00,000 - ₹ 25,00,000

    Job Overview:A high-performing Big Data Engineering Leader oversees the development of scalable data solutions, collaborating closely with cross-functional teams to drive business outcomes.Key Responsibilities:Design and implement complex data pipelines using batch processing and real-time streaming technologies.Collaborate with stakeholders to define data...

  • Big Data Architect

    2 weeks ago

    Malappuram, Kerala, India beBeeDataEngineer Full time ₹ 15,00,000 - ₹ 25,00,000

    Job DescriptionWe are seeking an experienced Data Engineer to join our team in Malappuram, India.The successful candidate will be responsible for designing, developing, and deploying scalable data pipelines and ETL processes using Databricks.About the RoleAs a Data Engineer, you will collaborate with stakeholders to understand data requirements and translate...

  • Big Data Solutions Expert

    7 days ago

    Malappuram, Kerala, India beBeeData Full time ₹ 1,00,00,000 - ₹ 2,00,00,000

    Job OverviewWe are seeking a skilled Senior Big Data Engineer to join our team.The ideal candidate will have experience in designing and implementing big data analytics platforms using Spark, Kafka, Glue, EMR, Hudi, Schema Registry, and Data Lineage.In addition to technical expertise, the successful candidate will possess excellent problem-solving skills,...

  • Big Data Engineer

    1 week ago

    Malappuram, Kerala, India beBeeDataScientist Full time ₹ 20,00,000 - ₹ 30,00,000

    About Big Data Development RoleKey responsibilities for this position include performing hands-on big data development work, collaborating with junior engineers, reviewing code, and assisting in designing solutions using ETL processes.This involves working with Hadoop, Hive, HDFS, Apache Spark, and Kafka to integrate data from various sources. Strong...

  • Big Data Specialist

    2 weeks ago

    Malappuram, Kerala, India beBeeData Full time ₹ 15,00,000 - ₹ 20,00,000

    Job Title:Big Data SpecialistJob Description:We are seeking a highly skilled Big Data Specialist to lead the design, development, and deployment of large-scale data systems. The ideal candidate will have expertise in Python and cloud technologies, preferably Azure.Responsibilities:Design, develop, and deploy big data systems to drive business growth and...

  • Big Data Solutions Architect

    2 weeks ago

    Malappuram, Kerala, India beBeeData Full time ₹ 20,00,000 - ₹ 25,00,000

    Job Opportunity:We are seeking an experienced Big Data Developer to drive data insights and inform business decisions.The ideal candidate will have a strong background in big data technologies, including Scala, AWS, and Python or Java.Key Responsibilities:Design and develop scalable big data architectures using Hadoop and Spark.Analyze large datasets to...

  • Expert Big Data Hadoop Scala Professional

    2 weeks ago

    Malappuram, Kerala, India beBeeData Full time ₹ 18,00,000 - ₹ 25,00,000

    Big Data Hadoop Scala EngineerWe're seeking a skilled Data Architect to design and implement large-scale data processing systems.Key Responsibilities:Develop and optimize data pipelines using Scala, Spark, and Hadoop tools.Design efficient database schemas for seamless data storage and retrieval.Work on performance enhancement and query...