Building Scalable Data Pipelines

1 day ago

Bengaluru, Karnataka, India beBeeKafkaEngineer Full time ₹ 15,00,000 - ₹ 25,00,000
Job Description

We are seeking an experienced Kafka Engineer to join our team. In this role, you will design, develop, and implement scalable Kafka-based data pipelines, producers, consumers, and stream processing using Kafka Streams/KSQL.

As a key member of the engineering team, you will collaborate directly with business teams to gather and analyze technical requirements and translate business needs into scalable Kafka-based architectures and data streaming workflows.

Key Responsibilities:

  • Work closely with business stakeholders to understand project requirements and deliver high-quality solutions.
  • Design and develop Kafka-based data pipelines, producers, consumers, and stream processing using Kafka Streams/KSQL.
  • Integrate Kafka with microservices, databases, and other systems using Kafka Connect and custom connectors.
  • Collaborate with cross-functional teams to ensure code quality and delivery timelines are met.
  • Support the system through UAT, go-live, and hyper-care phases, ensuring stability and reliability.

Required Skills and Qualifications

  • Chained Kafka Transactions: Ensures atomic writes to multiple topics using Kafka transactions.
  • Stateful Joins: Joins that maintain state across time or keys & Stateless Operations.
  • Kafka Streaming & KTable.
  • Kafka Integration.

Benefits

  • Opportunity to work on cutting-edge technology and contribute to the development of a robust and scalable data pipeline.
  • Collaborative and dynamic work environment.
  • Professional growth and development opportunities.

Others

  • Act as a mentor and technical reviewer for team members, ensuring code quality and delivery timelines are met.
  • Continuously evaluate and improve processes and procedures to ensure efficiency and effectiveness.

  • Data Engineer

    1 day ago

    Bengaluru, Karnataka, India beBeeDataEngineer Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

    We are seeking a skilled Data Engineer to design, develop, and maintain scalable data pipelines using Python and PySpark.Maintain high-quality data solutions using Spark, Hive, Nifi, Flink, and Spark Streaming.Built scalable Big Data solutions in industry projects.Work with streaming services like Kinesis, Kafka, or equivalent to support near real-time data...

  • Building Scalable Data Solutions

    3 days ago

    Bengaluru, Karnataka, India beBeeDataEngineer Full time US$ 98,240 - US$ 1,54,020

    Job Title: Senior Data EngineerWe are seeking a highly skilled and experienced Senior Data Engineer to join our team. As a key member of our data engineering group, you will play a critical role in designing, developing, and maintaining scalable data pipelines and data infrastructure.**Key Responsibilities:**Design and build scalable data pipelines that...

  • Principal Architect for Scalable Data Pipelines

    3 days ago

    Bengaluru, Karnataka, India beBeeDataEngineer Full time

    Job Opening: Senior Data EngineerA seasoned Senior Data Engineer is required to lead the design and development of scalable data pipelines using Talend, orchestrating complex ELT workflows into Snowflake.The ideal candidate will have expertise in Studio, TAC, and AWS services (S3, EC2, RDS, and No SQL) and a deep understanding of SQL skills and performance...

  • Senior Data Scientist

    23 hours ago

    Bengaluru, Karnataka, India beBeeData Full time ₹ 2,00,00,000 - ₹ 2,50,00,000

    Senior Data Scientist - Scalable Data PipelinesJob Title: Senior Data ScientistRole Overview:We are seeking a highly skilled and experienced Senior Data Scientist to lead the design, development, and maintenance of scalable data pipelines that process and transform large volumes of structured and unstructured data. The ideal candidate will have a strong...

  • Senior Scalable Data Pipeline Specialist

    7 days ago

    Bengaluru, Karnataka, India beBeeDataEngineer Full time ₹ 9,00,000 - ₹ 12,00,000

    Big Data Engineer PositionWe are seeking a highly skilled Big Data Engineer to join our team. The successful candidate will be responsible for designing, developing, and maintaining scalable data pipelines using Spark.

  • Data Engineer Position

    3 days ago

    Bengaluru, Karnataka, India beBeeDataEngineering Full time ₹ 1,50,00,000 - ₹ 2,00,00,000

    Data Engineering OpportunityWe are seeking a skilled and motivated Data Engineer to design, develop, and maintain scalable data pipelines that process and transform large volumes of structured and unstructured data.This is an exciting opportunity to work with our team at the forefront of data engineering. If you have a strong background in data engineering,...

  • Building Scalable Data Infrastructure

    1 day ago

    Bengaluru, Karnataka, India beBeeDataEngineering Full time ₹ 18,00,000 - ₹ 25,00,000

    Senior Data Engineering SpecialistWe are seeking a highly experienced Senior Data Engineering Specialist to lead our data warehousing efforts.About the Role:Design and develop scalable data pipelines using Snowflake to support business growth.Oversight of Snowflake data warehouse performance, ensuring optimal execution of complex queries.Collaboration with...

  • Building Scalable Data Solutions

    4 days ago

    Bengaluru, Karnataka, India beBeeData Full time US$ 1,05,000 - US$ 1,65,000

    Job Title: Data ArchitectWe are seeking a highly skilled data architect to design and implement scalable data solutions.Responsibilities:Create efficient data pipelines using Snowflake and AWS GlueDevelop and maintain robust data warehouses using Snowpipe, Time Travel, Streams & TasksCollaborate with cross-functional teams to integrate data from various...

  • Scalable Data Architect

    6 days ago

    Bengaluru, Karnataka, India beBeeDataEngineer Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

    Job Title: Data EngineerWe are seeking a highly skilled data engineer with hands-on experience in designing, building, and optimizing scalable data pipelines.Design and Development: Develop ETL/ELT pipelines using AWS Glue, Lambda, and DBT. Optimize data models in Snowflake for scalability and performance.Data Integration and Transformation: Write efficient...

  • Building Scalable AI Systems

    6 days ago

    Bengaluru, Karnataka, India beBeeDataScience Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

    Machine Learning EngineerAt Ag-EmployVision Pvt. Ltd., we are looking for a Machine Learning & Data Science Engineer to join our team.This is a VC funded startup at the intersection of AI & storytelling, where we are building creator-first tools that turn ideas into production-ready video.We are looking for an engineer with strong foundations in data...