Kafka /Streaming Data Engineer

2 weeks ago

Hyderabad Visakhapatnam Warangal, India Nexure Tech Full time ₹ 12,00,000 - ₹ 36,00,000 per year

Job Description Kafka/Streaming Data Engineer

Overview

We are seeking a Kafka/Streaming Data Engineer to join our team and support a new high-volume data ingestion and fulfillment architecture. The role requires strong hands-on expertise in Kafka cluster management, monitoring, and performance tuning, along with the ability to troubleshoot issues, propose solutions, and contribute to automation efforts.

Our current architecture based on a custom solution leveraging Ab Initio for CDC, Kafka for event streaming, and Apache Flink for downstream fulfillment into databases. The production environment is being built, and we need a Kafka expert to help fine-tune, monitor, and stabilize the ecosystem as it goes live.

Responsibilities

Manage and fine-tune Kafka clusters to ensure high throughput, scalability, and reliability.

Strong Ansible scripting skills for automating Kafka installation, configuration, and deployment. Build a production-ready, reliable, and scalable solution for replicating data from on-premise to cloud AWS

Monitor end-to-end data pipelines, including Ab Initio CDC jobs, Kafka topics, and Flink consumers.

Capture and report key metrics (e.g., records published to Kafka, records consumed by Flink, lag, latency, throughput).

Configure, optimize, and maintain Prometheus and Grafana dashboards for observability and alerting.

Proactively troubleshoot production issues and provide alternative solutions, working closely with fulfillment and development teams.

Participate in UAT and production readiness activities, including synthetic load testing and volume simulations.

Drive automation for repetitive operational tasks and cluster tuning activities.

Requirements

Strong hands-on experience with Apache Kafka (cluster management, partitions, replication, consumer groups, retention policies, monitoring).

Solid background in streaming data pipelines and real-time processing.

Experience with monitoring tools (Prometheus, Grafana) and alerting integrations.

Knowledge of Apache Flink, or equivalent ETL/streaming frameworks.

Familiarity with relational databases (Postgres) and CDC techniques.

Proven ability to troubleshoot, optimize, and fine-tune large-scale data systems.

Experience working in high-volume, production-grade environments (multi-billion row datasets preferred).

Strong problem-solving mindset: ability to propose workarounds and long-term fixes.

Excellent communication skills for collaborating across development, UAT, and fulfillment teams.

  • Kafka Engineer

    3 days ago

    Hyderabad, India EXL Full time

    Role - Kafka Engineer Skills Required: Kafka, Java, SQL Experience: 5-8 Years Location: Hyderabad [3 Days a Week] We are looking for a skilled Kafka Engineer with 5–7 years of experience in designing, building, and maintaining real-time streaming pipelines using Apache Kafka. The ideal candidate will have strong expertise in Java-based development, Kafka...

  • Kafka Engineer

    3 days ago

    hyderabad, India EXL Full time

    Role - Kafka Engineer Skills Required: Kafka, Java, SQL Experience: 5-8 Years Location: Hyderabad [3 Days a Week] We are looking for a skilled Kafka Engineer with 5–7 years of experience in designing, building, and maintaining real-time streaming pipelines using Apache Kafka. The ideal candidate will have strong expertise in Java-based development, Kafka...

  • Kafka Engineer

    2 days ago

    Hyderabad, India EXL Full time

    Role - Kafka Engineer Skills Required : Kafka, Java, SQL Experience : 5-8 Years Location : Hyderabad (3 Days a Week) We are looking for a skilled Kafka Engineer with 5–7 years of experience in designing, building, and maintaining real-time streaming pipelines using Apache Kafka. The ideal candidate will have strong expertise in Java-based development ,...

  • Kafka Engineer

    2 days ago

    Hyderabad, India EXL Full time

    Role - Kafka EngineerSkills Required: Kafka, Java, SQLExperience: 5-8 YearsLocation: Hyderabad (3 Days a Week)We are looking for a skilled Kafka Engineer with 5–7 years of experience in designing, building, and maintaining real-time streaming pipelines using Apache Kafka. The ideal candidate will have strong expertise in Java-based development, Kafka...

  • Kafka Engineer

    21 hours ago

    Hyderabad, India EXL Full time

    Role - Kafka EngineerSkills Required: Kafka, Java, SQLExperience: 5-8 YearsLocation: Hyderabad (3 Days a Week)We are looking for a skilled Kafka Engineer with 5–7 years of experience in designing, building, and maintaining real-time streaming pipelines using Apache Kafka. The ideal candidate will have strong expertise in Java-based development, Kafka...

  • Senior Data Engineer

    4 weeks ago

    Hyderabad, India Zensar Technologies Full time

    Position : Senior Data Engineer (ETL/Big Data/Streaming)Experience : 810 yearsLocation : Onsite - Hyderabad Job Overview :We are looking for an experienced Senior Data Engineer with strong expertise in ETL/ELT, big data, and real-time data streaming to join our team.The ideal candidate will have deep hands-on experience with Apache Flink, Confluent Kafka,...

  • Kafka Data Engineer

    4 hours ago

    Bengaluru, Chennai, Hyderabad, India Virtusa Full time ₹ 5,00,000 - ₹ 25,00,000 per year

    Data Engineer to build and manage data pipelines that support batch and streaming data solutions. The role requires expertise in creating seamless data flows across platforms like Data Lake/Lakehouse in Cloudera, Azure Databricks, Kafka for both batch and stream data pipelines etc.Responsibilities:Strong experience in develop, test, and maintain data...

  • Java, Spring boot, spark,Kafka Streaming

    2 weeks ago

    hyderabad, India Talent21 Management Shared Services Pvt. ltd. Full time

    Developing scalable data pipelines and streaming applications.Working with real-time data streams (Kafka).Processing large datasets using Spark.Building microservices or backend services in Java with Spring Boot.Ensuring data processing is efficient and reliable.

  • Java, Spring boot, spark,Kafka Streaming

    2 days ago

    Hyderabad, Telangana, India Talent21 Management Shared Services Pvt. ltd. Full time ₹ 6,00,000 - ₹ 12,00,000 per year

    Developing scalable data pipelines and streaming applications.Working with real-time data streams (Kafka).Processing large datasets using Spark.Building microservices or backend services in Java with Spring Boot.Ensuring data processing is efficient and reliable.

  • Python, Kafka

    3 days ago

    Hyderabad, Telangana, India Cognizant Technology Solutions Full time ₹ 20,00,000 - ₹ 25,00,000 per year

    Job SummaryWe are seeking a highly skilled Sr. Developer with 7 to 10 years of experience to join our team in the Asset & Wealth Management domain. The ideal candidate will have expertise in Spark Streaming Kafka Kafka Connect and Python. This role is hybrid and requires working during the day shift. The candidate will play a crucial role in developing and...