Kafka /Streaming Data Engineer

2 weeks ago

Hyderabad Visakhapatnam Warangal, India Nexure Tech Full time ₹ 12,00,000 - ₹ 36,00,000 per year

Job Description Kafka/Streaming Data Engineer

Overview

We are seeking a Kafka/Streaming Data Engineer to join our team and support a new high-volume data ingestion and fulfillment architecture. The role requires strong hands-on expertise in Kafka cluster management, monitoring, and performance tuning, along with the ability to troubleshoot issues, propose solutions, and contribute to automation efforts.

Our current architecture based on a custom solution leveraging Ab Initio for CDC, Kafka for event streaming, and Apache Flink for downstream fulfillment into databases. The production environment is being built, and we need a Kafka expert to help fine-tune, monitor, and stabilize the ecosystem as it goes live.

Responsibilities

Manage and fine-tune Kafka clusters to ensure high throughput, scalability, and reliability.

Strong Ansible scripting skills for automating Kafka installation, configuration, and deployment. Build a production-ready, reliable, and scalable solution for replicating data from on-premise to cloud AWS

Monitor end-to-end data pipelines, including Ab Initio CDC jobs, Kafka topics, and Flink consumers.

Capture and report key metrics (e.g., records published to Kafka, records consumed by Flink, lag, latency, throughput).

Configure, optimize, and maintain Prometheus and Grafana dashboards for observability and alerting.

Proactively troubleshoot production issues and provide alternative solutions, working closely with fulfillment and development teams.

Participate in UAT and production readiness activities, including synthetic load testing and volume simulations.

Drive automation for repetitive operational tasks and cluster tuning activities.

Requirements

Strong hands-on experience with Apache Kafka (cluster management, partitions, replication, consumer groups, retention policies, monitoring).

Solid background in streaming data pipelines and real-time processing.

Experience with monitoring tools (Prometheus, Grafana) and alerting integrations.

Knowledge of Apache Flink, or equivalent ETL/streaming frameworks.

Familiarity with relational databases (Postgres) and CDC techniques.

Proven ability to troubleshoot, optimize, and fine-tune large-scale data systems.

Experience working in high-volume, production-grade environments (multi-billion row datasets preferred).

Strong problem-solving mindset: ability to propose workarounds and long-term fixes.

Excellent communication skills for collaborating across development, UAT, and fulfillment teams.

  • Senior Data Engineer

    3 weeks ago

    Hyderabad, India SMARTWORK IT SERVICES Full time

    Position : Senior Data Engineer (ETL/Big Data/Streaming)Experience : 810 yearsLocation : Onsite Hybderabd / ChennaiMode of Interview : F2F Face to Face InterviewJob Overview : We are looking for an experienced Senior Data Engineer with strong expertise in ETL/ELT, big data, and real-time data streaming to join our team.The ideal candidate will have deep...

  • Senior Data Engineer

    7 days ago

    Hyderabad, India Zensar Technologies Full time

    Position : Senior Data Engineer (ETL/Big Data/Streaming)Experience : 810 yearsLocation : Onsite - Hyderabad Job Overview :We are looking for an experienced Senior Data Engineer with strong expertise in ETL/ELT, big data, and real-time data streaming to join our team.The ideal candidate will have deep hands-on experience with Apache Flink, Confluent Kafka,...

  • Data Engineer

    1 week ago

    Hyderabad, India CORTEX Consultants Full time

    Job Title : Data Engineer Kafka, Python, and Azure DatabricksLocation : Bangalore, Noida, Hyderabad (Hybrid 2 days in office per week)Experience : 5 to 8 YearsNotice Period : Immediate to 15 Days (Immediate Joiners Preferred)About the Role :We're looking for a seasoned Data Engineer to join our dynamic healthcare data engineering team. You'll be...

  • Kafka engineer

    1 week ago

    Hyderabad, India Fusion Plus Solutions Full time

    Job Description Kafka Environment Expertise: - Must have hands-on experience in Confluent or Apache Kafka environment, including managing Kafka Clusters and Apache Zookeeper. Data Streaming: - Capable of transforming data into Kafka topics and building streaming processors using Kafka. Confluent Tools: - Practical experience with Confluent-specific tools for...

  • Kafka engineer

    1 week ago

    Hyderabad, India Fusion Plus Solutions Full time

    Job Description - Must have hands-on experience in Confluent/Apache Kafka environment (Kafka Cluster, Apache Zookeeper). - Transform data into topics, streaming processors in Kafka, Confluent tools. - Knowledge on one of the cloud platforms like AWS or GCPMust have hands-on experience in Confluent/Apache Kafka environment (Kafka Cluster, Apache Zookeeper). -...

  • Kafka Developer

    19 hours ago

    Hyderabad, Telangana, India EXL Full time ₹ 20,00,000 - ₹ 25,00,000 per year

    Location: Hyderabad [Hybrid]Job Summary:We are looking for a skilled Kafka Engineer with 57 years of experience in designing, building, and maintaining real-time streaming pipelines using Apache Kafka. The ideal candidate will have strong expertise in Java-based development, Kafka architecture, and CI/CD integration. You'll play a key role in enabling...

  • Big Data Engineer

    1 week ago

    Hyderabad, India Teamware Solutions Full time

    Job Description Key Responsibilities: - Design and implement scalable, fault-tolerant data pipelines using Scala, Kafka, and other Big Data technologies. - Build real-time data streaming applications to process high-volume datasets from various sources. - Integrate and manage containerized applications using Docker and orchestrate deployments with...

  • Java, Spring boot, spark,Kafka Streaming

    22 hours ago

    Hyderabad, Telangana, India Talent21 Management Shared Services Pvt. ltd. Full time ₹ 6,00,000 - ₹ 12,00,000 per year

    Developing scalable data pipelines and streaming applications.Working with real-time data streams (Kafka).Processing large datasets using Spark.Building microservices or backend services in Java with Spring Boot.Ensuring data processing is efficient and reliable.

  • Kafka Developers

    3 weeks ago

    Hyderabad, India Tata Consultancy Services Full time

    Role- Kafka DevelopersExperiece-5+ yearsLocation- Hyderabad, Chennai, Bangalore, Kolkata, Delhi, Pune JD:SQL, Data Base Modeling., GCP, Data Streaming in GCP and Confluent Kafka ConnectorsFunctional Skills - Confluent Kafka Connectors, Data Streaming in GCPExperience in Confluent Kafka as Admin and Developer, having certification is a plusExperience in Kafka...

  • Kafka Developers

    2 weeks ago

    Hyderabad, India Tata Consultancy Services Full time

    Role- Kafka DevelopersExperiece-5+ yearsLocation- Hyderabad, Chennai, Bangalore, Kolkata, Delhi, Pune JD:SQL, Data Base Modeling., GCP, Data Streaming in GCP and Confluent Kafka ConnectorsFunctional Skills - Confluent Kafka Connectors, Data Streaming in GCPExperience in Confluent Kafka as Admin and Developer, having certification is a plusExperience in Kafka...