Kafka /Streaming Data Engineer

2 weeks ago

Hyderabad Visakhapatnam Warangal, India Nexure Tech Full time ₹ 12,00,000 - ₹ 36,00,000 per year

Job Description Kafka/Streaming Data Engineer

Overview

We are seeking a Kafka/Streaming Data Engineer to join our team and support a new high-volume data ingestion and fulfillment architecture. The role requires strong hands-on expertise in Kafka cluster management, monitoring, and performance tuning, along with the ability to troubleshoot issues, propose solutions, and contribute to automation efforts.

Our current architecture based on a custom solution leveraging Ab Initio for CDC, Kafka for event streaming, and Apache Flink for downstream fulfillment into databases. The production environment is being built, and we need a Kafka expert to help fine-tune, monitor, and stabilize the ecosystem as it goes live.

Responsibilities

Manage and fine-tune Kafka clusters to ensure high throughput, scalability, and reliability.

Strong Ansible scripting skills for automating Kafka installation, configuration, and deployment. Build a production-ready, reliable, and scalable solution for replicating data from on-premise to cloud AWS

Monitor end-to-end data pipelines, including Ab Initio CDC jobs, Kafka topics, and Flink consumers.

Capture and report key metrics (e.g., records published to Kafka, records consumed by Flink, lag, latency, throughput).

Configure, optimize, and maintain Prometheus and Grafana dashboards for observability and alerting.

Proactively troubleshoot production issues and provide alternative solutions, working closely with fulfillment and development teams.

Participate in UAT and production readiness activities, including synthetic load testing and volume simulations.

Drive automation for repetitive operational tasks and cluster tuning activities.

Requirements

Strong hands-on experience with Apache Kafka (cluster management, partitions, replication, consumer groups, retention policies, monitoring).

Solid background in streaming data pipelines and real-time processing.

Experience with monitoring tools (Prometheus, Grafana) and alerting integrations.

Knowledge of Apache Flink, or equivalent ETL/streaming frameworks.

Familiarity with relational databases (Postgres) and CDC techniques.

Proven ability to troubleshoot, optimize, and fine-tune large-scale data systems.

Experience working in high-volume, production-grade environments (multi-billion row datasets preferred).

Strong problem-solving mindset: ability to propose workarounds and long-term fixes.

Excellent communication skills for collaborating across development, UAT, and fulfillment teams.

  • Data Engineer

    4 weeks ago

    Hyderabad, India CORTEX Consultants Full time

    Job Title : Data Engineer Kafka, Python, and Azure DatabricksLocation : Bangalore, Noida, Hyderabad (Hybrid 2 days in office per week)Experience : 5 to 8 YearsNotice Period : Immediate to 15 Days (Immediate Joiners Preferred)About the Role :We're looking for a seasoned Data Engineer to join our dynamic healthcare data engineering team. You'll be...

  • Senior Data Engineer

    3 weeks ago

    Hyderabad, India Zensar Technologies Full time

    Position : Senior Data Engineer (ETL/Big Data/Streaming)Experience : 810 yearsLocation : Onsite - Hyderabad Job Overview :We are looking for an experienced Senior Data Engineer with strong expertise in ETL/ELT, big data, and real-time data streaming to join our team.The ideal candidate will have deep hands-on experience with Apache Flink, Confluent Kafka,...

  • Kafka Developer

    3 weeks ago

    Hyderabad, India EXL Full time

    Location: Hyderabad (Hybrid) Job Summary: We are looking for a skilled Kafka Engineer with 57 years of experience in designing, building, and maintaining real-time streaming pipelines using Apache Kafka. The ideal candidate will have strong expertise in Java-based development, Kafka architecture, and CI/CD integration. You'll play a key role in enabling...

  • Kafka Developer

    2 days ago

    Hyderabad, Telangana, India EXL Full time ₹ 20,00,000 - ₹ 25,00,000 per year

    Location: Hyderabad [Hybrid]Job Summary:We are looking for a skilled Kafka Engineer with 57 years of experience in designing, building, and maintaining real-time streaming pipelines using Apache Kafka. The ideal candidate will have strong expertise in Java-based development, Kafka architecture, and CI/CD integration. You'll play a key role in enabling...

  • Java, Spring boot, spark,Kafka Streaming

    2 days ago

    Hyderabad, Telangana, India Talent21 Management Shared Services Pvt. ltd. Full time ₹ 6,00,000 - ₹ 12,00,000 per year

    Developing scalable data pipelines and streaming applications.Working with real-time data streams (Kafka).Processing large datasets using Spark.Building microservices or backend services in Java with Spring Boot.Ensuring data processing is efficient and reliable.

  • Kafka Engineer

    4 weeks ago

    Hyderabad, India Techno Wise Full time

    Title : Kafka EngineerWe are looking for a skilled Kafka Engineer with 5-7 years of experience in designing, building, and maintaining real-time streaming pipelines using Apache Kafka. The ideal candidate will have strong expertise in Java-based development, Kafka architecture, and CI/CD integration. You'll play a key role in enabling real-time data...

  • Python, Kafka

    6 days ago

    Hyderabad, Telangana, India Cognizant Technology Solutions Full time ₹ 20,00,000 - ₹ 25,00,000 per year

    Job SummaryWe are seeking a highly skilled Sr. Developer with 7 to 10 years of experience to join our team in the Asset & Wealth Management domain. The ideal candidate will have expertise in Spark Streaming Kafka Kafka Connect and Python. This role is hybrid and requires working during the day shift. The candidate will play a crucial role in developing and...

  • Kafka

    6 days ago

    Hyderabad, Telangana, India Cognizant Full time ₹ 20,00,000 - ₹ 25,00,000 per year

    Job SummaryAs a Technical Lead you will be responsible for driving the development and implementation of cutting-edge solutions using Spring Boot AWS Snowflake Apache Flink Kafka Connect and Kafka. You will collaborate with cross-functional teams to deliver high-quality software solutions that align with business objectives. This role requires a deep...

  • Data Streaming Specialist

    2 days ago

    Hyderabad, Telangana, India HIKING IT Full time ₹ 12,00,000 - ₹ 36,00,000 per year

    Hiring Hiring We have requirement on TIBCO Streaming (CEP). If your interested please check this below jdJob Type: Contract positionExperience: 8+ yearsNotice period: Immediate joinersSkill Set: As mentioned belowJob Description:Over 10 years of experience in real-time data streaming, analytics, and solution architecture, delivering data-driven insights and...

  • Python, Kafka

    6 days ago

    Hyderabad, Telangana, India Cognizant Full time

    **Job Summary** **Responsibilities** - Implement and manage Kafka and Kafka Connect to facilitate seamless data integration and streaming. - Write efficient and scalable Python code to support data processing and analytics tasks. - Collaborate with cross-functional teams to gather requirements and design solutions that meet business needs. - Ensure the...