Kafka Engineer

2 weeks ago

Hyderabad Telangana India, Telangana EXL Full time

Role - Kafka Engineer

Skills Required: Kafka, Java, SQL

Experience: 5-8 Years

Location: Hyderabad [3 Days a Week]


We are looking for a skilled Kafka Engineer with 5–7 years of experience in designing, building, and maintaining real-time streaming pipelines using Apache Kafka. The ideal candidate will have strong expertise in Java-based development, Kafka architecture, and CI/CD integration. You’ll play a key role in enabling real-time data processing and ensuring the scalability, fault tolerance, and governance of data streaming across our enterprise systems.


Key Responsibilities:

  • Design and maintain Kafka-based streaming pipelines for high-throughput, low-latency real-time data processing.
  • Ensure scalability, fault tolerance, and schema governance using tools like Schema Registry (Avro/Protobuf).
  • Develop robust Kafka producers and consumers using Java, adhering to performance and reliability standards.
  • Collaborate with DevOps teams to automate CI/CD pipelines, deployment, and infrastructure provisioning for Kafka components.
  • Implement security best practices including encryption, authentication, and access control for Kafka clusters.
  • Monitor and troubleshoot Kafka cluster health, latency, throughput, and message delivery guarantees.
  • Document architecture, deployment strategies, and operational runbooks for Kafka-based systems.
  • Participate in code reviews, knowledge-sharing sessions, and architectural discussions.


Must-Have Skills:

  • 5–7 years of experience in backend or data engineering, with strong focus on Apache Kafka.
  • Proficiency in Java for building Kafka producers/consumers and integration with backend services.
  • Deep understanding of Kafka internals (brokers, partitions, consumer groups, offsets, replication, etc.).
  • Experience with schema management using Confluent Schema Registry, Avro, or Protobuf.
  • Experience with CI/CD pipelines for Kafka component deployment and integration.
  • Strong collaboration skills with DevOps teams to manage Kafka infrastructure, security, and automation.
  • Solid understanding of real-time data processing concepts, fault tolerance, and data consistency.


Good-to-Have Skills:

  • Experience with Kafka Streams, ksqlDB, or Apache Flink for stream processing.
  • Familiarity with Kafka Connect and connectors for data ingestion and export.
  • Exposure to Confluent Kafka Platform and Kafka REST Proxy.
  • Experience with monitoring and observability tools (e.g., Grafana, ELK Stack).
  • Understanding of container orchestration platforms like Kubernetes or OpenShift.
  • Knowledge of cloud-native Kafka deployments (AWS MSK).
  • Familiarity with Spring Boot for integrating Kafka in microservice environments

  • Kafka Admin

    1 week ago

    Hyderabad, Telangana, India Apptad Full time ₹ 10,80,000 - ₹ 16,65,000 per year

    Job Description:•Standing up and administer on premise Kafka cluster.•Ability to architect and create reference architecture for kafka Implementation standards• Provide expertise in Kafka brokers, zookeepers, Kafka connect, schema registry, KSQL, Rest proxy and Kafka Control center.• Ensure optimum performance, high availability and stability of...

  • Senior Data Engineer

    2 weeks ago

    Hyderabad, Telangana, India, Telangana Straive Full time

    Job SummaryWe're looking for a Senior Data Engineer with 5-8 years of experience to build and maintain scalable, production-grade data pipelines. The ideal candidate is a strong software engineer with hands-on experience in Spark (3.x), Scala, SQL, and Python. You'll be responsible for designing and implementing ETL/ELT solutions, collaborating with teams to...

  • Data Engineer

    2 weeks ago

    Hyderabad, Telangana, India, Telangana HCLTech Full time

    Data Engineer - Databricks Job DescriptionThe Data Engineer - Databricks will be responsible for building and optimizing our data pipelines, architectures, and data sets. He will work closely with data scientists, analysts, and other engineers to support their data needs and maximize the value of our data processing capabilities.Responsibilities· Design,...

  • Senior Data Engineer

    2 weeks ago

    Hyderabad, Telangana, India, Telangana Adaptive Technology Insights Full time

    About the Role:We are looking for an experienced Data Engineer with strong expertise in Google Cloud Platform (GCP) and BigQuery to join our growing data team. You will be responsible for designing, building, and optimizing data pipelines and architecture to support analytics, reporting, and data-driven decision-making across the organization.Key...

  • Lead Data Engineer GCP

    2 weeks ago

    Hyderabad, Telangana, India, Telangana Egen Full time

    Job Overview: We are looking for a skilled and motivated Lead Data Engineer with strong experience in Python programming and Google Cloud Platform (GCP) to join our data engineering team. The ideal candidate will be responsible for requirements gathering, designing, architecting the solution, developing, and maintaining robust and scalable ETL (Extract,...

  • Databricks Engineer

    2 weeks ago

    Hyderabad, Telangana, India, Telangana iDWTeam Full time

    We are seeking an experienced Databricks Engineer to join our data team. The ideal candidate will have strong expertise in Databricks, Spark, and cloud platforms (Azure/AWS/GCP), along with a solid understanding of data engineering best practices. You will work closely with cross-functional teams to build scalable data pipelines, optimize big-data workloads,...

  • Data Engineer

    2 weeks ago

    Hyderabad, Telangana, India, Telangana Straive Full time

    Key Responsibilities:· Design, build and maintain scalable batch and (optionally) streaming data pipelines using Apache Spark 3.x, Scala, Python & SQL· Implement core ETL/ELT logic in Scala and Python; author efficient Spark DataFrame/Dataset jobs.· Write and optimize complex SQL for ingestion, transformation and consumption layers.· Tune Spark jobs for...

  • Senior Data Engineer

    2 weeks ago

    Hyderabad, Telangana, India, Telangana EXL Full time

    Job Summary:We are seeking an experienced Data Engineer with strong expertise in AWS cloud services and Databricks to design and implement scalable, high-performance data pipelines. Key Responsibilities:Design, develop, and maintain data pipelines and workflows using AWS (S3, Redshift, Glue, EMR, Lambda, Kinesis, Athena) and Databricks.Leverage Databricks...

  • Data Engineer

    1 week ago

    Hyderabad, Telangana, India, Telangana Kanerika Inc Full time

    About the RoleWe are seeking a skilled Data Engineer with strong experience in Databricks, Microsoft Fabric, or Snowflake, along with Power BI expertise, to design, build, and optimize scalable data pipelines and analytics solutions. You will also play a key role in enabling effective data visualization and reporting for business stakeholders.Key...

  • Back End Developer

    2 weeks ago

    Hyderabad, Telangana, India, Telangana Tata Consultancy Services Full time

    .Responsibilities:*Design, develop, and maintain data pipelines using Spark, Python, Scala, and Java.*Write efficient and optimized SQL queries for data extraction, transformation, and loading (ETL) processes.*Work with Data Frames to manipulate and analyse large datasets.*Implement data storage and processing solutions using cloud technologies (preferably...