Current jobs related to Kafka lead platform engineer - hyderabad - SID Global Solutions

  • Kafka Streaming Platform Expert

    1 week ago

    Hyderabad, Telangana, India DSV Full time

    Hiring OverviewDSV is expanding its Hybrid Computing Platform department and seeking a skilled engineer to join our Event Streaming team. We are looking for a hands-on expert in Kafka streaming platform development, maintenance, and improvement.About the RoleThe successful candidate will be responsible for participating in development using Confluent Cloud,...

  • Kafka Engineer for Hybrid Computing Platform

    3 weeks ago

    Hyderabad, Telangana, India DSV Full time

    Event Streaming ExpertiseWe are seeking a skilled Kafka engineer to join our Hybrid Computing Platform department in Global IT. As a Site Reliability Engineer for Event Streaming Solutions, you will be responsible for developing, publishing, and maintaining infrastructure products.Key ResponsibilitiesParticipate in development using Confluent Cloud,...

  • Java-Kafka Lead

    4 weeks ago

    Hyderabad, Telangana, India Qode Full time

    Java-Kafka Lead Job DescriptionAt Qode, we're seeking a highly skilled Java-Kafka Lead to join our team. As a key member of our engineering team, you will be responsible for designing, developing, and deploying scalable and efficient Kafka-based systems.Key Responsibilities:Kafka Architecture and Design: Develop and implement robust Kafka architecture and...

  • Kafka Engineer

    4 weeks ago

    Hyderabad, Telangana, India DSV Full time

    About the RoleWe are seeking a highly skilled Kafka Engineer to join our Hybrid Computing Platform department in Global IT. As a key member of our Event Streaming team, you will be responsible for designing, developing, and maintaining our event streaming solutions using Confluent Cloud, Confluent Platform, and Apache Kafka.Key ResponsibilitiesParticipate in...

  • Hybrid Computing Platform Kafka Specialist

    2 weeks ago

    Hyderabad, Telangana, India DSV Full time

    Kafka Specialist & Site Reliability Engineer OpportunityWe are seeking a skilled Kafka Specialist & Site Reliability Engineer to join our Hybrid Computing Platform department in Global IT. As a member of our Event Streaming team, you will be responsible for designing, developing, and maintaining event streaming solutions using Confluent Cloud, Confluent...

  • Kafka Engineer

    1 day ago

    hyderabad, India Health Catalyst Full time

    About Us The healthcare industry is the next great frontier of opportunity for software development, and Health Catalyst is one of the most dynamic and influential companies in this space. We are working on solving national-level healthcare problems, and this is your chance to improve the lives of millions of people, including your family and friends....

  • Kafka Architect

    7 days ago

    Hyderabad, Telangana, India Alignity Solutions Full time

    About the RoleWe are looking for a highly skilled Kafka Architect to lead the design, development, and deployment of our Kafka-based data pipeline. As a Kafka Architect, you will be responsible for designing and implementing scalable, fault-tolerant, and high-performance Kafka clusters.Key Responsibilities:Kafka Cluster Design: Design and implement scalable...

  • Kafka Engineer

    6 days ago

    hyderabad, India Health Catalyst Full time

    About UsThe healthcare industry is the next great frontier of opportunity for software development, and Health Catalyst is one of the most dynamic and influential companies in this space. We are working on solving national-level healthcare problems, and this is your chance to improve the lives of millions of people, including your family and friends. Health...

  • Kafka Engineer

    4 weeks ago

    hyderabad, India Health Catalyst Full time

    About UsThe healthcare industry is the next great frontier of opportunity for software development, and Health Catalyst is one of the most dynamic and influential companies in this space. We are working on solving national-level healthcare problems, and this is your chance to improve the lives of millions of people, including your family and friends. Health...

  • Kafka Engineer

    4 weeks ago

    Hyderabad, Telangana, India DSV Full time

    Unlock the Power of Event StreamingAt DSV, we're shaping the future of hybrid computing platforms, and we need a talented Kafka expert to join our team. As a Site Reliability Engineer, you'll play a crucial role in ensuring the smooth operation of our event streaming solutions.Key Responsibilities:Design and implement scalable event streaming architectures...

  • Kafka Engineer

    1 week ago

    Hyderabad, India Health Catalyst Full time

    About UsThe healthcare industry is the next great frontier of opportunity for software development, and Health Catalyst is one of the most dynamic and influential companies in this space. We are working on solving national-level healthcare problems, and this is your chance to improve the lives of millions of people, including your family and friends. Health...

  • Kafka engineer

    1 week ago

    Hyderabad, India Health Catalyst Full time

    About UsThe healthcare industry is the next great frontier of opportunity for software development, and Health Catalyst is one of the most dynamic and influential companies in this space. We are working on solving national-level healthcare problems, and this is your chance to improve the lives of millions of people, including your family and friends. Health...

  • Kafka engineer

    4 weeks ago

    Hyderabad, India Health Catalyst Full time

    About Us The healthcare industry is the next great frontier of opportunity for software development, and Health Catalyst is one of the most dynamic and influential companies in this space. We are working on solving national-level healthcare problems, and this is your chance to improve the lives of millions of people, including your family and friends....

  • Kafka Engineer

    4 weeks ago

    Hyderabad, India Health Catalyst Full time

    About Us The healthcare industry is the next great frontier of opportunity for software development, and Health Catalyst is one of the most dynamic and influential companies in this space. We are working on solving national-level healthcare problems, and this is your chance to improve the lives of millions of people, including your family and friends....

  • Kafka Engineer

    3 weeks ago

    hyderabad, India Health Catalyst Full time

    About Us The healthcare industry is the next great frontier of opportunity for software development, and Health Catalyst is one of the most dynamic and influential companies in this space. We are working on solving national-level healthcare problems, and this is your chance to improve the lives of millions of people, including your family and friends....

  • Kafka Architect Lead

    1 week ago

    Hyderabad, Telangana, India Qode Full time

    Kafka Architect Lead PositionAt Qode, we are looking for a seasoned expert in designing and implementing scalable Kafka architectures. The ideal candidate will have extensive experience with Kafka Streams, KSQL, and the associated clustering model. Additionally, they should possess solid programming skills in Java and a deep understanding of best practices...

  • Kafka Specialist and Site Reliability Engineer for Hybrid Computing Platform

    3 weeks ago

    Hyderabad, Telangana, India DSV Full time

    About the RoleWe are seeking a highly skilled Kafka Specialist and Site Reliability Engineer to join our Hybrid Computing Platform department. As a key member of our team, you will be responsible for designing, implementing, and maintaining our event streaming solutions using Apache Kafka.Key ResponsibilitiesParticipate in development using Confluent Cloud,...

  • Kafka Developer Lead

    3 weeks ago

    Hyderabad, Telangana, India ProArch Full time

    Job Title: Sr. Kafka DeveloperAt ProArch, we are seeking a highly skilled and experienced Sr. Kafka Developer to join our team. As a key member of our engineering team, you will be responsible for designing, developing, and maintaining scalable and efficient Kafka-based messaging solutions and microservices applications.Key Responsibilities:Design and...

  • Kafka Architect

    4 weeks ago

    Hyderabad, Telangana, India Alignity Solutions Full time

    Job Title: Kafka SMEWe are seeking a highly skilled Kafka Subject Matter Expert (SME) to lead the architecture management and scaling of Kafka environments in 24x7 production settings. The ideal candidate will have a deep understanding of Apache Kafka, a proven track record of resolving complex production and non-production issues, and strong leadership...

  • Kafka Architect

    4 weeks ago

    Hyderabad, Telangana, India Alignity Solutions Full time

    Job Title: Kafka SMEWe are seeking a highly skilled Kafka Subject Matter Expert (SME) to lead the architecture, management, and scaling of Kafka environments in 24x7 production settings. The ideal candidate will have a deep understanding of Apache Kafka, a proven track record of resolving complex production and non-production issues, and strong leadership...

Kafka lead platform engineer

2 months ago

hyderabad, India SID Global Solutions Full time

Role : Kafka lead platform engineer

Location : Hyderabad

Work mode : on-site

Exp : 3-5 yrs


Daily Routine and Key Responsibilities:


1. Kafka Topic and Queue Management

  • Topic Creation and Management:
  • Create, configure, and manage Kafka topics based on the requirements from various teams.
  • Set appropriate partitions and replication factors to ensure data durability and scalability.
  • Monitor topic health, including under-replicated partitions, and take corrective actions as necessary.
  • Queue Configuration and Management:
  • Configure consumer groups and manage Kafka queues to ensure efficient data processing.
  • Tune consumer group settings, such as max.poll.records, fetch.min.bytes, and session.timeout.ms, to optimize throughput and latency.
  • Monitor and resolve issues related to consumer lag, ensuring that data is processed in a timely manner.

2. Configuration Management

  • Broker Configuration:
  • Adjust broker-level configurations (e.g., log.retention.hours, log.segment.bytes, zookeeper.connection.timeout.ms) to maintain optimal performance and stability.
  • Apply updates and patches to Kafka brokers and ensure configurations are in line with best practices.
  • Security Configuration:
  • Manage Kafka security settings, including SSL/TLS encryption, Kerberos, and SASL authentication.
  • Configure ACLs (Access Control Lists) to control access to topics and consumer groups, ensuring secure and compliant operations.
  • Schema Registry and Compatibility:
  • Work with the Confluent Schema Registry to enforce schema validation and compatibility.
  • Manage versioning of schemas and ensure backward and forward compatibility across different services.

3. Monitoring and Troubleshooting

  • Platform Monitoring:
  • Monitor Kafka clusters using tools like Prometheus, Grafana, and Confluent Control Center to track key metrics (e.g., broker CPU usage, disk I/O, topic throughput).
  • Set up and manage alerts for critical issues like broker failures, high consumer lag, and message backlog.
  • Issue Resolution:
  • Respond to incidents involving Kafka infrastructure, such as broker crashes, network partitions, or performance degradations.
  • Troubleshoot and resolve issues related to topic misconfigurations, consumer group errors, and producer failures.
  • Performance Tuning:
  • Continuously optimize Kafka performance by adjusting configurations based on observed behavior and workload patterns.
  • Conduct periodic health checks and capacity planning to anticipate and prevent potential bottlenecks.

4. Collaboration and Support

  • Inter-Team Collaboration:
  • Work closely with development, DevOps, and data engineering teams to understand their Kafka usage patterns and requirements.
  • Provide guidance on best practices for Kafka topic and consumer group configurations.
  • Support and Documentation:
  • Assist teams with onboarding to Kafka, including setting up topics, consumer groups, and security credentials.
  • Document Kafka configurations, operational procedures, and incident resolutions to ensure knowledge sharing and continuity.

5. Continuous Improvement

  • Automation:
  • Develop and maintain automation scripts for routine Kafka tasks, such as topic creation, scaling clusters, and performing rolling upgrades.
  • Integrate Kafka operations into CI/CD pipelines to streamline deployment and configuration changes.
  • Learning and Development:
  • Stay updated with the latest developments in Kafka and Confluent Platform, exploring new features and improvements.
  • Participate in knowledge-sharing sessions, both within the team and across the organization, to promote best practices in Kafka usage.


End-of-Day Review:

  • Status Review:
  • Review the day's activities, including topic and queue configurations, any incidents resolved, and performance optimizations made.
  • Update relevant stakeholders on Kafka platform status, ongoing tasks, and any issues that need attention.
  • Planning for Tomorrow:
  • Plan and prioritize tasks for the next day, ensuring that any pending requests or incidents are addressed promptly.


Tools and Technologies Used:

  • Kafka Management: Apache Kafka, Confluent Control Center
  • Monitoring and Alerts: Prometheus, Grafana, Elasticsearch, Kibana, Confluent Health+
  • Automation: Shell Scripting, Ansible, Jenkins, GitLab CI
  • Security: SSL/TLS, Kerberos, SASL, ACLs
  • Collaboration: Jira, Confluence, Slack


Qualifications:

  • Strong knowledge and hands-on experience with Apache Kafka and the Confluent Platform.
  • Proficiency in configuring and managing Kafka topics, queues, and consumer groups.
  • Experience with Kafka monitoring, troubleshooting, and performance tuning.
  • Understanding of Kafka security configurations and best practices.
  • Excellent problem-solving skills and the ability to work in a fast-paced environment.


  • Seniority Level
  • Mid-Senior level
  • Industry
  • Banking
  • Employment Type
  • Full-time
  • Job Functions
  • Engineering
  • Skills
  • Ap