Kafka Platform Engineers

2 weeks ago

Hyderabad, India SID Global Solutions Full time

Job Description -


The Confluent Kafka Platform Engineer plays a critical role in managing Kafka infrastructure on a day-to-day basis. This involves creating and managing Kafka topics, configuring queues, and ensuring the platform is optimized for performance and reliability. The engineer will collaborate with various teams to meet their streaming data needs while ensuring that Kafka is running smoothly and efficiently.


Daily Routine and Key Responsibilities:


1. Kafka Topic and Queue Management

  • Topic Creation and Management:
  • Create, configure, and manage Kafka topics based on the requirements from various teams.
  • Set appropriate partitions and replication factors to ensure data durability and scalability.
  • Monitor topic health, including under-replicated partitions, and take corrective actions as necessary.
  • Queue Configuration and Management:
  • Configure consumer groups and manage Kafka queues to ensure efficient data processing.
  • Tune consumer group settings, such as max.poll.records, fetch.min.bytes, and session.timeout.ms, to optimize throughput and latency.
  • Monitor and resolve issues related to consumer lag, ensuring that data is processed in a timely manner.

2. Configuration Management

  • Broker Configuration:
  • Adjust broker-level configurations (e.g., log.retention.hours, log.segment.bytes, zookeeper.connection.timeout.ms) to maintain optimal performance and stability.
  • Apply updates and patches to Kafka brokers and ensure configurations are in line with best practices.
  • Security Configuration:
  • Manage Kafka security settings, including SSL/TLS encryption, Kerberos, and SASL authentication.
  • Configure ACLs (Access Control Lists) to control access to topics and consumer groups, ensuring secure and compliant operations.
  • Schema Registry and Compatibility:
  • Work with the Confluent Schema Registry to enforce schema validation and compatibility.
  • Manage versioning of schemas and ensure backward and forward compatibility across different services.

3. Monitoring and Troubleshooting

  • Platform Monitoring:
  • Monitor Kafka clusters using tools like Prometheus, Grafana, and Confluent Control Center to track key metrics (e.g., broker CPU usage, disk I/O, topic throughput).
  • Set up and manage alerts for critical issues like broker failures, high consumer lag, and message backlog.
  • Issue Resolution:
  • Respond to incidents involving Kafka infrastructure, such as broker crashes, network partitions, or performance degradations.
  • Troubleshoot and resolve issues related to topic misconfigurations, consumer group errors, and producer failures.
  • Performance Tuning:
  • Continuously optimize Kafka performance by adjusting configurations based on observed behavior and workload patterns.
  • Conduct periodic health checks and capacity planning to anticipate and prevent potential bottlenecks.

4. Collaboration and Support

  • Inter-Team Collaboration:
  • Work closely with development, DevOps, and data engineering teams to understand their Kafka usage patterns and requirements.
  • Provide guidance on best practices for Kafka topic and consumer group configurations.
  • Support and Documentation:
  • Assist teams with onboarding to Kafka, including setting up topics, consumer groups, and security credentials.
  • Document Kafka configurations, operational procedures, and incident resolutions to ensure knowledge sharing and continuity.


Tools and Technologies Used:

  • Kafka Management: Apache Kafka, Confluent Control Center
  • Monitoring and Alerts: Prometheus, Grafana, Elasticsearch, Kibana, Confluent Health+
  • Automation: Shell Scripting, Ansible, Jenkins, GitLab CI
  • Security: SSL/TLS, Kerberos, SASL, ACLs
  • Collaboration: Jira, Confluence, Slack


NEED TO HAVE CERTIFICATION IN APACHE.


Qualifications:

  • Strong knowledge and hands-on experience with Apache Kafka and the Confluent Platform.
  • Proficiency in configuring and managing Kafka topics, queues, and consumer groups.
  • Experience with Kafka monitoring, troubleshooting, and performance tuning.
  • Understanding of Kafka security configurations and best practices.
  • Excellent problem-solving skills and the ability to work in a fast-paced environment.

  • Kafka Platform Engineers

    1 week ago

    hyderabad, India SID Global Solutions Full time

    Job Description -The Confluent Kafka Platform Engineer plays a critical role in managing Kafka infrastructure on a day-to-day basis. This involves creating and managing Kafka topics, configuring queues, and ensuring the platform is optimized for performance and reliability. The engineer will collaborate with various teams to meet their streaming data needs...

  • Kafka Platform Engineers

    3 weeks ago

    Hyderabad, India SID Global Solutions Full time

    Job Description –The Confluent Kafka Platform Engineer plays a critical role in managing Kafka infrastructure on a day-to-day basis. This involves creating and managing Kafka topics, configuring queues, and ensuring the platform is optimized for performance and reliability. The engineer will collaborate with various teams to meet their streaming data needs...

  • Kafka Platform Engineers

    4 weeks ago

    Hyderabad, India SID Global Solutions Full time

    Job Description –The Confluent Kafka Platform Engineer plays a critical role in managing Kafka infrastructure on a day-to-day basis. This involves creating and managing Kafka topics, configuring queues, and ensuring the platform is optimized for performance and reliability. The engineer will collaborate with various teams to meet their streaming data needs...

  • Kafka Platform Engineers

    3 weeks ago

    Hyderabad, India SID Global Solutions Full time

    Job Description – The Confluent Kafka Platform Engineer plays a critical role in managing Kafka infrastructure on a day-to-day basis. This involves creating and managing Kafka topics, configuring queues, and ensuring the platform is optimized for performance and reliability. The engineer will collaborate with various teams to meet their streaming data...

  • Kafka Platform Engineers

    2 weeks ago

    Hyderabad, India SID Global Solutions Full time

    Job Description -The Confluent Kafka Platform Engineer plays a critical role in managing Kafka infrastructure on a day-to-day basis. This involves creating and managing Kafka topics, configuring queues, and ensuring the platform is optimized for performance and reliability. The engineer will collaborate with various teams to meet their streaming data needs...

  • Kafka Platform Engineers

    1 week ago

    Hyderabad, India SID Global Solutions Full time

    Job Description - The Confluent Kafka Platform Engineer plays a critical role in managing Kafka infrastructure on a day-to-day basis. This involves creating and managing Kafka topics, configuring queues, and ensuring the platform is optimized for performance and reliability. The engineer will collaborate with various teams to meet their streaming data needs...

  • kafka platform engineer

    1 week ago

    hyderabad, India SID Global Solutions Full time

    Role : kafka platform engineerLocation : HyderabadWork mode : on-siteExp : 3-5 yrsDaily Routine and Key Responsibilities:1. Kafka Topic and Queue ManagementTopic Creation and Management:Create, configure, and manage Kafka topics based on the requirements from various teams.Set appropriate partitions and replication factors to ensure data durability and...

  • kafka platform engineer

    7 days ago

    Hyderabad, India SID Global Solutions Full time

    Role : kafka platform engineer Location : Hyderabad Work mode : on-site Exp : 3-5 yrs Daily Routine and Key Responsibilities: 1. Kafka Topic and Queue Management Topic Creation and Management: Create, configure, and manage Kafka topics based on the requirements from various teams. Set appropriate partitions and replication factors to...

  • kafka platform engineer

    2 weeks ago

    Hyderabad, India SID Global Solutions Full time

    Role : kafka platform engineerLocation : HyderabadWork mode : on-siteExp : 3-5 yrsDaily Routine and Key Responsibilities:1. Kafka Topic and Queue ManagementTopic Creation and Management:Create, configure, and manage Kafka topics based on the requirements from various teams.Set appropriate partitions and replication factors to ensure data durability and...

  • kafka platform engineer

    1 week ago

    Hyderabad, India SID Global Solutions Full time

    Role : kafka platform engineer Location : Hyderabad Work mode : on-site Exp : 3-5 yrs Daily Routine and Key Responsibilities: 1. Kafka Topic and Queue Management Topic Creation and Management: Create, configure, and manage Kafka topics based on the requirements from various teams. Set appropriate partitions and replication factors to ensure data...

  • Kafka lead platform engineer

    1 week ago

    hyderabad, India SID Global Solutions Full time

    Role : Kafka lead platform engineerLocation : HyderabadWork mode : on-siteExp : 3-5 yrsDaily Routine and Key Responsibilities:1. Kafka Topic and Queue ManagementTopic Creation and Management:Create, configure, and manage Kafka topics based on the requirements from various teams.Set appropriate partitions and replication factors to ensure data durability and...

  • Kafka lead platform engineer

    7 days ago

    Hyderabad, India SID Global Solutions Full time

    Role : Kafka lead platform engineer Location : Hyderabad Work mode : on-site Exp : 3-5 yrs Daily Routine and Key Responsibilities: 1. Kafka Topic and Queue Management Topic Creation and Management: Create, configure, and manage Kafka topics based on the requirements from various teams. Set appropriate partitions and replication factors to...

  • Kafka lead platform engineer

    2 weeks ago

    Hyderabad, India SID Global Solutions Full time

    Role : Kafka lead platform engineerLocation : HyderabadWork mode : on-siteExp : 3-5 yrsDaily Routine and Key Responsibilities:1. Kafka Topic and Queue ManagementTopic Creation and Management:Create, configure, and manage Kafka topics based on the requirements from various teams.Set appropriate partitions and replication factors to ensure data durability and...

  • Kafka lead platform engineer

    1 week ago

    Hyderabad, India SID Global Solutions Full time

    Role : Kafka lead platform engineer Location : Hyderabad Work mode : on-site Exp : 3-5 yrs Daily Routine and Key Responsibilities: 1. Kafka Topic and Queue Management Topic Creation and Management: Create, configure, and manage Kafka topics based on the requirements from various teams. Set appropriate partitions and replication factors to ensure data...

  • Kafka Admin

    2 months ago

    Hyderabad, India DATAECONOMY Full time

    Job DescriptionJob Description:We are seeking a skilled Core Kafka Administrator to join our dynamic team. As a Kafka Administrator, you will be responsible for managing and maintaining our Kafka clusters, ensuring their availability, performance, and security. You will play a critical role in designing Kafka infrastructure, configuring topics, monitoring...

  • Kafka Admin

    2 months ago

    Hyderabad, India DATAECONOMY Full time

    Job Description:We are seeking a skilled Core Kafka Administrator to join our dynamic team. As a Kafka Administrator, you will be responsible for managing and maintaining our Kafka clusters, ensuring their availability, performance, and security. You will play a critical role in designing Kafka infrastructure, configuring topics, monitoring cluster health,...

  • Kafka Developer

    3 weeks ago

    Hyderabad, India Health Catalyst Full time

    About UsThe healthcare industry is the next great frontier of opportunity for software development, and Health Catalyst is one of the most dynamic and influential companies in this space. We are working on solving national-level healthcare problems, and this is your chance to improve the lives of millions of people, including your family and friends. Health...

  • Kafka Developer

    4 weeks ago

    Hyderabad, India Health Catalyst Full time

    About UsThe healthcare industry is the next great frontier of opportunity for software development, and Health Catalyst is one of the most dynamic and influential companies in this space. We are working on solving national-level healthcare problems, and this is your chance to improve the lives of millions of people, including your family and friends. Health...

  • Kafka Developer

    3 weeks ago

    Hyderabad, India Health Catalyst Full time

    About Us The healthcare industry is the next great frontier of opportunity for software development, and Health Catalyst is one of the most dynamic and influential companies in this space. We are working on solving national-level healthcare problems, and this is your chance to improve the lives of millions of people, including your family and friends....

  • Kafka Developer

    4 weeks ago

    Hyderabad, India Health Catalyst Full time

    About UsThe healthcare industry is the next great frontier of opportunity for software development, and Health Catalyst is one of the most dynamic and influential companies in this space. We are working on solving national-level healthcare problems, and this is your chance to improve the lives of millions of people, including your family and friends. Health...