Kafka lead platform engineer

6 days ago

Hyderabad, India SID Global Solutions Full time

Role : Kafka lead platform engineer

Location : Hyderabad

Work mode : on-site

Exp : 3-5 yrs

Daily Routine and Key Responsibilities:

1. Kafka Topic and Queue Management

  • Topic Creation and Management:
  • Create, configure, and manage Kafka topics based on the requirements from various teams.
  • Set appropriate partitions and replication factors to ensure data durability and scalability.
  • Monitor topic health, including under-replicated partitions, and take corrective actions as necessary.
  • Queue Configuration and Management:
  • Configure consumer groups and manage Kafka queues to ensure efficient data processing.
  • Tune consumer group settings, such as max.poll.records, fetch.min.bytes, and session.timeout.ms, to optimize throughput and latency.
  • Monitor and resolve issues related to consumer lag, ensuring that data is processed in a timely manner.

2. Configuration Management

  • Broker Configuration:
  • Adjust broker-level configurations (e.g., log.retention.hours, log.segment.bytes, zookeeper.connection.timeout.ms) to maintain optimal performance and stability.
  • Apply updates and patches to Kafka brokers and ensure configurations are in line with best practices.
  • Security Configuration:
  • Manage Kafka security settings, including SSL/TLS encryption, Kerberos, and SASL authentication.
  • Configure ACLs (Access Control Lists) to control access to topics and consumer groups, ensuring secure and compliant operations.
  • Schema Registry and Compatibility:
  • Work with the Confluent Schema Registry to enforce schema validation and compatibility.
  • Manage versioning of schemas and ensure backward and forward compatibility across different services.

3. Monitoring and Troubleshooting

  • Platform Monitoring:
  • Monitor Kafka clusters using tools like Prometheus, Grafana, and Confluent Control Center to track key metrics (e.g., broker CPU usage, disk I/O, topic throughput).
  • Set up and manage alerts for critical issues like broker failures, high consumer lag, and message backlog.
  • Issue Resolution:
  • Respond to incidents involving Kafka infrastructure, such as broker crashes, network partitions, or performance degradations.
  • Troubleshoot and resolve issues related to topic misconfigurations, consumer group errors, and producer failures.
  • Performance Tuning:
  • Continuously optimize Kafka performance by adjusting configurations based on observed behavior and workload patterns.
  • Conduct periodic health checks and capacity planning to anticipate and prevent potential bottlenecks.

4. Collaboration and Support

  • Inter-Team Collaboration:
  • Work closely with development, DevOps, and data engineering teams to understand their Kafka usage patterns and requirements.
  • Provide guidance on best practices for Kafka topic and consumer group configurations.
  • Support and Documentation:
  • Assist teams with onboarding to Kafka, including setting up topics, consumer groups, and security credentials.
  • Document Kafka configurations, operational procedures, and incident resolutions to ensure knowledge sharing and continuity.

5. Continuous Improvement

  • Automation:
  • Develop and maintain automation scripts for routine Kafka tasks, such as topic creation, scaling clusters, and performing rolling upgrades.
  • Integrate Kafka operations into CI/CD pipelines to streamline deployment and configuration changes.
  • Learning and Development:
  • Stay updated with the latest developments in Kafka and Confluent Platform, exploring new features and improvements.
  • Participate in knowledge-sharing sessions, both within the team and across the organization, to promote best practices in Kafka usage.

End-of-Day Review:

  • Status Review:
  • Review the day's activities, including topic and queue configurations, any incidents resolved, and performance optimizations made.
  • Update relevant stakeholders on Kafka platform status, ongoing tasks, and any issues that need attention.
  • Planning for Tomorrow:
  • Plan and prioritize tasks for the next day, ensuring that any pending requests or incidents are addressed promptly.

Tools and Technologies Used:

  • Kafka Management: Apache Kafka, Confluent Control Center
  • Monitoring and Alerts: Prometheus, Grafana, Elasticsearch, Kibana, Confluent Health+
  • Automation: Shell Scripting, Ansible, Jenkins, GitLab CI
  • Security: SSL/TLS, Kerberos, SASL, ACLs
  • Collaboration: Jira, Confluence, Slack

Qualifications:

  • Strong knowledge and hands-on experience with Apache Kafka and the Confluent Platform.
  • Proficiency in configuring and managing Kafka topics, queues, and consumer groups.
  • Experience with Kafka monitoring, troubleshooting, and performance tuning.
  • Understanding of Kafka security configurations and best practices.
  • Excellent problem-solving skills and the ability to work in a fast-paced environment.

  • Seniority Level
  • Mid-Senior level
  • Industry
  • Banking
  • Employment Type
  • Full-time
  • Job Functions
  • Engineering
  • Skills
  • Ap

  • Kafka lead platform engineer

    1 week ago

    hyderabad, India SID Global Solutions Full time

    Role : Kafka lead platform engineerLocation : HyderabadWork mode : on-siteExp : 3-5 yrsDaily Routine and Key Responsibilities:1. Kafka Topic and Queue ManagementTopic Creation and Management:Create, configure, and manage Kafka topics based on the requirements from various teams.Set appropriate partitions and replication factors to ensure data durability and...

  • Kafka lead platform engineer

    2 weeks ago

    Hyderabad, India SID Global Solutions Full time

    Role : Kafka lead platform engineerLocation : HyderabadWork mode : on-siteExp : 3-5 yrsDaily Routine and Key Responsibilities:1. Kafka Topic and Queue ManagementTopic Creation and Management:Create, configure, and manage Kafka topics based on the requirements from various teams.Set appropriate partitions and replication factors to ensure data durability and...

  • Kafka lead platform engineer

    1 week ago

    Hyderabad, India SID Global Solutions Full time

    Role : Kafka lead platform engineer Location : Hyderabad Work mode : on-site Exp : 3-5 yrs Daily Routine and Key Responsibilities: 1. Kafka Topic and Queue Management Topic Creation and Management: Create, configure, and manage Kafka topics based on the requirements from various teams. Set appropriate partitions and replication factors to ensure data...

  • Kafka Platform Engineers

    1 week ago

    hyderabad, India SID Global Solutions Full time

    Job Description -The Confluent Kafka Platform Engineer plays a critical role in managing Kafka infrastructure on a day-to-day basis. This involves creating and managing Kafka topics, configuring queues, and ensuring the platform is optimized for performance and reliability. The engineer will collaborate with various teams to meet their streaming data needs...

  • Kafka Platform Engineers

    3 weeks ago

    Hyderabad, India SID Global Solutions Full time

    Job Description –The Confluent Kafka Platform Engineer plays a critical role in managing Kafka infrastructure on a day-to-day basis. This involves creating and managing Kafka topics, configuring queues, and ensuring the platform is optimized for performance and reliability. The engineer will collaborate with various teams to meet their streaming data needs...

  • Kafka Platform Engineers

    3 weeks ago

    Hyderabad, India SID Global Solutions Full time

    Job Description – The Confluent Kafka Platform Engineer plays a critical role in managing Kafka infrastructure on a day-to-day basis. This involves creating and managing Kafka topics, configuring queues, and ensuring the platform is optimized for performance and reliability. The engineer will collaborate with various teams to meet their streaming data...

  • Kafka Platform Engineers

    4 weeks ago

    Hyderabad, India SID Global Solutions Full time

    Job Description –The Confluent Kafka Platform Engineer plays a critical role in managing Kafka infrastructure on a day-to-day basis. This involves creating and managing Kafka topics, configuring queues, and ensuring the platform is optimized for performance and reliability. The engineer will collaborate with various teams to meet their streaming data needs...

  • Kafka Platform Engineers

    4 weeks ago

    Hyderabad, India SID Global Solutions Full time

    Job Description –The Confluent Kafka Platform Engineer plays a critical role in managing Kafka infrastructure on a day-to-day basis. This involves creating and managing Kafka topics, configuring queues, and ensuring the platform is optimized for performance and reliability. The engineer will collaborate with various teams to meet their streaming data needs...

  • Kafka Platform Engineers

    2 weeks ago

    Hyderabad, India SID Global Solutions Full time

    Job Description -The Confluent Kafka Platform Engineer plays a critical role in managing Kafka infrastructure on a day-to-day basis. This involves creating and managing Kafka topics, configuring queues, and ensuring the platform is optimized for performance and reliability. The engineer will collaborate with various teams to meet their streaming data needs...

  • Kafka Platform Engineers

    2 weeks ago

    Hyderabad, India SID Global Solutions Full time

    Job Description -The Confluent Kafka Platform Engineer plays a critical role in managing Kafka infrastructure on a day-to-day basis. This involves creating and managing Kafka topics, configuring queues, and ensuring the platform is optimized for performance and reliability. The engineer will collaborate with various teams to meet their streaming data needs...

  • Kafka Platform Engineers

    1 week ago

    Hyderabad, India SID Global Solutions Full time

    Job Description - The Confluent Kafka Platform Engineer plays a critical role in managing Kafka infrastructure on a day-to-day basis. This involves creating and managing Kafka topics, configuring queues, and ensuring the platform is optimized for performance and reliability. The engineer will collaborate with various teams to meet their streaming data needs...

  • kafka platform engineer

    1 week ago

    hyderabad, India SID Global Solutions Full time

    Role : kafka platform engineerLocation : HyderabadWork mode : on-siteExp : 3-5 yrsDaily Routine and Key Responsibilities:1. Kafka Topic and Queue ManagementTopic Creation and Management:Create, configure, and manage Kafka topics based on the requirements from various teams.Set appropriate partitions and replication factors to ensure data durability and...

  • kafka platform engineer

    6 days ago

    Hyderabad, India SID Global Solutions Full time

    Role : kafka platform engineer Location : Hyderabad Work mode : on-site Exp : 3-5 yrs Daily Routine and Key Responsibilities: 1. Kafka Topic and Queue Management Topic Creation and Management: Create, configure, and manage Kafka topics based on the requirements from various teams. Set appropriate partitions and replication factors to...

  • kafka platform engineer

    1 week ago

    Hyderabad, India SID Global Solutions Full time

    Role : kafka platform engineer Location : Hyderabad Work mode : on-site Exp : 3-5 yrs Daily Routine and Key Responsibilities: 1. Kafka Topic and Queue Management Topic Creation and Management: Create, configure, and manage Kafka topics based on the requirements from various teams. Set appropriate partitions and replication factors to ensure data...

  • kafka platform engineer

    2 weeks ago

    Hyderabad, India SID Global Solutions Full time

    Role : kafka platform engineerLocation : HyderabadWork mode : on-siteExp : 3-5 yrsDaily Routine and Key Responsibilities:1. Kafka Topic and Queue ManagementTopic Creation and Management:Create, configure, and manage Kafka topics based on the requirements from various teams.Set appropriate partitions and replication factors to ensure data durability and...

  • Lead Infrastructure Engineer

    2 weeks ago

    Hyderabad, India WELLS FARGO BANK Full time

    About this role:Wells Fargo is seeking a Lead Infrastructure EngineerIn this role, you will:Lead complex initiatives to develop infrastructure to provide solutions for business applicationsParticipate in various projects intended to continually improve or upgrade the infrastructureEvaluate internal and external software solutions which could be leveraged to...

  • Kafka Lead(Technical Lead)

    4 months ago

    hyderabad, India Anicalls (Pty) Ltd Full time

    Candidate should have:• Experience with developing KSQL queries and best practices of using KSQL vs streams• Strong fundamentals in Kafka administration, configuration, and troubleshooting• Practical experience with how to scale Kafka, K-Streams, and Connector infrastructures, with the motivation to build efficient platforms• Hands-on experience as a...

  • Lead Infrastructure Engineer

    1 month ago

    Hyderabad, India WELLS FARGO BANK Full time

    About this role:Wells Fargo is seeking a...In this role, you will:Lead complex initiatives to develop infrastructure to provide solutions for business applicationsParticipate in various projects intended to continually improve or upgrade the infrastructureEvaluate internal and external software solutions which could be leveraged to meet target state...

  • Kafka Developer

    3 weeks ago

    Hyderabad, India Health Catalyst Full time

    About UsThe healthcare industry is the next great frontier of opportunity for software development, and Health Catalyst is one of the most dynamic and influential companies in this space. We are working on solving national-level healthcare problems, and this is your chance to improve the lives of millions of people, including your family and friends. Health...

  • Kafka Developer

    4 weeks ago

    Hyderabad, India Health Catalyst Full time

    About UsThe healthcare industry is the next great frontier of opportunity for software development, and Health Catalyst is one of the most dynamic and influential companies in this space. We are working on solving national-level healthcare problems, and this is your chance to improve the lives of millions of people, including your family and friends. Health...