Kafka SME

Job Summary:

We are seeking an experienced and highly skilled Kafka Subject Matter Expert (SME) to join our team. As a Kafka SME, you will lead the design, implementation, and optimization of streaming platforms using Apache Kafka. You will provide technical leadership and expertise in integrating Kafka with various systems, solving performance issues, and ensuring the smooth operation of large-scale distributed architectures.

You will collaborate with cross-functional teams, offering guidance on best practices, troubleshooting complex Kafka-related issues, and driving improvements in data streaming solutions across the organization.

Key Responsibilities:

• Kafka Architecture & Design:
• Design and architect robust and scalable Kafka-based solutions for real-time data streaming, event processing, and analytics.
• Provide expertise in Kafka’s core concepts such as topics, partitions, producers, consumers, consumer groups, and replication.
• Ensure high availability, fault tolerance, and performance of Kafka clusters.
• Implementation & Integration:
• Lead the integration of Kafka with other platforms, services, and databases, ensuring seamless data flow across systems.
• Collaborate with software engineers to design and implement Kafka producers, consumers, and stream processing pipelines.
• Performance Optimization:
• Analyze and optimize Kafka cluster performance, including memory, disk, network utilization, and throughput.
• Troubleshoot and resolve issues related to Kafka performance, reliability, and scalability.
• Support & Troubleshooting:
• Act as the go-to expert for troubleshooting Kafka-related issues in production and non-production environments.
• Provide guidance on monitoring Kafka clusters, implementing alerts, and using tools such as Kafka Manager, Confluent Control Center, and others.
• Best Practices & Documentation:
• Establish and enforce best practices for deploying, managing, and scaling Kafka clusters.
• Create and maintain documentation for Kafka deployment, monitoring, configuration, and troubleshooting procedures.
• Mentorship & Knowledge Sharing:
• Mentor team members on Kafka-related topics, ensuring that the team stays up-to-date with best practices and new Kafka features.
• Conduct knowledge-sharing sessions and workshops to promote Kafka adoption within the organization.
• Security & Compliance:
• Implement security measures for Kafka clusters, including authentication, authorization, encryption, and access control.
• Ensure compliance with data privacy regulations and organizational security policies related to data streaming.

Required Skills and Qualifications:

• Experience:
• 5+ years of experience with Apache Kafka and related technologies (e.g., Kafka Streams, Kafka Connect, KSQL).
• Strong understanding of distributed systems, event-driven architecture, and real-time data processing.
• Technical Skills:
• Expertise in Kafka cluster setup, configuration, and management.
• Hands-on experience with Kafka topics, partitions, replication, and fault tolerance.
• Proficiency with Kafka-related tools like Kafka Manager, Kafka Streams, Kafka Connect, and Confluent platform tools.
• Experience in integrating Kafka with big data platforms, databases, and message queues.
• Strong programming skills in languages such as Java, Scala, Python, or Go for developing Kafka producers and consumers.
• Performance Tuning & Troubleshooting:
• Experience optimizing Kafka performance (e.g., producer/consumer throughput, replication strategies).
• Proficient in troubleshooting issues such as latency, consumer lag, and partition rebalancing.
• Cloud & Containerization:
• Experience with cloud platforms (AWS, Azure, GCP) and container orchestration tools like Kubernetes for managing Kafka clusters in cloud environments.
• Security & Compliance:
• Knowledge of Kafka security configurations, including SSL, SASL, ACLs, and data encryption at rest/in-transit.

Preferred Qualifications:

• Experience with Confluent Kafka, Confluent Cloud, and Kafka Streams.
• Familiarity with stream processing frameworks (e.g., Apache Flink, Apache Storm).
• Experience in deploying Kafka clusters at scale (multi-region, hybrid-cloud environments).
• Experience in Linux/Unix, Kubernetes, Monitoring tools.
• Certification in Kafka (Confluent Certified Developer for Apache Kafka or similar certifications).

Soft Skills:

• Strong problem-solving and analytical skills.
• Excellent communication and collaboration skills, with the ability to explain complex concepts to non-technical stakeholders.
• Strong leadership capabilities with experience in guiding teams and mentoring junior engineers.

Education:

• A Bachelor’s or Master’s degree in Computer Science, Information Technology, Engineering, or related field, or equivalent practical experience.