Kafka Admin

Job Description:

- #LI-DNI
- The primary objective of the Kafka Administrator is to develop and maintain the corporate messaging infrastructure ensuring message delivery, security, retention, and recoverability of the topics and messages.
- In this role, the Kafka Admin will design, build, test, and deploy quality messaging and streaming solutions that are consistent with project objectives, technology architecture standards and departmental best practices. The Kafka admin will design and implement event sourcing strategies, administer enterprise-wide messaging architecture, and provide daily management of Kafka eco-system.
- Specific responsibilities include:

- Manage large scale multi-nodes Kafka cluster environments residing on AWS and on-prem OpenShift platform.
- Handle all Kafka environment builds, including design, capacity planning, cluster setup, performance tuning and ongoing monitoring.
- Perform high-level, day-to-day operational maintenance, support, and upgrades for the Kafka Cluster.
- Creation of key performance metrics, measuring the utilization, performance and overall health of the cluster.
- Capacity planning and implementation of new/upgraded hardware and software releases as well as for storage infrastructure.
- Research and recommend innovative, and where possible, automate key administration tasks.
- Ability to closely collaborate with product managers and lead engineers.
- Provide guidance in the creation and modification of standards and procedures
- Create topics, setup redundancy cluster, deploy monitoring tools, alerts
- Proactively monitor and setup alerting mechanism for Kafka Cluster and supporting hardware to ensure system health and maximum availability
- Perform high-level, day-to-day operational maintenance, support, and upgrades for the Kafka cluster
- Provide technical expertise and guidance to production support.
- Hands-on experience with Kafka clusters hosted on ROSA in AWS and on-prem K8 Open Shift platform
- Experience in building Kafka pipelines using Terraform, Ansible, Cloud formation templates, shells etc.
- Experience in implementing security & authorization (permission based) on Kafka cluster.
- Aid System Administrators with setting up Kafka platform in provisioning, access lists Kerberos and SSL configurations.
- Experience in setting standards to automate deployments using Kubernetes, Docker, Chef or Jenkins
- Experience in open source and confluence Kafka, zookeepers, Kafka connect, schema registry, KSQL, Rest proxy and Kafka Control center.
- Experience in Kafka Mirror Maker or Confluent Replicator
- Experience in High availability cluster setup, maintenance and ongoing 24/7 support
- Establish best practice standards for configuring Source and Sink connectors.
- Hands on experience in standing up and administrating Kafka platform from scratch which includes creating a backup & mirroring of Kafka Cluster brokers, broker sizing, topic sizing, h/w sizing, performance monitoring, broker security, topic security, consumer/producer access management (ACL)
- Knowledge of Kafka API (development experience is a plus)
- Knowledge of best practices related to security, performance, and disaster recovery.
- Ability to concentrate on a wide range of loosely defined complex situations, which require creativity and originality, where guidance and counsel may be unavailable.
- Demonstrate a product mindset with an ability to set forward thinking and direction.
- Ability to synthesize large amounts of complex data into meaningful conclusions and present recommendations.
- Ability to maintain a positive attitude while working with high demands and short deadlines that leads to working after hours.
- Must have excellent communications and interpersonal skills

Not Required, but Preferred:

- Experience with Oracle Streams and/or another real-time streaming solution is a plus.
- Experience in setting up Prometheus/Grafana or ELK monitoring tools is a plus.
- Experience as Linux (RHEL) /Unix administrator is huge plus.
- Experience in PostgreSQL, SQL Server, No-SQL (Couchbase), Oracle, and AWS Cloud is a plus.
- Understanding of or experience with Programming languages like Python, PowerShell, etc is plus.
- Key Skills:

- Hands on experience in standing up and administrating Kafka platform from scratch which includes creating a backup & mirroring of Kafka Cluster brokers, broker sizing, topic sizing, h/w sizing, performance monitoring, broker security, topic security, consumer/producer access management (ACL)
- Knowledge of Kafka API (development experience is a plus)
- Knowledge of best practices related to security, performance, and disaster recovery.
- Ability to concentrate on a wide range of loosely defined complex situations, which require creativity and originality, where guidance and counsel may be unavailable.
- Demonstrate a product mindset with an ability to set forward thinking and direction.
- Ability to synthesize large amounts of complex data into meaningf