Freelance job opportunity

Company Description Threat XIntel is a startup cyber security company dedicated to protecting businesses and organizations from cyber threats. We provide services such as cloud security, web and mobile security testing, and Dev Sec Ops. Our goal is to offer customized, affordable solutions to meet the specific needs of our clients. By continuously monitoring and testing digital environments, we identify vulnerabilities before they can be exploited. At Threat XIntel, our mission is to deliver exceptional cyber security services that allow our clients to focus on growing their businesses. We are seeking an experienced Kafka/Streaming Data Engineer (Freelance/Contract) to join our team and support the rollout of a high-volume, real-time data ingestion and fulfillment platform . The role requires deep hands-on expertise in Apache Kafka cluster management, automation, monitoring, and performance tuning . You will work closely with our data engineering and fulfillment teams to ensure a scalable, reliable, and production-ready streaming ecosystem . Key Responsibilities Kafka Administration & Optimization Configure, manage, and tune multi-node Kafka clusters for high throughput, low latency, and fault tolerance. Define and optimize partitions, replication, retention policies, and consumer groups . Ensure stability of event-driven pipelines across on-prem and AWS cloud environments. Automation & Infrastructure as Code Develop and maintain Ansible playbooks for Kafka cluster provisioning, deployment, scaling, and upgrades. Automate repetitive operational tasks (broker restarts, topic creation, retention updates, etc.). Build reliable solutions for data replication between on-premise and AWS cloud . Monitoring & Observability Implement Prometheus and Grafana dashboards for Kafka brokers, producers, and consumers. Track and alert on lag, throughput, latency, and error rates . Perform load testing and failure simulations to validate production readiness. Streaming Pipeline Support Monitor and troubleshoot Ab Initio CDC jobs, Kafka topics, and Apache Flink consumers . Ensure data integrity and timely delivery into downstream databases. Support UAT and production go-live with synthetic volume tests. Troubleshooting & Performance Tuning Diagnose and resolve issues related to broker performance, consumer lag, data loss, or scaling bottlenecks . Propose short-term fixes and implement long-term architectural improvements . Required Skills & Experience Kafka Expertise: Strong hands-on experience with Kafka cluster administration in production-grade environments . Deep understanding of broker tuning, partitioning strategies, consumer offsets, replication, and retention policies . Automation & Scripting: Proficiency with Ansible for automated deployments and configuration management. Strong scripting skills in Python, Bash, or Shell . Monitoring & Alerting: Hands-on experience with Prometheus and Grafana setup, tuning, and alert configuration. Streaming & Data Engineering: Knowledge of Apache Flink or equivalent real-time stream-processing frameworks. Familiarity with CDC tools (Ab Initio CDC preferred). Databases: Good understanding of Postgre SQL and integration patterns with Kafka. Production Experience: Worked with large-scale, high-throughput data pipelines (multi-billion row datasets). Experience troubleshooting real-time ingestion systems under heavy load . Soft Skills: Strong problem-solving skills with a “production-first” mindset . Excellent communication for cross-team collaboration.