Freelance Job Opportunity – Kafka/Streaming Data Engineer

5 days ago

Chennai, India ThreatXIntel Full time

Company Description

ThreatXIntel is a startup cyber security company dedicated to protecting businesses and organizations from cyber threats. We provide services such as cloud security, web and mobile security testing, and DevSecOps. Our goal is to offer customized, affordable solutions to meet the specific needs of our clients. By continuously monitoring and testing digital environments, we identify vulnerabilities before they can be exploited. At ThreatXIntel, our mission is to deliver exceptional cyber security services that allow our clients to focus on growing their businesses.


We are seeking an experienced Kafka/Streaming Data Engineer (Freelance/Contract) to join our team and support the rollout of a high-volume, real-time data ingestion and fulfillment platform.

The role requires deep hands-on expertise in Apache Kafka cluster management, automation, monitoring, and performance tuning. You will work closely with our data engineering and fulfillment teams to ensure a scalable, reliable, and production-ready streaming ecosystem.

Key Responsibilities

  • Kafka Administration & Optimization
  • Configure, manage, and tune multi-node Kafka clusters for high throughput, low latency, and fault tolerance.
  • Define and optimize partitions, replication, retention policies, and consumer groups.
  • Ensure stability of event-driven pipelines across on-prem and AWS cloud environments.
  • Automation & Infrastructure as Code
  • Develop and maintain Ansible playbooks for Kafka cluster provisioning, deployment, scaling, and upgrades.
  • Automate repetitive operational tasks (broker restarts, topic creation, retention updates, etc.).
  • Build reliable solutions for data replication between on-premise and AWS cloud.
  • Monitoring & Observability
  • Implement Prometheus and Grafana dashboards for Kafka brokers, producers, and consumers.
  • Track and alert on lag, throughput, latency, and error rates.
  • Perform load testing and failure simulations to validate production readiness.
  • Streaming Pipeline Support
  • Monitor and troubleshoot Ab Initio CDC jobs, Kafka topics, and Apache Flink consumers.
  • Ensure data integrity and timely delivery into downstream databases.
  • Support UAT and production go-live with synthetic volume tests.
  • Troubleshooting & Performance Tuning
  • Diagnose and resolve issues related to broker performance, consumer lag, data loss, or scaling bottlenecks.
  • Propose short-term fixes and implement long-term architectural improvements.

Required Skills & Experience


  • Kafka Expertise:
  • Strong hands-on experience with Kafka cluster administration in production-grade environments.
  • Deep understanding of broker tuning, partitioning strategies, consumer offsets, replication, and retention policies.
  • Automation & Scripting:
  • Proficiency with Ansible for automated deployments and configuration management.
  • Strong scripting skills in Python, Bash, or Shell.
  • Monitoring & Alerting:
  • Hands-on experience with Prometheus and Grafana setup, tuning, and alert configuration.
  • Streaming & Data Engineering:
  • Knowledge of Apache Flink or equivalent real-time stream-processing frameworks.
  • Familiarity with CDC tools (Ab Initio CDC preferred).
  • Databases:
  • Good understanding of PostgreSQL and integration patterns with Kafka.
  • Production Experience:
  • Worked with large-scale, high-throughput data pipelines (multi-billion row datasets).
  • Experience troubleshooting real-time ingestion systems under heavy load.
  • Soft Skills:
  • Strong problem-solving skills with a “production-first” mindset.
  • Excellent communication for cross-team collaboration.

  • Freelance job opportunity

    4 days ago

    Chennai, India ThreatXIntel Full time

    Company Description Threat XIntel is a startup cyber security company dedicated to protecting businesses and organizations from cyber threats. We provide services such as cloud security, web and mobile security testing, and Dev Sec Ops. Our goal is to offer customized, affordable solutions to meet the specific needs of our clients. By continuously...

  • Freelance Kafka

    1 day ago

    Chennai, Tamil Nadu, India ThreatXIntel Full time ₹ 20,00,000 - ₹ 25,00,000 per year

    Company DescriptionThreatXIntel is a startup cybersecurity company dedicated to protecting businesses and organizations from cyber threats. Our team offers a range of services, including cloud security, web and mobile security testing, and DevSecOps. We deliver customized, affordable solutions to meet the specific needs of our clients, ensuring that every...

  • Kafka Engineer

    3 weeks ago

    Chennai, Tamil Nadu, India BAPON Full time

    Job DescriptionCompany DescriptionBAPON is a forward-thinking IT consulting company dedicated to empowering businesses with intelligent automation, seamless digital integration, and transformative cloud solutions. We are Cyber Essential Plus and ISO 9001 & ISO 27001 certified, ensuring the highest standards of quality, security, and compliance. Our core...

  • Data Engineer

    1 week ago

    Chennai, India CORTEX Consultants Full time

    About the job :Cortex is urgently hiring for a Data Engineer to join our team! We're seeking a highly skilled individual with a strong background in Kafka, Python, and Azure Databricks to drive our healthcare data engineering projects.Experience : 5 to 8 yearsLocation : Bangalore, Noida, and Hyderabad (Hybrid, 2 days a week in the office is...

  • Kafka Platform Engineer

    3 weeks ago

    Chennai, Tamil Nadu, India Intraedge Technologies Ltd. Full time

    Job Description :We're looking for a highly skilled Kafka Platform Engineer with 3+ years of experience to join our team. The ideal candidate will be responsible for designing, configuring, and deploying Kafka-based event streaming solutions for our clients. You'll be a key player in ensuring seamless integration with client applications and providing expert...

  • Kafka Developer

    3 days ago

    Bengaluru, Chennai, Hyderabad, India Sdna Global Full time ₹ 20,00,000 - ₹ 25,00,000 per year

    Job DescriptionKafka DeveloperConfluent Kafka Engineers with in-depth knowledge of Confluent/Kafka ecosystem products/technologiesExperience 10-15 years of experience Deployment and Management of Confluent Kafka.Streams based application problem investigationExperience and hands on expertise in KSql , KStreams and Kafka connectors.In-depth knowledge on...

  • Senior Data Architect for Kafka Integration

    2 weeks ago

    Chennai, Tamil Nadu, India beBeeKafka Full time ₹ 1,04,000 - ₹ 1,30,878

    Kafka Integration SpecialistWe are seeking an experienced professional to fill the role of Kafka Integration Specialist. This position requires a strong understanding of designing, developing, and integrating Apache Kafka solutions. The successful candidate will be responsible for creating and maintaining Kafka-based data pipelines, managing clusters, and...

  • Kafka Developer

    24 hours ago

    Bengaluru, Chennai, Hyderabad, India Tanisha Systems Full time ₹ 20,00,000 - ₹ 25,00,000 per year

    Role & responsibilitiesPreferred candidate profileNotie Period- Immediate joiner onlyJob DescriptionKafka DeveloperConfluent Kafka Engineers with in-depth knowledge of Confluent/Kafka ecosystem products/technologies Experience 10-15 years of experience Deployment and Management of Confluent Kafka. Streams based application problem investigation Experience...

  • Kafka Trainer

    2 days ago

    Chennai, Tamil Nadu, India Nithminds Private limited Full time

    **Job Summary**: **Key Responsibilities**: - Develop and deliver structured training programs on **Apache Kafka, Kafka Streams, Kafka Connect, and related technologies**. - Conduct **hands-on labs and live demonstrations** covering Kafka’s core components such as **producers, consumers, topics, partitions, brokers, and Zookeeper**. - Train learners on...

  • Data Engineer

    6 days ago

    Chennai, India NTT DATA Full time

    Req ID: NTT DATA strives to hire exceptional, innovative and passionate individuals who want to grow with us. If you want to be part of an inclusive, adaptable, and forward-thinking organization, apply now. We are currently seeking a Data Engineer - Azure to join our team in Chennai, Tamil Nādu (IN-TN), India (IN). "Job Duties: Key Responsibilities: ...