Kafka Developer

Kafka Developer Experienced Associate (36 Years)

Our Analytics & Insights Managed Services team brings a unique combination of industry expertise, technology, data management, and managedservices experience to create sustained outcomes for our clients and improve business performance. We empower companies to transform their approach to analytics and insights while building your skills in exciting new directions. Have a voice at our table to help design, build, and operate the next generation of Big Data streaming and batch services leveraging Kafka, Hadoop, and HDFS.

Job Requirements and Preferences

Basic Qualifications

• Minimum Degree Required:

Bachelor's degree in computer science, Data Engineering, Information Systems, or a related technical field

• Minimum Years of Experience:

25 years of handson experience developing and supporting Big Data solutions with Apache Kafka and Hadoop ecosystems, Kafka (core brokers, Streams API), Connectors via Informatica, Qlik Replicate, ADF, Python & Pyspark, SQL, Data Modeling

Preferred Qualifications

• Degree Preferred:

Master's degree in data science, Analytics, Computer Science, or related discipline

• Preferred Fields of Study:

Data Processing/Analytics, Management Information Systems, Software Engineering

Preferred Knowledge & Skills

As an Experienced Associate, you'll contribute to the design, implementation, and optimization of realtime and batch data pipelines, collaborating closely with engineers and data scientists:

• Kafka Streaming & Messaging

– Develop and maintain Kafka producers, consumers, and stream-processing applications using Java, Scala, or Python

– Design topic layouts, partitioning schemes, and data retention policies to support highthroughput, lowlatency use cases

• Kafka (core brokers, Streams API), Connectors via Informatica, Qlik Replicate, ADF, Python & Pyspark, SQL, Data Modeling

• Hadoop Ecosystem & HDFS

– Work with HDFS to ingest and store large volumes of structured and unstructured data

– Build MapReduce or Spark jobs to process historical datasets in batch

• Stream Processing Frameworks

– Implement realtime transforms and stateful operations with Kafka Streams, Apache Flink, or Spark Structured Streaming

– Handle exactlyonce semantics, windowing, and watermarks in streaming pipelines

• Data Integration & Orchestration

– Integrate Kafka with other systems (JDBC sources, REST APIs, NoSQL stores) and Hadoop components via Sqoop, NiFi, or custom connectors

– Orchestrate complex workflows using Apache Airflow, Oozie, or NiFi

• Performance Tuning & Reliability

– Tune Kafka brokers, consumer groups, and Hadoop cluster settings for scalability and resilience

– Implement monitoring and alerting (Prometheus, Grafana, Confluent Control Center) to maintain SLAs

• Cloud & Hybrid Deployments

– Deploy and manage Kafka clusters and Hadoop services on AWS, Azure, or GCP (MSK, HDInsight, Dataproc)

– Use InfrastructureasCode (Terraform, CloudFormation) and containerization (Docker, Kubernetes) for repeatable environments

• Security & Governance

– Apply encryption (TLS), authentication (SASL), and ACLs within Kafka and secure HDFS permissions

– Collaborate on data cataloging, lineage, and compliance standards

• Collaboration & Communication

– Partner with data scientists, BI teams, and stakeholders to translate requirements into scalable streaming and batch solutions

– Document architecture diagrams, runbooks, and bestpractice guides

• Continuous Learning & Innovation

– Stay current on emerging Big Data and streaming technologies (Kafka Connect, ksqlDB, Pulsar)

– Share knowledge through code reviews, brownbag sessions, and contributions to internal accelerators