Apache Nifi

Job Title: Senior Data Engineer - Apache NiFi & Kafka Expert Location: Ahmedabad About the Role: We are seeking a Senior Data Engineer with deep, high-level expertise in Apache NiFi and Apache Kafka. This role demands advanced proficiency — from architecture design to large-scale deployment, optimization, troubleshooting, and security. You will build and manage complex, high-performance data pipelines powering our mission-critical systems. Key Responsibilities:Architect, design, and optimize enterprise-grade NiFi data flows for large-scale ingestion, transformation, and routing.Manage Kafka clusters at scale (multi-node, multi-datacenter setups), ensuring high availability, fault tolerance, and maximum throughput.Create custom NiFi processors and develop advanced flow templates and best practices.Handle advanced Kafka configurations — partitioning, replication, producer tuning, consumer optimization, rebalancing, etc.Implement stream processing using Kafka Streams and manage Kafka Connect integrations with external systems (databases, APIs, cloud storage).Design secure pipelines with end-to-end encryption, authentication (SSL/SASL), and RBAC for both NiFi and Kafka.Proactively monitor and troubleshoot performance bottlenecks in real-time streaming environments.Collaborate with infrastructure teams for scaling, backup, and disaster recovery planning for NiFi/Kafka.Mentor junior engineers and enforce best practices for data flow and streaming architectures. Required Skills and Qualifications:5+ years of hands-on production experience with Apache NiFi and Apache Kafka.Deep understanding of NiFi architecture (flow file repository, provenance, state management, backpressure handling).Mastery over Kafka internals — brokers, producers/consumers, Zookeeper (or KRaft mode), offsets, ISR, topic configurations.Strong experience with Kafka Connect, Kafka Streams, Schema Registry, and data serialization formats (Avro, Protobuf, JSON).Expertise in tuning NiFi and Kafka for ultra-low latency and high throughput.Strong scripting and automation skills (Shell, Python, Groovy, etc.).Experience with monitoring tools: Prometheus, Grafana, Confluent Control Center, NiFi Registry, NiFi Monitoring dashboards.Solid knowledge of security best practices in data streaming (encryption, access control, secret management).Hands-on experience deploying on cloud platforms (AWS MSK, Azure Event Hubs, GCP Pub/Sub with Kafka connectors).Bachelor's or Master’s degree in Computer Science, Data Engineering, or equivalent field. Preferred (Bonus) Skills:Experience with containerization and orchestration (Docker, Kubernetes, Helm).Knowledge of stream processing frameworks like Apache Flink or Spark Streaming.Contributions to open-source NiFi/Kafka projects (a huge plus). Soft Skills:Analytical thinker with exceptional troubleshooting skills.Ability to architect solutions under tight deadlines.Leadership qualities for guiding and mentoring engineering teams.Excellent communication and documentation skills.

**Job Type**: Contractual / Temporary
Contract length: 11 months

Pay: ₹40,000.00 - ₹95,000.00 per month

Schedule:

- UK shift

Work Location: In person