High-Performance Distributed Systems Leader

We are seeking a seasoned software professional to lead our efforts in delivering high-performance distributed systems.

This is an exciting opportunity to work on cutting-edge projects that involve designing, developing, and deploying scalable solutions that can handle billions of cyber security events processing per day.

You will play a key role in ensuring that our real-time data pipelines and services meet enterprise-grade service level agreements (SLAs).

1. Develop and execute performance strategies across distributed systems, including Hadoop, Spark, Kafka, Elasticsearch/OpenSearch, Big Data Components, and APIs for each release.
2. Create realistic performance test scenarios for data pipelines and microservices based on production-like workloads.
3. Proactively identify bottlenecks, resource contention, and latency issues using tools such as JMeter, Spark UI, Kafka Manager, Elastic Monitoring, and App Dynamics.
4. Provide deep-dive analysis and recommendations on tuning and scaling Spark jobs, Kafka topics/partitions, ES queries, and API endpoints.
5. Collaborate with developers, architects, and infrastructure teams to integrate performance feedback into design and implementation.
6. Simplify and benchmark real-time and batch data flow at scale using synthetic and production-like datasets and own this framework end-to-end for synthetic data generator.
7. Lead the initiative to build a performance testing framework that integrates with CI/CD pipelines.
8. Establish and track SLAs for throughput, latency, CPU/memory utilization, and Garbage collection.
9. Create performance dashboards and visualization using Prometheus/Grafana, Kibana, or equivalent.
10. Document performance test findings and create technical reports for leadership and engineering teams.
11. Recommend performance optimization to Dev and Platform groups.

• Bachelor's degree in computer science, Engineering, or related field.
• 8+ years of overall experience in distributed systems and backend performance engineering.
• 4+ years of JAVA development experience with Microservices architecture.
• Proficient in scripting (Python, Bash) for automation and test data generation.
• 4+ years of hands-on experience with Apache Spark performance tuning, memory management, and DAG optimization.
• 3+ years of experience with Kafka topic optimization, producer/consumer tuning, and lag monitoring.
• 3+ years of experience with Elasticsearch/OpenSearch query profiling, indexing strategies, and cluster optimization.
• 3+ years of experience with performance testing tools such as JMeter or similar.
• Excellent programming and designing skills and Hands-on experience on Spring, Hibernate.
• Deep understanding of middleware and microservices performance including REST APIs.
• Strong knowledge of profiling, debugging, and observability tools (e.g., Spark UI, Athena, Grafana, ELK).
• Experience designing and running benchmarks at scale for high-throughput environments in PBs.
• Experience with containerized workloads and performance testing in Kubernetes/Docker environments.
• Solid understanding of cloud-native architecture (OCI) and distributed systems design.
• Strong knowledge of Linux operating systems and performance-related improvements.
• Familiarity with CI/CD integration for performance testing (e.g., Jenkins, GitHub).
• Knowledge of data lake architecture, caching solutions, and message queues.
• Strong communication skills and experience influencing cross-functional engineering teams.

• Prior experience in any analytics platform on Big Data would be beneficial.