Kafka Engineers

2 days ago

Hyderabad, India CriticalRiver Inc. Full time

Job Title: Kafka Lead

Experience 10+ yrs.

Location: Hyderabad


Scope

The scope of work involves the finalization and optimization of the Kafka Streams solution for the IEE job, focusing on achieving production-level performance, scalability, observability, and cost efficiency. The key objectives are outlined as follows:

Cost Optimization

  • Estimate and benchmark the current cost.
  • Prioritize iterative performance improvements with a focus on cost efficiency.

Functional Enhancements

  • Ensure functional parity with the main branch.
  • Upgrade library versions, including Java, Spring Boot, Spring, Kafka Streams, etc.
  • Eliminate Hibernate dependencies.
  • Remove unnecessary libraries and redundant functionalities.

Integration and Optimization

  • Optimize boot-up time, including multi-threaded loading of data sources (e.g., whitelists, lookups).
  • Continuously identify and resolve performance bottlenecks.
  • Logging and Metrics
  • Implement detailed logging and metrics collection for monitoring and analysis.

Production Readiness

Prepare for production deployment by addressing the following areas:

  • Containerization of the solution
  • Development of runbooks
  • Creation of monitoring dashboards and alert mechanisms
  • Deployment scripts for streamlined rollouts
  • Auto-scalable container orchestration
  • Cluster sizing recommendation tool
  • Auto-healing mechanisms for handling: slow or erroneous policies and invalid events

Quality Assurance

  • Develop comprehensive automated tests, including: Unit tests, integration tests with data mocking and policy types with violation handling
  • Execute extensive unit tests, Sonar scans, and security scans.
  • Implement CI/CD pipelines for continuous deployment to Dev and Performance environments based on quality gate criteria.

Documentation

  • Create comprehensive documentation, including: Design documents, Code walkthroughs, Runbooks for operational guidance

Deliverables & Milestones

Optimized the (Kafka Streams) codebase with production-level performance enhancements, implemented monitoring and alerting and AWS resources, and provided comprehensive documentation for production deployment, including architecture details and operational procedures like runbooks. Additionally, conducted end-to-end testing with recommendations for any further improvements. The project will be executed in phases with the following key milestones:

  • Weekly Progress Reports: Regular updates on the project status and achievements.
  • Bi-Weekly Demos: Demonstrations every two weeks to showcase progress and gather feedback.
  • Ongoing Cost Optimization Monitoring: Continuous visibility into the current state of the Cost Optimization estimates
  • Functional Testing: Conduct parallel testing for functional parity with the existing Spark-based IEE job on a selected tenant within the first two months.
  • Multi-Tenant Testing: Perform multiple parallel tests across different tenants prior to final sign-off



Why Join Us?

At CriticalRiver, you’ll have the opportunity to lead and shape a key practice within our organization, work with high-caliber clients, and make a meaningful impact on their digital transformation journey. We are committed to fostering a culture of innovation, growth, and collaboration where your ideas and leadership will thrive.

CriticalRiver Inc. is an employee-owned technology consulting and IT services firm that empowers its workforce to have a direct stake in the company's growth and success. This ownership model fosters a collaborative and committed culture where employees are motivated to deliver high-quality solutions and services for clients. By giving its team members a shared interest in the company's achievements, CriticalRiver enhances innovation, accountability, and a strong sense of community across all levels of the organization.

  • Kafka Engineer

    2 weeks ago

    hyderabad, India Health Catalyst Full time

    About Us The healthcare industry is the next great frontier of opportunity for software development, and Health Catalyst is one of the most dynamic and influential companies in this space. We are working on solving national-level healthcare problems, and this is your chance to improve the lives of millions of people, including your family and friends....

  • Kafka Engineer

    3 weeks ago

    hyderabad, India Health Catalyst Full time

    About UsThe healthcare industry is the next great frontier of opportunity for software development, and Health Catalyst is one of the most dynamic and influential companies in this space. We are working on solving national-level healthcare problems, and this is your chance to improve the lives of millions of people, including your family and friends. Health...

  • Kafka Engineer

    3 weeks ago

    Hyderabad, India Health Catalyst Full time

    About UsThe healthcare industry is the next great frontier of opportunity for software development, and Health Catalyst is one of the most dynamic and influential companies in this space. We are working on solving national-level healthcare problems, and this is your chance to improve the lives of millions of people, including your family and friends. Health...

  • Kafka engineer

    3 weeks ago

    Hyderabad, India Health Catalyst Full time

    About UsThe healthcare industry is the next great frontier of opportunity for software development, and Health Catalyst is one of the most dynamic and influential companies in this space. We are working on solving national-level healthcare problems, and this is your chance to improve the lives of millions of people, including your family and friends. Health...

  • Kafka Engineer

    1 month ago

    hyderabad, India Health Catalyst Full time

    About UsThe healthcare industry is the next great frontier of opportunity for software development, and Health Catalyst is one of the most dynamic and influential companies in this space. We are working on solving national-level healthcare problems, and this is your chance to improve the lives of millions of people, including your family and friends. Health...

  • Kafka engineer

    1 month ago

    Hyderabad, India Health Catalyst Full time

    About Us The healthcare industry is the next great frontier of opportunity for software development, and Health Catalyst is one of the most dynamic and influential companies in this space. We are working on solving national-level healthcare problems, and this is your chance to improve the lives of millions of people, including your family and friends....

  • Kafka Engineer

    1 month ago

    Hyderabad, India Health Catalyst Full time

    About Us The healthcare industry is the next great frontier of opportunity for software development, and Health Catalyst is one of the most dynamic and influential companies in this space. We are working on solving national-level healthcare problems, and this is your chance to improve the lives of millions of people, including your family and friends....

  • Kafka Engineer

    1 month ago

    hyderabad, India Health Catalyst Full time

    About Us The healthcare industry is the next great frontier of opportunity for software development, and Health Catalyst is one of the most dynamic and influential companies in this space. We are working on solving national-level healthcare problems, and this is your chance to improve the lives of millions of people, including your family and friends....

  • Kafka Architect

    3 weeks ago

    Hyderabad, Telangana, India Alignity Solutions Full time

    About the RoleWe are looking for a highly skilled Kafka Architect to lead the design, development, and deployment of our Kafka-based data pipeline. As a Kafka Architect, you will be responsible for designing and implementing scalable, fault-tolerant, and high-performance Kafka clusters.Key Responsibilities:Kafka Cluster Design: Design and implement scalable...

  • Kafka Architect

    1 month ago

    Hyderabad, Telangana, India ScaleneWorks Full time

    ScaleneWorks - Confluent Kafka DeveloperJob Summary: We are seeking a highly skilled Confluent Kafka Developer to join our team at ScaleneWorks. The ideal candidate will have a strong background in developing high-performance data pipelines and streaming applications using Confluent Kafka.Key Responsibilities:Design, develop, and deploy Confluent Kafka...

  • Kafka Developer

    3 months ago

    Hyderabad, India Health Catalyst Full time

    About UsThe healthcare industry is the next great frontier of opportunity for software development, and Health Catalyst is one of the most dynamic and influential companies in this space. We are working on solving national-level healthcare problems, and this is your chance to improve the lives of millions of people, including your family and friends. Health...

  • Kafka Developer

    3 months ago

    Hyderabad, India Health Catalyst Full time

    About UsThe healthcare industry is the next great frontier of opportunity for software development, and Health Catalyst is one of the most dynamic and influential companies in this space. We are working on solving national-level healthcare problems, and this is your chance to improve the lives of millions of people, including your family and friends. Health...

  • Kafka Developer

    3 months ago

    Hyderabad, India Health Catalyst Full time

    About UsThe healthcare industry is the next great frontier of opportunity for software development, and Health Catalyst is one of the most dynamic and influential companies in this space. We are working on solving national-level healthcare problems, and this is your chance to improve the lives of millions of people, including your family and friends. Health...

  • Kafka Developer

    3 months ago

    Hyderabad, India Health Catalyst Full time

    About Us The healthcare industry is the next great frontier of opportunity for software development, and Health Catalyst is one of the most dynamic and influential companies in this space. We are working on solving national-level healthcare problems, and this is your chance to improve the lives of millions of people, including your family and friends....

  • Kafka SME

    2 months ago

    Hyderabad, India Alignity Solutions Full time

    Do you love a career where you Experience, Grow & Contribute at the same time, while earning at least 10% above the market? If so, we are excited to have bumped onto you.Learn how we are redefining the meaning of work, and be a part of the team raved by Clients, Job-seekers and Employees.Jobseeker Video Testimonials Employee Glassdoor ReviewsIf you are...

  • Kafka Infrastructure Specialist

    3 weeks ago

    Hyderabad, Telangana, India Alignity Solutions Full time

    Kafka Infrastructure Specialist RoleWe are seeking a highly skilled Kafka Subject Matter Expert (SME) to lead the architecture management and scaling of Kafka environments in 24x7 production settings. The ideal candidate will have a deep understanding of Apache Kafka, a proven track record of resolving complex production and non-production issues, and strong...

  • Kafka Engineers

    1 day ago

    Hyderabad, India CriticalRiver Inc. Full time

    Job Title:Kafka LeadExperience10+ yrs.Location:HyderabadScopeThe scope of work involves the finalization and optimization of the Kafka Streams solution for the IEE job, focusing on achieving production-level performance, scalability, observability, and cost efficiency. The key objectives are outlined as follows:Cost OptimizationEstimate and benchmark the...

  • Kafka Architecture Specialist

    1 month ago

    Hyderabad, Telangana, India Alignity Solutions Full time

    Job OverviewWe are seeking a highly skilled Kafka Subject Matter Expert (SME) to lead the architecture, management, and scaling of Kafka environments in 24x7 production settings. As a key member of our team, you will work closely with cross-functional teams, provide technical guidance to junior engineers, and ensure the high availability, scalability, and...

  • Centralized Kafka Specialist

    3 weeks ago

    hyderabad, India ValueLabs Full time

    Dear Aspirants, Greetings from ValueLabs !! We are seeking a skilled Centralized Kafka Specialist who can join our team immediately. Primary Skills : Centralized Kafka Specialist Experience: 9- 12 years Location: Hyderabad Notice period: Immediate -7 days Joiners Key Responsibilities: Design and implement Kafka clusters and topologies to meet the...

  • Centralized Kafka Specialist

    4 weeks ago

    Hyderabad, India ValueLabs Full time

    Dear Aspirants,Greetings from ValueLabs !!We are seeking a skilled Centralized Kafka Specialist who can join our team immediately.Primary Skills : Centralized Kafka SpecialistExperience: 9- 12 yearsLocation: HyderabadNotice period: Immediate -7 days JoinersKey Responsibilities:Design and implement Kafka clusters and topologies to meet the organization's...