Distributed Data Processing Specialist

2 days ago

Hyderabad, Telangana, India beBeeDataEngineer Full time ₹ 1,50,00,000 - ₹ 2,50,00,000
Real-Time Data Engineer

Our organization is seeking a skilled Real-Time Data Engineer to design and build high-throughput, low-latency real-time data processing pipelines using Apache Kafka. The ideal candidate will have a deep understanding of Kafka internals, including brokers, partitions, consumer groups, offsets, replication, and schema management.

Responsibilities:
  • Design and maintain scalable and fault-tolerant streaming pipelines for real-time data processing.
  • Develop robust Kafka producers and consumers using Java, adhering to performance and reliability standards.
  • Collaborate with DevOps teams to automate CI/CD pipelines, deployment, and infrastructure provisioning for Kafka components.
  • Implement security best practices, including encryption, authentication, and access control for Kafka clusters.
  • Monitor and troubleshoot Kafka cluster health, latency, throughput, and message delivery guarantees.
Requirements:
  • 5-7 years of experience in backend or data engineering, with a strong focus on Apache Kafka.
  • Proficiency in Java for building Kafka producers/consumers and integration with backend services.
  • Deep understanding of Kafka internals (brokers, partitions, consumer groups, offsets, replication, etc.).
  • Experience with schema management using Confluent Schema Registry, Avro, or Protobuf.
  • Experience with CI/CD pipelines for Kafka component deployment and integration.
Desirable Skills:
  • Experience with Kafka Streams, ksqlDB, or Apache Flink for stream processing.
  • Familiarity with Kafka Connect and connectors for data ingestion and export.
  • Understanding of container orchestration platforms like Kubernetes or OpenShift.

This is an exciting opportunity for a motivated and experienced engineer to join our team and contribute to the development of our real-time data processing capabilities. If you are passionate about working with distributed systems and have a strong background in data engineering, we encourage you to apply.

  • Data Processing Specialist

    2 days ago

    Hyderabad, Telangana, India beBeeDataEngineer Full time ₹ 1,04,000 - ₹ 1,30,878

    Job OpportunityAre you passionate about working with large data sets? Do you have experience with distributed systems and pipelines?We are seeking a Data Engineer to join our team. As a Data Engineer, you will be responsible for designing, building, and maintaining large-scale data processing systems.10+ yrs: Proven experience in designing and implementing...

  • Distributed Messaging Specialist

    6 days ago

    Hyderabad, Telangana, India beBeeKafka Full time ₹ 8,00,000 - ₹ 15,00,000

    Job Title: Distributed Messaging SpecialistWe are looking for a talented Distributed Messaging Specialist to join our team. The ideal candidate will have expertise in designing, implementing, and maintaining distributed messaging systems.Key Responsibilities:Design and implement high-performance distributed messaging systems using Kafka.Develop and maintain...

  • Senior Data Engineer

    4 days ago

    Hyderabad, Telangana, India beBeeData Full time ₹ 13,93,311 - ₹ 25,11,156

    Job Opportunity:We seek a seasoned data engineer to spearhead the design and implementation of scalable data processing solutions utilizing PySpark. The ideal candidate will have 7+ years of experience in distributed data processing, with expertise in Reltio MDM and its applications.The successful candidate will be responsible for developing and maintaining...

  • Data Transformation Specialist

    12 hours ago

    Hyderabad, Telangana, India beBeeDataTransformation Full time ₹ 1,50,00,000 - ₹ 2,00,00,000

    Job Title: Data Transformation SpecialistWe are seeking a highly skilled Data Transformation Specialist to join our team.As a Data Transformation Specialist, you will be responsible for designing, implementing, and optimizing scalable data pipelines and transformations using Palantir Foundry. The ideal candidate has expertise in Python, PySpark, and SQL,...

  • Data Product Owner, Distribution Data Office

    4 weeks ago

    Hyderabad, Telangana, India Invesco Full time

    Job DescriptionYou will be responsible for:- Executes and drives distribution data deliverables within the defined overall data strategy and roadmap (functional analyses, requirements gathering and translation to specific data deliverables, ensuring adherence to architectural & technical standards, etc.) as related to their projects- Partners with business...

  • Operational Distribution Specialist

    6 days ago

    Hyderabad, Telangana, India beBeeOperations Full time ₹ 90,00,000 - ₹ 1,20,00,000

    Job Title: Operations Manager - Distribution NetworkAbout the RoleThe External Operations Manager plays a pivotal role in the operational execution of clinical distribution, focusing on the external aspects of the clinical trial distribution supply chain. This individual oversees all outsourcing activities related to clinical distribution and acts as the key...

  • Data Engineering Specialist

    4 days ago

    Hyderabad, Telangana, India beBeeDataEngineer Full time ₹ 1,50,00,000 - ₹ 2,00,00,000

    Data Engineering SpecialistWe're seeking an accomplished Data Engineer to spearhead the development of scalable data pipelines, AI-driven workflows, and data-driven automation using Databricks, AWS EMR, and LangChain.The ideal candidate will have a strong background in data engineering and LLM integration, with expertise in PySpark, ETL design, and...

  • Distributed Messaging System Expert

    1 week ago

    Hyderabad, Telangana, India beBeeKafka Full time ₹ 15,00,000 - ₹ 25,00,000

    Kafka Infrastructure SpecialistThe role of Kafka Infrastructure Specialist is to ensure the smooth operation of our distributed messaging system. This involves designing, developing, and executing tests to guarantee that our Kafka systems meet the required standards.Key Responsibilities:Develop a strong understanding of Kafka concepts, including topics,...

  • Senior Data Engineer

    3 days ago

    Hyderabad, Telangana, India Enable Data Full time ₹ 15,00,000 - ₹ 20,00,000 per year

    Experience Required: 8+YearsMode of work: RemoteSkills Required: Azure DataBricks, Eventhub, Kafka, Architecture,Azure Data Factory, Pyspark, Python, SQL, SparkNotice Period : Immediate Joiners/ Permanent/Contract role (Can join within September 15th 2025)Design, develop, and maintain scalable and robust data solutions in the cloud using Apache Spark and...

  • JDE Sales and Distribution

    4 days ago

    Hyderabad, Telangana, India Birlasoft Limited Full time

    Job Description- Lead the Distribution team by conducting meetings/interviews and facilitating large group/cross-functional sessions with stakeholders.- Writes functional and data requirements specifications and develops business process and data models that define the business needs.- Leads the Integration testing effort, including development of test...