Expert Large Scale Data Processing Professional

4 days ago

Chennai, Tamil Nadu, India beBeeSoftwareEngineer Full time US$ 1,50,000 - US$ 2,00,000
Job Title: Big Data Engineer

The role of a Software Engineer (Data) is crucial in the development of large-scale data processing systems for LLM research. We are seeking an experienced professional with deep expertise in designing, building, and operating scalable data infrastructure to support distributed computing and data orchestration.



Key Responsibilities:
  • Design and Build Scalable Data Infrastructure: Develop and maintain high-throughput systems for data ingestion, processing, and transformation to support LLM model development.
  • Develop Synthetic Datasets: Utilize state-of-the-art solutions to create synthetic datasets.
  • Collaborate with Research Teams: Work closely with researchers to deliver critical data assets for model development and evaluation.
  • Implement Monitoring and Alerting: Establish monitoring and alerting systems to ensure platform reliability and performance.
  • Build Traceability and Quality Control Systems: Develop systems for traceability, reproducibility, and robust quality control to ensure adherence to industry compliance standards.


Required Skills and Qualifications:
  • 5+ Years Experience: In data infrastructure, ideally supporting high-scale applications or research platforms.
  • Distributed Computing Frameworks: Fluency in distributed computing frameworks.
  • Cloud Infrastructure: Deep familiarity with cloud infrastructure, data storage architectures, and batch + streaming pipelines.
  • Specialized Hardware: Experience with specialized hardware (GPUs, TPUs) computing and GPU cluster.
  • Databases and Storage Systems: Strong knowledge of databases, storage systems, and how architecture choices impact performance at scale.
  • Microservices Architectures: Familiarity with microservices architectures, containerization and orchestration, and both synchronous and asynchronous processing.
  • Performance Optimization: Extensive experience with performance optimization and memory management in high-volume data systems.


Working Conditions:
This role is fully remote, India-based. Compensation and benefits will vary based on background, skills, and experience levels.

  • Large Scale Data Processing Specialist

    3 days ago

    Chennai, Tamil Nadu, India beBeeDataEngineering Full time ₹ 2,00,00,000 - ₹ 2,50,00,000

    GCP Data Engineer PositionWe are seeking a highly skilled data engineer to join our organization.This individual will be responsible for designing, developing, and maintaining large-scale data processing systems on Google Cloud Platform.Expertise in Snowflake data warehousing is required, including SQL, Snowpipe, Streams, and data sharing.Collaboration with...

  • Senior Data Engineer – Large-Scale Solutions Specialist

    1 day ago

    Chennai, Tamil Nadu, India beBeeDataEngineer Full time ₹ 12,00,000 - ₹ 15,00,000

    Job TitleSenior Data Engineer – Large-Scale Solutions SpecialistWe are seeking a highly skilled Senior Data Engineer to join our team. As a key member of our data engineering group, you will be responsible for designing, developing, and optimizing large-scale data processing solutions using PySpark and Python.

  • Expert in Large Scale Cluster Management

    4 days ago

    Chennai, Tamil Nadu, India beBeeKubernetes Full time ₹ 20,00,000 - ₹ 25,00,000

    We are seeking an experienced Kubernetes Expert who will be responsible for designing, implementing, and managing large-scale Kubernetes clusters with a strong focus on performance, security, and reliability.Key Responsibilities :- Design, deploy, and manage highly available Kubernetes clusters across multi-cloud and on-prem environments.- Implement security...

  • Senior Large Scale Data Engineer

    3 days ago

    Chennai, Tamil Nadu, India beBeeDataDeveloper Full time ₹ 20,00,000 - ₹ 25,00,000

    Big Data Developer JobWe are seeking a highly skilled Big Data Developer to join our dynamic team. The ideal candidate will have expertise in designing and implementing large-scale data processing systems using modern data technologies.The role involves designing, developing, and maintaining scalable Big Data solutions using Hadoop ecosystem, Spark, and...

  • Large Scale Distributed Computing Solutions Expert

    1 week ago

    Chennai, Tamil Nadu, India beBeeExpert Full time ₹ 18,00,000 - ₹ 22,00,000

    Job Title:We are seeking a seasoned software development expert to design and build large-scale distributed computing solutions.As a senior developer, you will drive the team's technical direction, strategize and shape our long-term vision and architecture. You will own major deliverables end-to-end, drive roadmaps, and provide technical leadership to the...

  • Senior Large Scale Data Developer

    10 hours ago

    Chennai, Tamil Nadu, India beBeeData Full time US$ 1,25,000 - US$ 1,75,000

    Job Title: Data EngineerJob Description:We are looking for a skilled professional to design, develop and maintain large-scale data systems.Key Responsibilities:Bachelor's or master's degree with 3+ years of experience in Computer Science, Data Science, Engineering, or a related field.Proficient in software development lifecycle and software engineering...

  • Optimizing Large-Scale HPC Platforms

    1 day ago

    Chennai, Tamil Nadu, India beBeeHighperformance Full time ₹ 1,80,00,000 - ₹ 2,50,00,000

    Senior Linux Engineer - Quantitative ResearchWe are seeking an experienced Linux engineer to lead the design, engineering, and automation of large-scale High-Performance Computing (HPC) platforms for quantitative finance and advanced research.Develop and manage scalable HPC environments with a focus on performance optimization.Research and experiment with...

  • DevOps Expert for Large-Scale Infrastructure

    6 days ago

    Chennai, Tamil Nadu, India beBeeDevOps Full time ₹ 90,00,000 - ₹ 1,40,00,000

    Job DescriptionWe are seeking a seasoned DevOps expert to join our infrastructure team. The ideal candidate will have hands-on experience with deploying, automating, maintaining, and managing production systems to ensure high availability and scalability.The successful candidate will be responsible for implementing and managing CI/CD pipelines using...

  • Data Processing Expert

    2 days ago

    Chennai, Tamil Nadu, India beBeeDatabricks Full time ₹ 5,00,000 - ₹ 12,00,000

    Azure Databricks is a cloud-based big data and analytics platform.Job OpportunityWe are seeking a skilled professional to design, develop, and maintain large-scale data processing systems using Azure Databricks.The ideal candidate will have at least 5 years of experience in designing and implementing data pipelines using Azure Data Factory, Azure Databricks,...

  • Senior Big Data Professional

    5 days ago

    Chennai, Tamil Nadu, India beBeeData Full time ₹ 1,50,00,000 - ₹ 2,00,00,000

    Big Data Expert WantedWe're seeking a seasoned Big Data professional with expertise in Java or Python to take on this exciting challenge.The ideal candidate will have a strong background in designing and implementing large-scale data processing systems, coupled with a solid understanding of modern data technologies.Design, develop, and maintain scalable Big...