Lead Large-Scale Data Engineering Projects

2 days ago

dindigul, India beBeeDataEngineer Full time

Senior Data Engineer JobJob Title: Senior Data Engineer – Python Expert (Freelance)Location: Remote / HybridEmployment Type: Contract/ FreelanceRole SummaryWe are seeking a seasoned Senior Data Engineer to design, develop, and own robust data pipelines that power our large language model development. As a senior individual contributor, you will be the team's expert on data ingestion, processing, and quality for all AI training.Your primary mission is to build scalable, automated systems that transform massive raw datasets into pristine, model-ready formats. While your focus will be on data engineering, your expertise will be valued in collaborating on model training runs and experiments. You're the perfect fit if you're a Python expert who thrives on solving large-scale data challenges and enjoys working at the intersection of data engineering and machine learning.Main Responsibilities1. Pipeline Architecture & Development: Design, develop, and own ETL/ELT pipelines in Python for ingesting and processing terabyte-scale text datasets.2. Data Quality Management: Implement rigorous data cleaning, deduplication, filtering, and normalization strategies to ensure high-quality data for model training.3. Data Transformation & Structuring: Efficiently structure and format diverse datasets for consumption by LLM training frameworks.4. Collaboration & Support: Work closely with AI researchers and ML engineers to understand data requirements, define metrics, and support the model training lifecycle.5. Optimization & Maintenance: Continuously optimize data processing workflows for speed, cost, and reliability.6. Secondary ML Support: Occasionally assist in launching, monitoring, and debugging data-related issues during model training runs.Required Skills & Qualifications8+ years of professional experience in data engineering, data processing, or backend software engineering.Expert-level proficiency in Python and its data ecosystem (e.g., Pandas, NumPy, Dask, Polars).Proven experience building and maintaining large-scale data pipelines.Deep understanding of data structures, data modeling, and software engineering best practices (Git, CI/CD, testing).Experience handling diverse data formats (JSON, CSV, XML, Parquet) at scale.Excellent problem-solving skills and attention to detail.Strong communication & collaboration skills, with experience working in a team environment.Preferred Qualifications (Nice-to-Haves)Hands-on experience with the data preprocessing pipeline for an LLM (e.g., LLaMA, BERT, GPT-family).Strong experience with big data frameworks like Apache Spark or Ray.Experience with Hugging Face libraries (Transformers, Datasets, Tokenizers).Familiarity with ML frameworks like PyTorch or TensorFlow.Proficiency with cloud platforms (AWS, GCP, Azure) and their data/storage services.Why Join UsCutting-edge AI & ML projectsInnovative team cultureCompetitive compensation with continuous learning opportunities

  • Optimize Large-Scale Data Architectures

    1 week ago

    dindigul, India beBeeDataScience Full time

    Lead Data Solutions ArchitectThis is an exciting opportunity for a skilled data solutions architect to join our team. As a lead data solutions architect, you will be responsible for designing and implementing large-scale data architectures that optimize data extraction, transformation, and loading from various sources using Azure data ingestion and...

  • Developer and Designer for Large-Scale Projects

    4 days ago

    dindigul, India beBeeTechnical Full time

    Job Title: Technical FreshersFusion Practices is a leading IT consultancy delivering HR and finance transformations across various sectors.Our company has expertise in Oracle Cloud ERP, HR & Payroll, and has won several awards, including the ERP Innovation of the Year award.As a Technical Fresher, you will help enterprises adapt and modernize their business...

  • Large Scale Exhibition Project Leader

    2 weeks ago

    dindigul, India beBeeExhibition Full time

    Exhibition Project Manager RoleWe are seeking a seasoned professional to lead our team in organizing large-scale business-to-business exhibitions and trade shows.Event Execution: Lead a cross-functional team to deliver high-impact industry exhibitions that meet revenue targets and exceed client expectations.Sales Strategy: Develop and implement effective...

  • Large Scale AI Engineer

    2 weeks ago

    dindigul, India beBeeMachineLearning Full time

    Key Performance IndicatorsWe are seeking a highly skilled engineer to join our team as a Machine Learning Observability Platform Engineer. This role involves building and maintaining large-scale, reliable ML systems that power critical insights across enterprise environments.The successful candidate will help design and enhance our open-source observability...

  • Lead Data Engineer

    1 week ago

    Dindigul, India Ironbook AI Full time

    We are seeking an experienced and driven Lead Data Engineer to spearhead the design and development of a modern, cloud-native data warehouse on AWS. This role is critical to building a scalable, secure, and efficient data platform that supports analytics, reporting, and AI use cases across the organization. The ideal candidate is both technically hands-on...

  • Senior Data Architect

    2 weeks ago

    dindigul, India beBeeDataEngineer Full time

    Data Engineer – Python ExpertJob Title: Data Engineer – Python Expert (Freelance Role)Location: Remote/HybridEmployment Type: Contract/FreelanceJob Description:We seek a seasoned Senior Data Engineer to design, develop, and own robust data pipelines that power our large language model development. As a senior Individual Contributor, you will be the...

  • Leading Data Engineering Expert

    1 week ago

    dindigul, India beBeeDataEngineer Full time

    Data Engineer OpportunityWe are seeking a skilled data engineer to join our team.The ideal candidate will have extensive experience with SQL, Ab-Initio, Teradata, and Google Cloud Platform (GCP), as well as strong expertise in designing, developing, and optimizing large-scale data pipelines. They will be responsible for implementing scalable, reliable, and...

  • Data Migration Project Lead

    2 weeks ago

    dindigul, India beBeeBusiness Full time

    Business Intelligence Specialist Migrate large datasets from legacy systems to S4 HANA data objects, ensuring seamless integration and minimizing disruption to business operations.Portfolio Management: Oversee the mapping and transformation design for various data objects across different workstreams.Offer to Cash (O2C): Coordinate the flow of goods and...

  • Cloud Data Engineer

    1 week ago

    dindigul, India beBeeDataEngineer Full time

    Job Title: Cloud Data EngineerOverview:This is a challenging role that requires expertise in data engineering to build, deploy, and maintain large-scale data systems on the Google Cloud Platform.Responsibilities:Perform ongoing support activities and project efforts as needed.Triage identified issues across Account source platforms, integrations, and...

  • Executive Project Lead

    2 weeks ago

    dindigul, India beBeeProject Full time

    Job DescriptionWe are seeking a seasoned project management professional to lead our complex projects. As Senior Project Manager, you will oversee the entire project lifecycle, ensuring successful delivery and alignment with business objectives.This role involves managing cross-functional teams, coordinating with stakeholders, and driving process...