Data Engineer

24 hours ago

Kollam, India Aceolution Full time

Job Title: Data Engineer – Python Expert(Freelance Role)Location: Remote / HybridEmployment Type: Contract/ FreelanceRole Summary We are looking for a seasoned Senior Data Engineer to architect, build, and own the data pipelines that power our large language model (LLM) development. As a senior Individual Contributor (IC), you will be the team's expert on data ingestion, processing, and quality for all AI training. Your primary mission is to build scalable, automated systems that transform massive, raw datasets into pristine, model-ready formats. While your focus will be on data engineering, your expertise will be valued in collaborating on model training runs and experiments. You're the perfect fit if you are a Python expert who thrives on solving large-scale data challenges and enjoys working at the intersection of data engineering and machine learning.Key Responsibilities Architect & Build: Design, develop, and own robust, scalable, and automated ETL/ELT pipelines in Python for ingesting and processing terabyte-scale text datasets. Data Quality: Implement rigorous data cleaning, deduplication, filtering, and normalization strategies. Define and enforce data quality standards to ensure the highest integrity for model training.Data Transformation: Efficiently structure and format diverse datasets (JSON, Parquet, etc.) for consumption by LLM training frameworks. Collaboration: Work closely with our team of AI researchers and ML engineers to understand data requirements, define metrics, and support the model training lifecycle. Optimization: Continuously optimize data processing workflows for speed, cost, and reliability.ML Support (Secondary): Occasionally assist in launching, monitoring, and debugging data-related issues during model training runs.Required Qualifications 8+ years of professional experience in data engineering, data processing, or backend software engineering. Expert-level proficiency in Python and its data ecosystem (e.g., Pandas, NumPy, Dask, Polars). Proven experience building and maintaining large-scale data pipelines. Deep understanding of data structures, data modeling, and software engineering best practices (Git, CI/CD, testing). Experience handling and parsing diverse data formats (JSON, CSV, XML, Parquet) at scale. Excellent problem-solving skills and a meticulous attention to detail. Strong communication and collaboration skills, with experience working in a team environment.Preferred Qualifications (Nice-to-Haves) Hands-on experience with the data preprocessing pipeline for an LLM (e.g., LLaMA, BERT, GPT-family). Strong experience with big data frameworks like Apache Spark or Ray. Experience with Hugging Face libraries (Transformers, Datasets, Tokenizers). Familiarity with ML frameworks like PyTorch or TensorFlow. Proficiency with cloud platforms (AWS, GCP, Azure) and their data/storage services.Why Join UsOpportunity to lead cutting-edge AI and ML projects.Collaborative and innovative team culture.Competitive compensation with continuous learning opportunities.📩 If you are interested, please share your updated CV to sharmila@aceolution.com along with your expected rate per hour.

  • Data Engineer

    4 weeks ago

    Kollam, India Magma Consultancy Full time

    Location: RemoteType: Part-time (20–25 hours per week)About the RoleWe are seeking a skilled Data Engineer to support our growing data operations on a part-time basis. The ideal candidate has hands-on experience in building, optimizing, and maintaining data pipelines and architectures. You’ll work closely with analysts, developers, and business teams...

  • Data Engineer

    4 weeks ago

    Kollam, India Digivance Solutions Full time

    Position: Data EngineerExperience: 5–10 YearsLocation: Chennai, Bengaluru, Pune, Hyderabad, Mumbai, Delhi NCR(candidates will be required to work at any of these locations in hybrid mode)Key ResponsibilitiesCollaborate with business and technology stakeholders to understand current and future data requirements.Design, build, and maintain reliable,...

  • Data Engineer

    4 weeks ago

    Kollam, India Digivance Solutions Full time

    Position: Data EngineerExperience: 5–10 YearsLocation: Chennai, Bengaluru, Pune, Hyderabad, Mumbai, Delhi NCR(candidates will be required to work at any of these locations in hybrid mode)Key ResponsibilitiesCollaborate with business and technology stakeholders to understand current and future data requirements.Design, build, and maintain reliable,...

  • Data Engineer

    1 week ago

    Kollam, India Digitalzone Full time

    About the RoleAs a Data Engineer, you will design, build, and optimize data pipelines and real-time systems that power AI-driven decisioning and analytics.ResponsibilitiesDevelop and maintain scalable ETL/ELT pipelines using Python, Airflow, and dbt Build and optimize real-time streaming pipelines using Kafka, RabbitMQ, Spark and event-driven microservices...

  • Senior Data Engineer

    16 hours ago

    Kollam, India Tata Consultancy Services Full time

    Role**: Senior Data EngineerRequired Technical Skill Set: Senior Data EngineerDesired Experience Range: 7 - 12 yrsNotice Period: Immediate to 90Days onlyLocation of Requirement: BangaloreWe are currently planning to do a Virtual InterviewJob Description:Primary Skills (Java, Python, etc.)Python, Scala/Java, Spark, SQL, Snowflake, BDD, Databricks, AWS Key...

  • Data engineer

    1 day ago

    Kollam, India Quantiphi Full time

    Role: Associate Architect - Data Experience Level: 6.5 to 12 Years Work location: Mumbai, Bangalore, Trivandrum Role & Responsibilities: 6+ years of experience in building and managing Data Lakes, Data Warehouse, Data Integration, Data Migration and Business Intelligence/Artificial Intelligence solutions on GCP Cloud. Ability to understand business...

  • Data Engineer

    4 days ago

    Kollam, India Add Min Web World Full time

    Job Title: Data Engineer (Qlik Experience)Location: Mohali (On-site)Experience: Minimum 2 YearWorking Days: 5 Days a WeekJob Description:We are looking for a dedicated Data Engineer who has hands-on experience working with Qlik and other data tools. The ideal candidate should have a strong understanding of data processing, integration, and reporting, along...

  • Data Engineer

    3 weeks ago

    Kollam, India Taiyō.AI Full time

    About UsTaiyo.AI is the world's first infrastructure intelligence platform. We are building the largest universaland industry standard database of opportunities (tenders, projects, news) and threats (economy,climate, geopolitics, finance, logistics, etc.) for real assets. Taiyo.AI has been instrumental in shapinghow infrastructure companies (infra investors,...

  • Python API + Data Engineer

    2 weeks ago

    Kollam, India Digitrix Software LLP Full time

    Experience : 5 to 8 yearsJob description:Python API + Data EngineerWe are looking for aself-motivated Python Data Engineerwith strong skills inbackend/API development and large-scale data processing , who can work independently and thrive in a fast-paced, agile environment.Key ResponsibilitiesDesign, develop, and optimizedata pipelinesusingPySpark, AWS Glue,...

  • Data integration reliability and devops engineer

    4 weeks ago

    Kollam, India RapidBrains Full time

    Job Title: Data Integration Reliability & Dev Ops Engineer Experience: 7–10 Years Location: Remote Contract Duration: Long Term Work Time: Normal IST (with 2–3 hours overlap as per client need)We are seeking a highly skilled Data Integration Reliability & Dev Ops Engineer to join our team. This role focuses on maintaining the operational reliability and...