Data Engineer

2 weeks ago

Remote, India Prospus Consulting Pvt. Ltd. Full time

Job Title: Data Engineer

Location: Remote

Employment Type: Full-time

About the Role We are looking for a highly skilled Data Engineer with strong expertise in data ingestion, transformation and indexing pipelines for AI agents. The ideal candidate will have experience designing scalable data pipelines, implementing chunking strategies, and ensuring vector DB quality and optimal retrieval for AI agents.

Key Responsibilities

  • ETL & Data Feeds: Manage, optimize, and monitor ETL workflows and data ingestion pipelines.
  • Chunking Strategies for RAG: Design and implement effective document chunking methodologies to improve retrieval quality.
  • Vector Database Management: Create and maintain metadata enrichment and indexing logic; monitor and fine-tune vector DB performance.
  • Pipeline Scalability: Build and maintain scalable data pipelines using tools like Pandas, Spark, and Airflow.
  • Logging & Analytics: Develop pipelines to track retrieval success/failure and feed insights back into model optimization.
  • Data Privacy & Compliance: Ensure all pipelines adhere to data privacy regulations and organizational compliance standards.
  • Troubleshooting: Identify and resolve data-related root causes impacting system responses.

Required Skills & Qualifications

  • Strong proficiency in Pandas, Spark, Airflow, or similar data engineering tools.
  • Proven expertise in Vector Databases (e.g., Pinecone, Weaviate, Milvus, FAISS).
  • Hands-on experience with chunking strategies for RAG pipelines.
  • Solid understanding of scalable ETL/data pipelines.
  • Experience with metadata management and indexing strategies.
  • Strong debugging, optimization, and performance tuning skills.
  • Knowledge of data privacy laws (GDPR, HIPAA, etc.) and compliance practices.

Preferred Qualifications

  • Experience with ETL and ML data pipelines.
  • Familiarity with chunking and metadata strategies for RAG.
  • Proficiency in Python and SQL for data transformation and querying.

Job Type: Full-time

Pay: ₹501, ₹1,244,400.38 per year

Benefits:

  • Paid sick time
  • Paid time off
  • Provident Fund
  • Work from home

Work Location: Remote

  • AWS/Snowflake Support Engineer

    1 week ago

    Remote, India Data Engineer Academy LLP Full time ₹ 10,000 - ₹ 1,50,000 per year

    We're Hiring: AWS/Snowflake Support Engineer (SME Role)Remote | Full Time | 7AM EST to 3PM EST| Pay: $ /MonthlyAbout the OpportunityThis role is a blend of a Subject Matter Expert (SME) and support engineer. The primary responsibility is to clear project issues for students while working on multiple projects involving AWS, Snowflake, and dbt. The engineer...

  • sme sql

    6 days ago

    Remote, India Data Engineer Academy LLP Full time ₹ 76,800 - ₹ 96,000 per year

    We're Hiring: SQL Subject Matter Expert (SME) – Mock Interviews & Mentoring (Part-Time)Remote (India) | Work From Home | Flexible Timings | Pay: $600 – $800 / 80 hoursAbout the OpportunityData Engineer Academy is looking for an experienced SQL Subject Matter Expert (SME) to conduct mock interviews and provide mentorship to aspiring data professionals....

  • Senior Azure Data Engineer

    2 weeks ago

    Remote, India Data PlatformExperts Full time ₹ 5,00,000 - ₹ 15,00,000 per year

    Job Description:We are looking for a skilled and motivated AI Azure Data Engineer with strong expertise in Microsoft Azure data services. The ideal candidate must have hands-on experience with Databricks or Microsoft Fabric and hold a valid certification in either. You will be responsible for designing, developing, and deploying scalable data pipelines,...

  • Subject Matter Expert Specialist

    1 week ago

    Remote, India Data Engineer Academy LLP Full time ₹ 7,20,000 - ₹ 12,00,000 per year

    We're Hiring: Subject Matter Expert (SME) – Snowflake & Databricks on AWS CloudRemote | Part Time | Flexible Timings | Pay: ₹600 – ₹1000/HourAbout the OpportunityWe are seeking a highly experienced SME with extensive expertise in Snowflake, Databricks, and AWS cloud platforms.What We're Looking ForMinimum 10 years of experience in cloud data...

  • GIS Data Intern

    6 days ago

    Remote, India Engineer Philosophy Web Services Pvt Ltd Full time ₹ 88,800 - ₹ 14,40,000 per year

    Job Title: GIS Data Intern (Freshers Welcome)Company: Engineer Philosophy Web Services Pvt. Ltd.Location: Indore, Madhya Pradesh (Remote)Job Type: Internship / Full-timeJob Description:Engineer Philosophy Web Services Pvt. Ltd. is seeking a motivated and detail-oriented GIS Data Intern to join our Indore-based team. This internship provides hands-on...

  • Expert AI Data Rater

    2 weeks ago

    remote, India Welo Data Full time

    Expert AI Data Rater, English (India) — Shape Global Digital Ads QualityLocation: Remote – Based in India | Contract: Full-Time, Fixed-Term | Start Date: ASAPAt Welo Data, we're not just rating ads; we're actively defining the future of digital content and AI experiences. As a global leader in localization and data transformation, we operate one of the...

  • Expert AI Data Rater

    3 days ago

    Remote, India Welo Data Full time ₹ 1,20,000 - ₹ 2,40,000 per year

    Expert AI Data Rater, English (India) — Shape Global Digital Ads QualityLocation: Remote – Based in India | Contract: Full-Time, Fixed-Term | Start Date: ASAPAt Welo Data, we're not just rating ads; we're actively defining the future of digital content and AI experiences. As a global leader in localization and data transformation, we operate one of the...

  • Expert AI Data Rater

    3 days ago

    Remote, India Welo Data Full time

    Expert AI Data Rater, English (India) — Shape Global Digital Ads Quality Location: Remote – Based in India | Contract: Full-Time, Fixed-Term | Start Date: ASAP At Welo Data, we're not just rating ads; we're actively defining the future of digital content and AI experiences. As a global leader in localization and data transformation, we operate one of the...

  • Data Engineer

    2 weeks ago

    remote, India Riskonnect Full time

    Data Engineer – Microsoft Fabric & Advanced Analytics Summary We are seeking a skilled Data Engineer with hands-on experience in Microsoft Fabric and advanced analytics platforms. The ideal candidate will design, optimize, and maintain data models and data pipelines to support reporting, analytics, and data science needs across the enterprise. You will...

  • Data Engineer

    1 week ago

    Remote, India TriDevSofts Full time ₹ 6,00,000 - ₹ 18,00,000 per year

    Data Engineer – DBT & RedshiftExp YearsLocation: RemoteJob SummaryWe are seeking a skilled Data Engineer with hands-on experience in DBT (Data Build Tool) and Amazon Redshift to design, build, and maintain scalable data pipelines and models. You'll collaborate with analytics and engineering teams to transform raw data into actionable insights using modern...