Data Pipeline Engineer- Web Services

2 weeks ago

Remote, India Forage Ai Full time

We are seeking a Data Pipeline Engineer to develop, optimize, and maintain production-grade data pipelines focused on web data extraction and ETL workflows. This is a hands-on role requiring strong experience with Python (as the primary programming language), spaCy, LLMs, webcrawling, and cloud deployment in containerized environments. You ll have opportunities to propose, experiment with, and implement GenAI-driven approaches, innovative automations, and new strategies as part of our product and pipeline evolution. Candidates should have 5-8 years of relevant experience in data engineering, software engineering, or related fields.

Key Responsibilities:

  • Design, build, and manage scalable pipelines for ingesting, processing, and storing web and API data.
  • Develop robust web crawlers and scrapers in Python (Scrapy, lxml, Playwright) for structured and unstructured data.
  • Create and monitor ETL workflows for data cleansing, transformation, and loading into PostgreSQL and MongoDB.
  • Apply spaCy for NLP tasks and integrate/fine-tune modern LLMs for analytics.
  • DriveGenAI-based innovation and automation in core data workflows.
  • Develop and deploy secure REST APIs and web services for data access and interoperability.
  • Integrate RabbitMQ,Kafka, SQS(for distributed queueing), and Redis (for caching) into data workflows; also proficient with distributed queue tools such as Celery, TaskIQ.
  • Containerize and deploy solutions using Docker on AWS(EC2, ECS, Lambda).
  • Collaborate with data teams, maintain pipeline documentation, and enforce data quality standards.
  • Maintain and enhance legacy in-house applications as required.

Technical Skills Requirements:

  • Primary programming language is Python; must have experience writing independent Python packages.
  • Experience with multithreading and asynchronous programming in Python.
  • Advanced Python skills, including web crawling (Scrapy, lxml, Playwright) and strong SQL/data handling abilities.
  • Experience with PostgreSQL (SQL) and MongoDB (NoSQL).
  • Proficient with workflow orchestration tools such as Airflow.
  • Hands-on experience with RabbitMQ, Kafka, SQS(for queueing/distributed processing), and Redis (for caching).
  • Practical experience with spaCy for NLP and integration of at least one LLM platform (OpenAI, HuggingFace, etc.).
  • Experience with GenAI/LLMs, prompt engineering, or integrating GenAI features into data products.
  • Proficiency with Docker and AWS services (EC2, ECS, Lambda).
  • Experienced in developing secure, scalable REST APIs using FastAPI and/or Flask.
  • Familiarity with third-party APIs integration, including authentication, data handling, and rate limiting.
  • Proficient in using Git for version control and collaboration.
  • Strong analytical, problem-solving, and documentation skills.
  • Bachelor s or Master s degree in Computer Science or related field.

What We Offer:

  • High ownership and autonomy in shaping technical solutions and system architecture.
  • Opportunities to learn modern technologies and propose technical initiatives including GenAI-based approaches.
  • Collaborative, supportive, and growth-oriented engineering culture.
  • Exposure to a broad set of business and technical problems.
  • Structured onboarding and domain training.
  • Work-from-Home Infrastructure.

Infrastructure Requirements:

Since this is a completely work-from-home position, you will also require the following

  • Business-grade computer (modern processor i7, i9 , 16 GB+ RAM) with no performance obstacles.
  • Reliable high-speed internet for video calls and remote work.
  • Quality headphones camera for clear audio and video. Stable power supply and backup options in case of outages.

  • AWS/Snowflake Support Engineer

    1 week ago

    Remote, India Data Engineer Academy LLP Full time

    We're Hiring: AWS/Snowflake Support Engineer (SME Role)Remote | Full Time | 7AM EST to 3PM EST| Pay: $ /MonthlyAbout the OpportunityThis role is a blend of a Subject Matter Expert (SME) and support engineer. The primary responsibility is to clear project issues for students while working on multiple projects involving AWS, Snowflake, and dbt. The engineer...

  • Python Sme

    1 week ago

    Remote, India Data Engineer Academy Full time

    **We're Hiring**: Python Expert (SME) - Content Creation on Pandas, SQL, and AWS Remote | Part Time | Flexible Timings | Pay: $500 to $600/Month **About the Opportunity**: We are looking for a highly skilled Python Expert with strong command over Pandas, SQL, and working knowledge of AWS. In this content-focused role, you'll be responsible for creating...

  • Azure Data Architect

    2 weeks ago

    Remote, India Data PlatformExperts Full time

    **Job Description: Azure Data Architect** **Key Responsibilities**: - **Design and Implementation**: - Design and implement end-to-end data solutions (data models, pipelines, data lakes, data warehouses) in Azure. - Optimize and maintain existing Azure data structures and integration processes. - Ensure architectural solutions are scalable, maintainable,...

  • Data Engineer

    13 hours ago

    Remote, India Techmango Technology Services Full time

    Dear Candidate,Greetings of the dayI am Amutha and I'm reaching out to you regarding an exciting opportunity with TechMango. You can connect with me on LinkedIn Or Email: Techmango Technology Services is a full-scale software development services company founded in 2014 with a strong focus on emerging technologies. It holds a primary objective of delivering...

  • Data Engineer

    9 hours ago

    Remote, India AXIRE HR SOLUTION Full time

    Position: Data EngineerExperience: 6 Months to 3 YearsVacancy: 02Location: RemoteJoining: Immediate Joiners PreferredAbout the Role:We are looking for a skilled Data Engineer with strong Python expertise and hands-on experience in handling large datasets, data cleaning, analysis, and visualization. The ideal candidate should be capable of building data...

  • AWS Data Engineer

    8 hours ago

    Remote, India Techmango Technology Services Full time

    Job Location: RemoteJob Experience: 8-20 YearsModel of Work: RemoteTechnologies: AWS Redshift SnowflakeFunctional Area: Software DevelopmentJob Summary:Job Title: AWS Data Engineer - Redshift & SnowflakeLocation: Madurai - RemoteExperience: 8+ YearsEmployment Type: Full-timeAbout the Role We are looking for a skilled AWS Data Engineer with hands-on...

  • Data Engineer Advisor

    4 weeks ago

    Remote, India NTT Data Full time

    Job Description NTT DATA strives to hire exceptional, innovative and passionate individuals who want to grow with us. If you want to be part of an inclusive, adaptable, and forward-thinking organization, apply now. We are currently seeking a Data Engineer Advisor to join our team in Remote, Karntaka (IN-KA), India (IN). The SAP-to-Databricks Migration...

  • Content Writer Intern

    14 hours ago

    Remote, India Engineer Philosophy Web Services Pvt Ltd Full time

    **Job Description: Content Writer** We are hiring a **Content Writer** to join our team at **Engineer Philosophy Web Services Pvt. Ltd.** in Indore. This role offers flexibility with both **Work-from-Home (WFH)** and **Office** options. **Responsibilities**: - Create engaging and original content for websites, blogs, and social media. - Write, edit, and...

  • Data QA Engineer

    6 days ago

    Remote, India Rojgar group Full time

    About the Role:We are seeking an experienced Data QA Engineer to join our team and ensure the accuracy, consistency, and reliability of our data products and pipelines. You will play a key role in validating data at scale, implementing automated quality checks, and collaborating closely with engineering and product teams. The ideal candidate has a strong...

  • Web Crawling

    2 weeks ago

    Remote, India FullStackTechies Full time

    Web Crawling & Data Extraction Engineer (WFH)Experience: 1–7 YearsLocation: Remote (Work from Home)Mode of Engagement: Full-timeNo of Positions: 3 to 8Educational Qualification: Bachelor's degree in Computer Science, IT, or related fieldIndustry: IT / Software Services / Data & AINotice Period: Immediate JoinersWhat We Are Looking ForStrong hands-on...