Data Engineering Manager – Web Crawling

1 day ago

Remote, India FullStackTechies Full time ₹ 10,000 - ₹ 20,00,000 per year

Data Engineering Manager – Web Crawling & Pipeline Architecture

Experience: 7 to 12 Years

Location: Remote / Bangalore

Engagement: Full-time

Positions: 2

Qualification: B.E / B.Tech / M.Tech / MCA / Computer Science / IT

Industry: IT / Data / AI / E-commerce / FinTech / Healthcare

Notice Period: Immediate

What We Are Looking For

  • Proven experience leading data engineering teams with strong ownership of web crawling systems and pipeline architecture.
  • Expertise in designing, building, and optimizing scalable data pipelines, preferably using workflow orchestration tools such as Airflow or Celery.
  • Hands-on proficiency in Python and SQL for data extraction, transformation, processing, and storage.
  • Experience working with cloud platforms such as AWS, GCP, or Azure for data infrastructure, deployments, and pipeline operations.
  • Deep understanding of web crawling frameworks, proxy rotation, anti-bot strategies, session handling, and compliance with global data collection standards (GDPR/CCPA-safe crawling).
  • Strong expertise in AI-driven automation, including integrating AI agents or frameworks like Crawl4ai into scraping, validation, and pipeline workflows..

Responsibilities

  • Lead and mentor data engineering and web crawling teams, ensuring high-quality delivery and adherence to best practices.
  • Architect, implement, and optimize scalable data pipelines that support high-volume data ingestion, transformation, and storage.
  • Build and maintain robust crawling systems using modern frameworks, handling IP rotation, throttling, and dynamic content extraction.
  • Establish pipeline orchestration using Airflow, Celery, or similar distributed processing technologies.
  • Define and enforce data quality, validation, and security measures across all data flows and pipelines.
  • Collaborate with product, engineering, and analytics teams to translate data requirements into scalable technical solutions.
  • Develop monitoring, logging, and performance metrics to ensure high availability and reliability of data systems.
  • Oversee cloud-based deployments, cost optimization, and infrastructure improvements on AWS/GCP/Azure.
  • Integrate AI agents or LLM-based automation for tasks such as error resolution, data validation, enrichment, and adaptive crawling

Qualifications

  • Bachelor's or master's degree in engineering, Computer Science, or related field.
  • 7–12 years of relevant experience in data engineering, pipeline design, or large-scale web crawling systems.
  • Strong expertise in Python, SQL, and modern data processing practices.
  • Experience working with Airflow, Celery, or similar workflow automation tools.
  • Solid understanding of proxy systems, anti-bot techniques, and scalable crawler architecture.
  • Hands-on experience with cloud data platforms (AWS/GCP/Azure).
  • Experience with AI/LLM frameworks (Crawl4ai, LangChain, LlamaIndex, AutoGen, OpenAI, or similar).
  • Strong analytical, architectural, and leadership skills.

Job Type: Full-time

Pay: ₹1,000, ₹2,000,000.04 per year

Benefits:

  • Health insurance
  • Provident Fund
  • Work from home

Application Question(s):

  • Years of hands-on experience do you have in data engineering
  • Years of hands-on experience do you have building and delivering data engineering projects
  • Years of hands-on experience do you have in web crawling or scraping systems?
  • years have you worked hands-on with Python and SQL in production environments?
  • years of hands-on experience do you have with Airflow, Celery, or any pipeline orchestration tools?
  • Years of hands-on experience do you have with Crawl4AI for automated scraping?
  • Notice period (days)
  • Current CTC (monthly)

Work Location: Remote

  • Data Engineering Manager – Web Crawling

    1 day ago

    Remote, India Outsource Bigdata Full time ₹ 8,00,000 - ₹ 11,00,000 per year

    Data Engineering Manager – Web Crawling & Pipeline ArchitectureAimleapIndia (Remote)Posted on November 29, 2025AIMLEAP is Hiring:Data Engineering Manager – Web Crawling & Pipeline ArchitectureExperience: 7 to 12 YearsLocation: Remote / BangaloreEngagement: Full-timePositions: 2Qualification: B.E / B.Tech / M.Tech / MCA / Computer Science / ITIndustry: IT...

  • AEO and SEO Engineer

    1 week ago

    Remote, India Colours Of Fashion Full time ₹ 4,80,000 per year

    We are looking for experiencedAEO and SEO Engineer Specialist analyzes, tests, and modifies websites to optimize them for search engines, leading to higher rankings in major search results like Google and Bing)A Search Engine Optimization (SEO) Specialist analyzes, tests, and modifies websites to optimize them for search engines, leading to higher rankings...

  • Web Engineer

    5 days ago

    remote, India Oliv AI Full time

    About UsOliv.AI is a SalesTech global startup headquartered in San Francisco, debuting the world's first team of AI Agents for sales. With our recent $5.2M Seed funding, we solve one of the biggest problems for revenue teams: unreliable deal data. Oliv captures Deal Intelligence from every meeting, call, and email—without any rep involvement. The result...

  • Sr. PHP Web Developer

    24 hours ago

    Remote, India J H Web Solutions Full time ₹ 30,00,000 - ₹ 40,00,000 per year

    We are seeking a skilled PHP Developer with experience in MySQL, hybrid applications, and ERP systems to join our team. The ideal candidate will be responsible for developing and maintaining the university's ERP system, ensuring seamless integration and functionality.Key Responsibilities:Develop and maintain PHP-based web applications using MySQL.Work on...

  • Web Designer

    6 days ago

    Remote, India Blive web solutions pvt ltd Full time

    **Job Overview** We are looking for an ambitious and energetic Web Designer who has a deep understanding of various designing aspects. **Responsibilities and Duties**: - Conceptualising project’s creative ideas. - Preparing project’s design plans as per client’s requirements. - Efficiently incorporating functionalities and features into projects. -...

  • GIS Data Intern

    1 week ago

    Remote, India Engineer Philosophy Web Services Pvt Ltd Full time ₹ 88,800 - ₹ 14,40,000 per year

    Job Title: GIS Data Intern (Freshers Welcome)Company: Engineer Philosophy Web Services Pvt. Ltd.Location: Indore, Madhya Pradesh (Remote)Job Type: Internship / Full-timeJob Description:Engineer Philosophy Web Services Pvt. Ltd. is seeking a motivated and detail-oriented GIS Data Intern to join our Indore-based team. This internship provides hands-on...

  • Web Engineer

    4 days ago

    Remote - India Oliv AI Full time ₹ 12,00,000 - ₹ 36,00,000 per year

    About UsOliv.AI is a SalesTech global startup headquartered in San Francisco, debuting the world's first team of AI Agents for sales. With our recent $5.2M Seed funding, we solve one of the biggest problems for revenue teams: unreliable deal data. Oliv captures Deal Intelligence from every meeting, call, and email—without any rep involvement. The result...

  • Web Scraper

    1 day ago

    Remote, India STerry Full time ₹ 8,00,000 - ₹ 12,00,000 per year

    Web Scraper / Data EngineerLocation: RemoteJob Type: Full-time About STERRYAt STERRY, we're not your average Growth Marketing Agency—we're the rocket fuel behind crowdfunding and e-commerce success. Since day one, we've helped clients pull in over $100 million in trackable online revenue. We build strategies that go beyond brand and marketing—we deliver...

  • Data Engineer

    3 days ago

    Remote, India AXIRE HR SOLUTION Full time ₹ 3,00,000 - ₹ 8,00,000 per year

    Position: Data EngineerExperience: 6 Months to 3 YearsVacancy: 02Location: RemoteJoining: Immediate Joiners PreferredAbout the Role:We are looking for a skilled Data Engineer with strong Python expertise and hands-on experience in handling large datasets, data cleaning, analysis, and visualization. The ideal candidate should be capable of building data...

  • Senior Data Engineer

    3 days ago

    Remote, India AABM Cloud Data Solutions Full time ₹ 20,00,000 per year

    Senior Data Engineer (Remote, India)10+ Years Data Engineer Experience RequiredSQL, Python, Snowflake - Mandatory hands-on experienceExperience with any ETL Tools (SSIS Preferred), Cloud Environments, and DB2/DBT is a plus.Job Type: Full-timePay: From ₹2,000,000.00 per yearApplication Question(s):How many years of Python Experience do you have?How many...