Data Engineering Manager – Web Crawling

1 week ago

New Delhi, India AIMLEAP Full time

Data Engineering Manager – Web Crawling & Pipeline ArchitectureExperience:7 to 12 Years Location:Remote / Bangalore Engagement:Full-time Positions:2 Qualification:B.E / B.Tech / M.Tech / MCA / Computer Science / IT Industry:IT / Data / AI / E-commerce / FinTech / Healthcare Notice Period:ImmediateWhat We Are Looking ForProven experienceleading data engineering teamswith strong ownership of web crawling systems and pipeline architecture. Expertise indesigning, building, and optimizing scalable data pipelines , preferably using workflow orchestration tools such asAirfloworCelery . Hands-on proficiency inPythonandSQLfor data extraction, transformation, processing, and storage. Experience working with cloud platforms such asAWS, GCP, or Azurefor data infrastructure, deployments, and pipeline operations. Deep understanding ofweb crawling frameworks , proxy rotation, anti-bot strategies, session handling, and compliance with global data collection standards (GDPR/CCPA-safe crawling). Strong expertise inAI-driven automation , including integrating AI agents or frameworks likeCrawl4aiinto scraping, validation, and pipeline workflows..ResponsibilitiesLead and mentor data engineering and web crawling teams, ensuring high-quality delivery and adherence to best practices. Architect, implement, and optimizescalable data pipelinesthat support high-volume data ingestion, transformation, and storage. Build and maintainrobust crawling systemsusing modern frameworks, handling IP rotation, throttling, and dynamic content extraction. Establish pipeline orchestration usingAirflow, Celery , or similar distributed processing technologies. Define and enforcedata quality, validation, and security measuresacross all data flows and pipelines. Collaborate with product, engineering, and analytics teams to translate data requirements into scalable technical solutions. Develop monitoring, logging, and performance metrics to ensure high availability and reliability of data systems. Oversee cloud-based deployments, cost optimization, and infrastructure improvements on AWS/GCP/Azure. IntegrateAI agents or LLM-based automationfor tasks such as error resolution, data validation, enrichment, and adaptive crawlingQualifications Bachelor's or master's degree in engineering, Computer Science, or related field. 7–12 years of relevant experience indata engineering, pipeline design, or large-scale web crawling systems . Strong expertise inPython, SQL , and modern data processing practices. Experience working withAirflow, Celery , or similar workflow automation tools. Solid understanding ofproxy systems, anti-bot techniques , and scalable crawler architecture. Hands-on experience withcloud data platforms(AWS/GCP/Azure). Experience withAI/LLM frameworks(Crawl4ai, LangChain, LlamaIndex, AutoGen, OpenAI, or similar). Strong analytical, architectural, and leadership skills.

  • Web Crawling Engineer

    3 weeks ago

    New Delhi, India Forage AI Full time

    We are seeking a Web Crawling Engineer who will be responsible for building and maintaining web crawlers, extracting valuable insights from the web, and ensuring data quality. The ideal candidate will have strong Python programming skills and experience in web scraping frameworks, browser automation tools, and handling anti-scraping mechanisms.Salary budget...

  • Web Crawling Engineer

    4 weeks ago

    New Delhi, India Forage AI Full time

    We are seeking aWeb Crawling Engineerwho will be responsible for building and maintaining web crawlers, extracting valuable insights from the web, and ensuring data quality. The ideal candidate will have strong Python programming skills and experience in web scraping frameworks, browser automation tools, and handling anti-scraping mechanisms. Salary budget...

  • Web Crawling Engineer

    22 hours ago

    New Delhi, India Forage AI Full time

    We are seeking aWeb Crawling Engineerwho will be responsible for building and maintaining web crawlers, extracting valuable insights from the web, and ensuring data quality. The ideal candidate will have strong Python programming skills and experience in web scraping frameworks, browser automation tools, and handling anti-scraping mechanisms. Salary budget...

  • Web Crawling Engineer

    4 weeks ago

    Delhi, India Forage AI Full time

    We are seeking a Web Crawling Engineer who will be responsible for building and maintaining web crawlers, extracting valuable insights from the web, and ensuring data quality. The ideal candidate will have strong Python programming skills and experience in web scraping frameworks, browser automation tools, and handling anti-scraping mechanisms.Salary budget...

  • Web Crawling Engineer

    3 weeks ago

    Delhi, India Forage AI Full time

    We are seeking a Web Crawling Engineer who will be responsible for building and maintaining web crawlers, extracting valuable insights from the web, and ensuring data quality. The ideal candidate will have strong Python programming skills and experience in web scraping frameworks, browser automation tools, and handling anti-scraping mechanisms. Salary budget...

  • Data Engineering Manager – Web Crawling

    3 days ago

    New Delhi, India AIMLEAP Full time

    Data Engineering Manager – Web Crawling & Pipeline ArchitectureExperience: 7 to 12 YearsLocation: Remote / BangaloreEngagement: Full-timePositions: 2Qualification: B.E / B.Tech / M.Tech / MCA / Computer Science / ITIndustry: IT / Data / AI / E-commerce / FinTech / HealthcareNotice Period: ImmediateWhat We Are Looking For- Proven experience leading data...

  • Web Crawling Engineer

    3 weeks ago

    Delhi, India Forage AI Full time

    We are seeking aWeb Crawling Engineerwho will be responsible for building and maintaining web crawlers, extracting valuable insights from the web, and ensuring data quality. The ideal candidate will have strong Python programming skills and experience in web scraping frameworks, browser automation tools, and handling anti-scraping mechanisms.Salary budget...

  • Data Engineering Manager – Web Crawling

    3 days ago

    New Delhi, India AIMLEAP Full time

    Data Engineering Manager – Web Crawling & Pipeline ArchitectureExperience:7 to 12 Years Location:Remote / Bangalore Engagement:Full-time Positions:2 Qualification:B.E / B.Tech / M.Tech / MCA / Computer Science / IT Industry:IT / Data / AI / E-commerce / FinTech / Healthcare Notice Period:ImmediateWhat We Are Looking ForProven experienceleading data...

  • Data Engineering Manager – Web Crawling

    1 week ago

    Delhi, India AIMLEAP Full time

    Data Engineering Manager – Web Crawling & Pipeline Architecture Experience: 7 to 12 Years Location: Remote / Bangalore Engagement: Full-time Positions: 2 Qualification: B.E / B.Tech / M.Tech / MCA / Computer Science / IT Industry: IT / Data / AI / E-commerce / FinTech / Healthcare Notice Period: Immediate What We Are Looking For Proven experience leading...

  • Data engineering manager – web crawling

    1 week ago

    Delhi, India AIMLEAP Full time

    Data Engineering Manager – Web Crawling & Pipeline ArchitectureExperience:7 to 12 YearsLocation:Remote / BangaloreEngagement:Full-timePositions:2Qualification:B. E / B. Tech / M. Tech / MCA / Computer Science / ITIndustry:IT / Data / AI / E-commerce / Fin Tech / HealthcareNotice Period:ImmediateWhat We Are Looking ForProven experienceleading data...