Data Engineering Manager – Web Crawling

2 weeks ago

Kolkata, India AIMLEAP Full time

Data Engineering Manager – Web Crawling & Pipeline ArchitectureExperience: 7 to 12 YearsLocation: Remote / BangaloreEngagement: Full-timePositions: 2Qualification: B.E / B.Tech / M.Tech / MCA / Computer Science / ITIndustry: IT / Data / AI / E-commerce / FinTech / HealthcareNotice Period: ImmediateWhat We Are Looking ForProven experience leading data engineering teams with strong ownership of web crawling systems and pipeline architecture.Expertise in designing, building, and optimizing scalable data pipelines , preferably using workflow orchestration tools such as Airflow or Celery .Hands-on proficiency in Python and SQL for data extraction, transformation, processing, and storage.Experience working with cloud platforms such as AWS, GCP, or Azure for data infrastructure, deployments, and pipeline operations.Deep understanding of web crawling frameworks , proxy rotation, anti-bot strategies, session handling, and compliance with global data collection standards (GDPR/CCPA-safe crawling).Strong expertise in AI-driven automation , including integrating AI agents or frameworks like Crawl4ai into scraping, validation, and pipeline workflows..ResponsibilitiesLead and mentor data engineering and web crawling teams, ensuring high-quality delivery and adherence to best practices.Architect, implement, and optimize scalable data pipelines that support high-volume data ingestion, transformation, and storage.Build and maintain robust crawling systems using modern frameworks, handling IP rotation, throttling, and dynamic content extraction.Establish pipeline orchestration using Airflow, Celery , or similar distributed processing technologies.Define and enforce data quality, validation, and security measures across all data flows and pipelines.Collaborate with product, engineering, and analytics teams to translate data requirements into scalable technical solutions.Develop monitoring, logging, and performance metrics to ensure high availability and reliability of data systems.Oversee cloud-based deployments, cost optimization, and infrastructure improvements on AWS/GCP/Azure.Integrate AI agents or LLM-based automation for tasks such as error resolution, data validation, enrichment, and adaptive crawlingQualificationsBachelor's or master's degree in engineering, Computer Science, or related field.7–12 years of relevant experience in data engineering, pipeline design, or large-scale web crawling systems .Strong expertise in Python, SQL , and modern data processing practices.Experience working with Airflow, Celery , or similar workflow automation tools.Solid understanding of proxy systems, anti-bot techniques , and scalable crawler architecture.Hands-on experience with cloud data platforms (AWS/GCP/Azure).Experience with AI/LLM frameworks (Crawl4ai, LangChain, LlamaIndex, AutoGen, OpenAI, or similar).Strong analytical, architectural, and leadership skills.

  • Senior Web Crawling Engineer

    1 week ago

    kolkata, India beBeeDataEngineer Full time

    Data Engineering Manager - Web Crawling Job OverviewThe ideal candidate will lead and mentor data engineering and web crawling teams to ensure high-quality delivery and adherence to best practices.Key Responsibilities:Architect, implement, and optimize scalable data pipelines that support high-volume data ingestion, transformation, and storage.Build and...

  • Senior Web Data Extractor

    2 weeks ago

    kolkata, India beBeeCrawling Full time

    We're seeking a Web Crawling Engineer to build and maintain web crawlers, extract valuable insights from the web, and ensure data quality.The ideal candidate has strong Python programming skills and experience with web scraping frameworks, browser automation tools, and anti-scraping mechanisms.Key Responsibilities:Maintain and enhance existing web scraping...

  • Chief Data Architect

    2 weeks ago

    kolkata, India beBeeDataEngineering Full time

    Job Overview:Data Engineering Manager - Web CrawlingThis role is a key position within our data engineering organization.Lead and mentor senior engineers to drive the design, implementation, and optimization of scalable data pipelines.Develop and maintain robust crawling systems using modern frameworks.The ideal candidate will have:Bachelor's or master's...

  • Senior Data Systems Designer

    1 week ago

    kolkata, India beBeeData Full time

    As a Senior Data Systems Designer, you will be responsible for developing and implementing large-scale data engineering systems.You will leverage your expertise in data crawling, avoidance techniques, and VPN usage to extract and process data from public sources.With experience in designing ETL processes using cloud-native technologies, you will ensure...

  • Web Content Writer

    7 hours ago

    Kolkata, India Ivan web solutions Full time

    Ivan web solutions, Kolkata, IN We are hiring for Content writer who has experience writing web content writing. **Required Skills**: - Develop unique and engaging content to meet in-house and clients multiple requirements - Create social media and search engine optimized content. - Communicate effectively with clients, meet strict deadlines, and follow...

  • Operations Manager

    7 days ago

    kolkata, India beBeeGeneral Manager Full time

    Key Responsibilities:To effectively manage a hotel with 35-50 keys.To oversee multiple facets of operations, including front office, housekeeping, human resource management, F&B, guest experience, R&M & engineering, security & safety.To supervise various functions and human resources.To manage operational quality with attention to detail.To handle and...

  • Senior SEO Specialist

    2 weeks ago

    Kolkata, India Leapstrat Interactive Full time

    OverviewWe’re looking for a highly experienced Senior SEO Specialist who is both technical and strategic — someone who understands traditional SEO, local SEO, eCommerce SEO, and the rapidly evolving world of AI-driven search.You will help shape our clients organic visibility by implementing scalable SEO frameworks designed for both classic search engines...

  • Senior SEO Specialist

    2 weeks ago

    Kolkata, India Leapstrat Interactive Full time

    OverviewWe’re looking for a highly experienced Senior SEO Specialist who is both technical and strategic — someone who understands traditional SEO, local SEO, eCommerce SEO, and the rapidly evolving world of AI-driven search.You will help shape our clients organic visibility by implementing scalable SEO frameworks designed for both classic search engines...

  • Senior SEO Specialist

    2 weeks ago

    kolkata, India Leapstrat Interactive Full time

    OverviewWe’re looking for a highly experienced Senior SEO Specialist who is both technical and strategic — someone who understands traditional SEO, local SEO, eCommerce SEO, and the rapidly evolving world of AI-driven search.You will help shape our clients organic visibility by implementing scalable SEO frameworks designed for both classic search engines...

  • Senior SEO Specialist

    2 weeks ago

    Kolkata, India Leapstrat Interactive Full time

    Overview We’re looking for a highly experienced Senior SEO Specialist who is both technical and strategic — someone who understands traditional SEO, local SEO, eCommerce SEO, and the rapidly evolving world of AI-driven search. You will help shape our clients organic visibility by implementing scalable SEO frameworks designed for both classic search...