Senior Web Crawling Engineer

1 week ago


kolkata, India beBeeDataEngineer Full time

Data Engineering Manager - Web Crawling Job OverviewThe ideal candidate will lead and mentor data engineering and web crawling teams to ensure high-quality delivery and adherence to best practices.Key Responsibilities:Architect, implement, and optimize scalable data pipelines that support high-volume data ingestion, transformation, and storage.Build and maintain robust crawling systems using modern frameworks, handling IP rotation, throttling, and dynamic content extraction.Establish pipeline orchestration using Airflow, Celery, or similar distributed processing technologies.Requirements and QualificationsTechnical Skills:Strong understanding of proxy systems, anti-bot techniques, and scalable crawler architecture.Hands-on experience with cloud data platforms (AWS/GCP/Azure) and AI/LLM frameworks (Crawl4ai, LangChain, LlamaIndex, AutoGen, OpenAI).BenefitsThis role offers a unique opportunity to work on cutting-edge projects and collaborate with a talented team of professionals.



  • Kolkata, India AIMLEAP Full time

    Data Engineering Manager – Web Crawling & Pipeline ArchitectureExperience: 7 to 12 YearsLocation: Remote / BangaloreEngagement: Full-timePositions: 2Qualification: B.E / B.Tech / M.Tech / MCA / Computer Science / ITIndustry: IT / Data / AI / E-commerce / FinTech / HealthcareNotice Period: ImmediateWhat We Are Looking ForProven experience leading...


  • kolkata, India beBeeCrawling Full time

    We're seeking a Web Crawling Engineer to build and maintain web crawlers, extract valuable insights from the web, and ensure data quality.The ideal candidate has strong Python programming skills and experience with web scraping frameworks, browser automation tools, and anti-scraping mechanisms.Key Responsibilities:Maintain and enhance existing web scraping...

  • Chief Data Architect

    2 weeks ago


    kolkata, India beBeeDataEngineering Full time

    Job Overview:Data Engineering Manager - Web CrawlingThis role is a key position within our data engineering organization.Lead and mentor senior engineers to drive the design, implementation, and optimization of scalable data pipelines.Develop and maintain robust crawling systems using modern frameworks.The ideal candidate will have:Bachelor's or master's...


  • kolkata, India beBeeData Full time

    As a Senior Data Systems Designer, you will be responsible for developing and implementing large-scale data engineering systems.You will leverage your expertise in data crawling, avoidance techniques, and VPN usage to extract and process data from public sources.With experience in designing ETL processes using cloud-native technologies, you will ensure...

  • Senior SEO Specialist

    2 weeks ago


    Kolkata, India Leapstrat Interactive Full time

    OverviewWe’re looking for a highly experienced Senior SEO Specialist who is both technical and strategic — someone who understands traditional SEO, local SEO, eCommerce SEO, and the rapidly evolving world of AI-driven search.You will help shape our clients organic visibility by implementing scalable SEO frameworks designed for both classic search engines...

  • Senior SEO Specialist

    2 weeks ago


    Kolkata, India Leapstrat Interactive Full time

    OverviewWe’re looking for a highly experienced Senior SEO Specialist who is both technical and strategic — someone who understands traditional SEO, local SEO, eCommerce SEO, and the rapidly evolving world of AI-driven search.You will help shape our clients organic visibility by implementing scalable SEO frameworks designed for both classic search engines...

  • Senior SEO Specialist

    2 weeks ago


    Kolkata, India Leapstrat Interactive Full time

    Overview We’re looking for a highly experienced Senior SEO Specialist who is both technical and strategic — someone who understands traditional SEO, local SEO, eCommerce SEO, and the rapidly evolving world of AI-driven search. You will help shape our clients organic visibility by implementing scalable SEO frameworks designed for both classic search...

  • Web Content Writer

    2 days ago


    Kolkata, India Ivan web solutions Full time

    Ivan web solutions, Kolkata, IN We are hiring for Content writer who has experience writing web content writing. **Required Skills**: - Develop unique and engaging content to meet in-house and clients multiple requirements - Create social media and search engine optimized content. - Communicate effectively with clients, meet strict deadlines, and follow...

  • Senior Architect

    1 week ago


    Kolkata, West Bengal, India Web Spiders Full time ₹ 12,00,000 - ₹ 36,00,000 per year

    Web Spiders Group is seeking an accomplished Senior Architect to lead the technical architecture for a key client project involving ERP and procurement systems. The ideal candidate will bring deep expertise in Frappe Framework, coupled with a strong foundation in cloud-native architecture (AWS) and microservices-based solutions. This role requires a...


  • kolkata, India beBeeSoftware Full time

    Job Title: Senior Frontend DeveloperAbout the RoleWe are seeking a talented and product-focused Senior Software Engineer-Web to join our growing web team.Key ResponsibilitiesCreate highly responsive and interactive web applications using modern web frameworks.Collaborate closely with product managers, designers, and backend engineers to deliver seamless user...