Chief Web Crawling Architect

5 days ago

Kolhapur, India beBeeDataEngineering Full time

Data Engineering Manager - Web Crawling & Pipeline Architecture LeadThe ideal candidate will have a proven track record of leading data engineering teams, with strong ownership of web crawling systems and pipeline architecture.Expertise in designing, building, and optimizing scalable data pipelines using workflow orchestration tools like Airflow or Celery.Hands-on proficiency in Python and SQL for data extraction, transformation, processing, and storage.Experience working with cloud platforms such as AWS, GCP, or Azure for data infrastructure, deployments, and pipeline operations.Deep understanding of web crawling frameworks, proxy rotation, anti-bot strategies, session handling, and compliance with global data collection standards (GDPR/CCPA-safe crawling).Strong expertise in AI-driven automation, including integrating AI agents or frameworks like Crawl4ai into scraping, validation, and pipeline workflows.About the Role:Lead and mentor high-performing data engineering and web crawling teams, ensuring timely delivery and adherence to best practices.Design, implement, and optimize scalable data pipelines that support high-volume data ingestion, transformation, and storage.Build and maintain robust crawling systems using modern frameworks, handling IP rotation, throttling, and dynamic content extraction.Establish pipeline orchestration using Airflow, Celery, or similar distributed processing technologies.Define and enforce data quality, validation, and security measures across all data flows and pipelines.Collaborate with product, engineering, and analytics teams to translate data requirements into scalable technical solutions.Develop monitoring, logging, and performance metrics to ensure high availability and reliability of data systems.Oversee cloud-based deployments, cost optimization, and infrastructure improvements on AWS/GCP/Azure.Integrate AI agents or LLM-based automation for tasks such as error resolution, data validation, enrichment, and adaptive crawling.We Offer:A competitive salary package, opportunities for career growth and professional development, and a collaborative work environment.

  • Master Data Architect

    5 days ago

    Kolhapur, India beBeeArchitect Full time

    Job Title: Master Data Architect    We are looking for a skilled Data Architect to lead our team in designing and implementing a robust data management system. RequirementsWe seek a highly motivated individual with experience in data crawling, bot detection avoidance techniques, and use of VPN. You should have expertise in data engineering, including...

  • Senior Code Architect

    2 days ago

    Kolhapur, India beBeeExpert Full time

    Opportunity Overview:We are seeking a skilled Technical Architect with hands-on coding expertise to contribute to our team's success. The ideal candidate will possess experience in data/content crawling, bot detection avoidance techniques, and VPN usage.Key Skills and Expertise:Modelling and managing complex multi-dimensional entities and graphsKnowledge of...

  • Senior Data-Driven Technical Architect

    15 hours ago

    Kolhapur, India beBeeDataDriven Full time

    Job OpportunityWe are seeking a highly skilled Technical Architect with hands-on coding experience to fill this role.About the Role:Data and Content Extraction: Utilize data crawling techniques from public sources, implement bot detection avoidance methods, and leverage VPN protocols for secure access.Data Engineering: Design and manage complex data models,...

  • Senior Data Extraction Specialist

    5 days ago

    kolhapur, India beBeeCrawler Full time

    Web Data ExtractorWe're seeking a highly skilled Web Data Extractor to design and maintain web crawlers, extract valuable insights, and ensure data quality.Maintain and enhance existing web scraping projectsDevelop crawlers using Python-based tools and frameworksUtilize browser automation tools to handle dynamic contentClean, validate, and integrate...

  • Chief Software Architect

    5 days ago

    Kolhapur, India beBeeBackend Full time

    Job Title: Senior Backend DeveloperWe are seeking a highly skilled Senior Backend Developer to join our team. The ideal candidate will have expertise in designing, developing, and maintaining backend applications for customers worldwide.Key Responsibilities:Design, develop, and maintain scalable backend applications using Python/MySQL hosted on Amazon Web...

  • Chief Software Architect

    2 days ago

    Kolhapur, India beBeeCloudEngineer Full time

    Cloud EngineerJob Summary:A Cloud Engineer will be part of our EDP Data Platform team for a major Insurance client.The person will work with different stakeholders to architect and build the EDP application platform to support clients' internal data teams to onboard, provision data in Data Cloud.The application will be architected using Micro Services...

  • Chief Data Architect

    2 days ago

    Kolhapur, India beBeeDataEngineer Full time

    Job OpportunityWe are seeking a skilled data engineer to collaborate on client projects with an asset management firm.Collaborate with analysts to understand and anticipate project requirements.Design, implement, and maintain web scrapers for diverse alternative datasets.Perform data cleaning, exploration, and transformation of scraped data.Work closely with...

  • Chief Monitoring Architect

    14 hours ago

    Kolhapur, India beBeeExpert Full time

    Job DescriptionWe seek a highly experienced architect to lead the design, assessment, migration, and optimization of enterprise-grade monitoring, observability, and automation platforms.This critical role will provide expert guidance, define the future-state tooling architecture, and ensure operational readiness during the separation process.Key...

  • Enterprise Web Applications Developer

    2 days ago

    Kolhapur, India beBeeFullStack Full time

    Job DescriptionWe are seeking an experienced Full Stack Developer to lead the development of our enterprise web applications. As a key member of our team, you will play a vital role in designing, developing, and delivering cutting-edge software solutions.You will be responsible for leading the frontend and backend development teams, guiding them on complex...

  • Senior web scraping engineer

    3 weeks ago

    Kolhapur, India Zomunk Full time

    About us We're building a product that relies heavily on collecting structured data from a number of known websites. We need someone experienced who can own this part of the system end-to-end; from writing scrapers to making sure they scale and stay reliable. This is a senior role. We're looking for someone who has already dealt with the real-world problems...