Highly Experienced Data Engineer Seeking Scalable Web Crawling Solutions

2 weeks ago

meerut, India beBeeData Full time

Job Title: Web Crawling Data Engineering ManagerJob Description:This position is responsible for leading the development and implementation of scalable data pipelines using web crawling technologies. The ideal candidate will have experience in designing, building, and optimizing complex data systems, as well as a deep understanding of cloud-based infrastructure and workflow orchestration tools.The successful candidate will be able to lead a team of engineers and collaborate with cross-functional teams to translate business requirements into technical solutions. Strong expertise in AI-driven automation and integration of AI agents or frameworks like Crawl4ai is also required.Additionally, the candidate must have hands-on proficiency in Python and SQL, as well as experience working with cloud platforms such as AWS, GCP, or Azure. Deep understanding of web crawling frameworks, proxy rotation, anti-bot strategies, session handling, and compliance with global data collection standards (GDPR/CCPA-safe crawling) is necessary.Required Skills and Qualifications:Proven experience leading data engineering teams with strong ownership of web crawling systems and pipeline architecture.Expertise in designing, building, and optimizing scalable data pipelines, preferably using workflow orchestration tools such as Airflow or Celery.Hands-on proficiency in Python and SQL for data extraction, transformation, processing, and storage.Experience working with cloud platforms such as AWS, GCP, or Azure for data infrastructure, deployments, and pipeline operations.Deep understanding of web crawling frameworks, proxy rotation, anti-bot strategies, session handling, and compliance with global data collection standards (GDPR/CCPA-safe crawling).Strong expertise in AI-driven automation, including integrating AI agents or frameworks like Crawl4ai into scraping, validation, and pipeline workflows.Benefits:This role offers a unique opportunity to work on cutting-edge technology and collaborate with a talented team of engineers. The ideal candidate will be able to adapt to changing priorities and deliver high-quality results in a fast-paced environment.Others:The successful candidate will be expected to collaborate with product, engineering, and analytics teams to ensure seamless integration of data systems and meet business objectives.

  • Senior Web Data Extractor

    2 weeks ago

    meerut, India beBeeWebCrawler Full time

    We are seeking a talented Web Crawling Engineer to enhance our web scraping and data crawling capabilities.Key Responsibilities:Maintain existing web scraping projects to ensure efficiency and scalability.Develop and refine crawlers using Python-based tools and frameworks, focusing on browser automation for dynamic content.Clean, validate, and integrate...

  • Senior Data Extraction Specialist

    1 week ago

    meerut, India beBeeCrawler Full time

    Job OverviewWe are seeking a skilled Web Crawling Engineer to join our team. As a key member of our data extraction and automation company, you will be responsible for building and maintaining web crawlers to extract valuable insights from the web.Your primary focus will be on ensuring data quality and handling anti-scraping mechanisms. You will also...

  • Web Solutions Engineer

    3 days ago

    meerut, India beBeeDeveloper Full time

    We're seeking a skilled and innovative Web Solutions Engineer to join our team.As a key member of our development team, you'll play a vital role in building cutting-edge web portals using Next.js and TypeScript.Main Responsibilities:Create responsive, accessible UI components in Next.js + TypeScript.Implement Progressive Web App (PWA) features and...

  • Technical Lead

    2 weeks ago

    meerut, India beBeeDataEngineering Full time

    Job Title: Technical Architect with Hands-on CodingWe are seeking a seasoned Technical Architect to join our team. This individual will be responsible for designing and implementing data engineering solutions using cloud-native technologies.Key Responsibilities:Architect and implement data engineering solutions using cloud-native technologies.Experience in...

  • Data Extraction Specialist

    5 days ago

    meerut, India beBeewebcrawler Full time

    Web Crawler DeveloperSeeking a skilled Web Crawler Developer to craft and maintain web crawlers, extracting valuable insights from the web, and ensuring data quality. The ideal candidate will have strong programming skills in Python and experience in web scraping frameworks, browser automation tools, and handling anti-scraping mechanisms.This role involves...

  • Software Engineer

    1 week ago

    meerut, India beBeeBackend Full time

    Job DescriptionAs a key member of our engineering team, we're seeking an experienced Backend Developer to join us in building scalable and efficient software solutions.Architect and develop robust backend services using Node.js/TypeScript (Express.js/NestJS).Design and implement APIs and microservices, including caching layers, background job workers, and...

  • Technical Architect and Data Engineer

    1 week ago

    meerut, India beBeeDataEngineer Full time

    Job Title:Technical Architect and Data EngineerAbout the Role:We are seeking a skilled Technical Architect with expertise in data engineering, graph databases, cloud-native technologies, ETL processes, and search products.Key Responsibilities:Design and implement data crawling strategies using public Internet sources, ensuring effective bot detection...

  • Scalable Solutions Developer

    2 weeks ago

    meerut, India beBeeTechnical Full time

    Senior Technical Consultant OpportunityWe are seeking an experienced Senior Technical Consultant to design, develop, customize and integrate scalable solutions within the Dynamics 365 Finance & Operations ecosystem.The ideal candidate will have strong technical expertise, hands-on development experience and the ability to convert business requirements into...

  • Chief Data Engineering Expert

    2 weeks ago

    meerut, India beBeeDataEngineer Full time

    Data EngineerWe are seeking a highly experienced data engineer to lead our microservices architecture and drive innovation.Responsibilities include designing, developing, and deploying data engineering solutions within a microservices environment.You will be responsible for architecting scalable data pipelines for ingestion, processing, and storage of data...

  • Data Engineering Solutions Developer

    2 weeks ago

    meerut, India beBeeSoftware Full time

    About Our RoleWe are seeking a talented individual to join our data engineering team. Our mission is to build and maintain software development kits (SDKs) and tools that enable seamless analytics integration across web and mobile platforms. Key Responsibilities:Design, develop, and deploy scalable SDKs for various industries.Collaborate with...