Crawling Expert Wanted for Scalable Web Data Solutions

1 week ago


India NewsCatcher (YC S22) Full time

We are seeking an experienced Crawling Expert to join our team at NewsCatcher (YC S22). This is a fantastic opportunity to work on scalable web data solutions, leveraging your skills in web crawling and scraping engineering.

Job Overview

The successful candidate will be responsible for designing, constructing, testing, and maintaining robust, reliable, and scalable crawling pipeline infrastructure. You will work closely with our team to identify areas for improvement and implement optimizations to enhance server performance and resource utilization.

Key Responsibilities

  • Design and implement a robust crawling platform using Scrapy and Crawlee.
  • Add automated mechanisms for fixing non-working crawlers.
  • Develop metrics to track website coverage.
  • Maintain accurate and up-to-date documentation of server configurations, procedures, and policies.
  • Provide technical support and training to team members as needed.

Requirements

  • Proven experience as a Web Crawling & Scraping Engineer or similar role.
  • Strong knowledge of web scraping and web crawling techniques.
  • Familiarity with streaming/batch data processing frameworks such as RabbitMQ.
  • Solid understanding of SQL and NoSQL databases.
  • Kubernetes/Docker expertise is essential.
  • Excellent problem-solving skills and strong communication abilities.

Preferred Qualifications

  • Experience with ElasticSearch/OpenSearch.

Compensation Package

  • A competitive salary range: $120,000 - $180,000 per year.
  • Up to 24 days of paid vacation and 16 days of sick leave/holidays.
  • Learning and development compensation.

  • Web crawling

    4 days ago


    India NewsCatcher Full time

    We seek a highly skilled and motivated Web Crawling & Scraping Engineer. We crawl over 100k news websites daily and we are looking for someone who is passioned about Web Crawling the same way we are. Curious to explore new ways of handling difficult website cases and automating our crawling techniques. Functions: Crawling Platform: Design, construct,...

  • Web Crawling

    1 week ago


    India NewsCatcher (YC S22) Full time

    🌟 We seek a highly skilled and motivated Web Crawling & Scraping Engineer. We crawl over 100k news websites daily and we are looking for someone who is passioned about Web Crawling the same way we are. Curious to explore new ways of handling difficult website cases and automating our crawling techniques.Functions:Crawling Platform: Design, construct,...

  • Web Crawling

    1 week ago


    India NewsCatcher (YC S22) Full time

    We seek a highly skilled and motivated Web Crawling & Scraping Engineer. We crawl over 100k news websites daily and we are looking for someone who is passioned about Web Crawling the same way we are. Curious to explore new ways of handling difficult website cases and automating our crawling techniques. Functions: Crawling Platform: Design, construct,...

  • Web Crawling

    1 week ago


    India NewsCatcher (YC S22) Full time

     We seek a highly skilled and motivated Web Crawling & Scraping Engineer. We crawl over 100k news websites daily and we are looking for someone who is passioned about Web Crawling the same way we are. Curious to explore new ways of handling difficult website cases and automating our crawling techniques. Functions: Crawling Platform: Design, construct,...


  • india NewsCatcher (YC S22) Full time

    🌟 We seek a highly skilled and motivated Web Crawling & Scraping Engineer. We crawl over 100k news websites daily and we are looking for someone who is passioned about Web Crawling the same way we are. Curious to explore new ways of handling difficult website cases and automating our crawling techniques.Functions:Crawling Platform: Design, construct,...


  • India Black Panda Enterprises Full time

    Are you a passionate and skilled professional looking for a challenging role at Black Panda Enterprises?About the JobWe are seeking a talented Web Crawler Developer to join our team. This is a work-from-home position that requires high-speed internet connectivity, a capable business-grade computer, and a stable power connection.Job SummaryThis role involves...

  • Web Developer

    3 weeks ago


    India American HealthCare Academy Full time

    Job Title: Web Developer - Healthcare Solutions ExpertLocation: RemoteCompany: American Health Care AcademyOverview:American Health Care Academy is seeking an experienced Senior Web Developer with expertise in LAMP stack, Laravel, and AWS to design and maintain web applications supporting healthcare training and certification.Responsibilities:We are looking...


  • india Suncoast Web Solutions Full time

    About the Role: Suncoast Web Solutions is looking for a talented and experienced Graphic Designer specializing in website design. If you have a passion for creating visually stunning, user-friendly websites for businesses and possess at least 5 years of relevant experience, we’d love to hear from you! As part of our team, you'll work remotely, designing...


  • india Suncoast Web Solutions Full time

    About the Role:Suncoast Web Solutions is looking for a talented and experienced Graphic Designer specializing in website design. If you have a passion for creating visually stunning, user-friendly websites for businesses and possess at least 5 years of relevant experience, we’d love to hear from you! As part of our team, you'll work remotely, designing and...


  • india Wavicle Data Solutions Full time

    Job Description: We are seeking a highly experienced Lead Data Engineer with over 8 years of expertise in data engineering. As a Lead Data Engineer, you will play a pivotal role in architecting and implementing data solutions. Your proficiency in Python, PySpark, AWS, Databricks, SQL, and leadership skills will be crucial for success. Key Responsibilities:...


  • india Wavicle Data Solutions Full time

    Job Description:We are seeking a highly experienced Lead Data Engineer with over 8 years of expertise in data engineering.As a Lead Data Engineer, you will play a pivotal role in architecting and implementing data solutions.Your proficiency in Python, PySpark, AWS, Databricks, SQL, and leadership skills will be crucial for success.Key Responsibilities:Lead...


  • india Wavicle Data Solutions Full time

    Job Description: We are seeking a highly experienced Lead Data Engineer with over 8 years of expertise in data engineering. As a Lead Data Engineer, you will play a pivotal role in architecting and implementing data solutions. Your proficiency in Python, PySpark, AWS, Databricks, SQL, and leadership skills will be crucial for success. Key Responsibilities:...


  • Anywhere in India/Multiple Locations Serenetic Software Pvt Ltd Full time

    We are seeking a highly skilled Software Architect to design, develop and maintain our web applications using ASP.NET Core and C#. As a key member of our team at Serenetic Software Pvt Ltd, you will play a vital role in shaping the future of our technology stack.Key Responsibilities:Design and implement scalable web applications using ASP.NET Core and C#Work...


  • India Zeproc Full time

    Job Title: Senior Technical SEO ExpertLocation: RemoteJob Type: Full-time contractAbout Zeproc:Zeproc is a fast-growing e-commerce marketplace connecting industrial suppliers with businesses across Norway and Europe. Our mission is to make industrial supplies accessible, user-friendly, and seamlessly integrated into modern procurement workflows. We are...


  • India Zeproc Full time

    Job Title: Senior Technical SEO Expert Location: Remote Job Type: Full-time contract About Zeproc: Zeproc is a fast-growing e-commerce marketplace connecting industrial suppliers with businesses across Norway and Europe. Our mission is to make industrial supplies accessible, user-friendly, and seamlessly integrated into modern procurement workflows.  We...

  • Software Engineer

    2 weeks ago


    India Forage AI Full time

    Software EngineerRole Summary - In this role, you’ll be working with an amazingly passionate and talented team of engineers and data scientists who are working at the bleeding edge of data science and data Automation.Requirements:A bachelor’s degree in Computer Science/Information Technology engineering is preferred. 2-3 years of experience in web...


  • india Forage AI Full time

    Software EngineerRole Summary - In this role, you’ll be working with an amazingly passionate and talented team of engineers and data scientists who are working at the bleeding edge of data science and data Automation.Requirements:A bachelor’s degree in Computer Science/Information Technology engineering is preferred. 2-3 years of experience in web...


  • india Forage AI Full time

    Software EngineerRole Summary - In this role, you’ll be working with an amazingly passionate and talented team of engineers and data scientists who are working at the bleeding edge of data science and dataAutomation.Requirements:A bachelor’s degree in Computer Science/Information Technology engineering is preferred.0-2 years of experience in web crawling...

  • Software Engineer

    1 week ago


    India Forage AI Full time

    Software EngineerRole Summary - In this role, you’ll be working with an amazingly passionate and talented team of engineers and data scientists who are working at the bleeding edge of data science and dataAutomation.Requirements:A bachelor’s degree in Computer Science/Information Technology engineering is preferred.0-2 years of experience in web crawling...

  • Software Engineer

    2 weeks ago


    India Forage AI Full time

    Software Engineer Role Summary - In this role, you’ll be working with an amazingly passionate and talented team of engineers and data scientists who are working at the bleeding edge of data science and data Automation. Requirements: A bachelor’s degree in Computer Science/Information Technology engineering is preferred. 2-3 years of...