
Web Crawling Engineer
4 days ago
We are seeking a Web Crawling Engineer who will be responsible for building and maintaining web crawlers, extracting valuable insights from the web, and ensuring data quality. The ideal candidate will have strong Python programming skills and experience in web scraping frameworks, browser automation tools, and handling anti-scraping mechanisms.
Key Responsibilities:
- Maintain and enhance existing web scraping and data crawling projects.
- Develop and refine crawlers using Python-based tools and frameworks.
- Utilize browser automation tools (e.g., Playwright, Selenium) to handle dynamic content.
- Clean, validate, and integrate extracted data into downstream storage systems.
- Implement and manage solutions for anti-bot measures (CAPTCHAs, IP rotation, etc.).
- Optimize crawling efficiency and ensure compliance with web crawling best practices.
- Collaborate with cross-functional teams to improve data acquisition strategies.
Required Skills & Qualifications:
- Proficiency in Python and 2 years of work experience of web scraping frameworks (especially Scrapy).
- Strong knowledge of browser automation tools such as Playwright or Selenium.
- Solid understanding of HTML, CSS, and selector languages (XPath/CSS).
- Experience in handling anti-scraping challenges and ensuring robust data extraction.
- Familiarity with distributed scraping techniques and data pipelines.
- Ability to troubleshoot and optimize web crawlers for performance and reliability.
- Strong analytical and problem-solving skills with attention to detail.
- Excellent communication and inter-personal skills.
Other Infrastructure Requirements
Since this is a completely work-from-home position, you will also require the following -
● High-speed internet connectivity for video calls and efficient work.
● Capable business-grade computer (e.g., modern processor, 8 GB+ of RAM, and
no other obstacles to interrupted, efficient work).
● Headphones with clear audio quality.
● Stable power connection and backups in case of internet/power failure.
-
Web Crawling Specialist
6 hours ago
India Forage AI Full timeForage AI is seeking a skilled Web Crawling Specialist to join our remote team. As a Web Crawling Specialist, you will play a crucial role in developing and refining crawlers using Python-based tools and frameworks. This involves building and maintaining web crawlers, extracting valuable insights from the web, and ensuring data quality.To excel in this role,...
-
Senior Software Engineer
4 days ago
India Adglobal360 Full timeWe are seeking an experienced Senior Software Engineer to join our team as a Web Crawling Expert. In this role, you will be responsible for designing and developing robust web crawling solutions using Python and related libraries.">The ideal candidate will have strong expertise in web scraping and data extraction using Python, proficiency in web scraping...
-
Python Web Crawling Expert
7 days ago
India Hindustan Times Full timeWe are Hindustan Times, a pioneering media house, and we are seeking a talented Python Web Crawling Expert to join our team. Our platform provides valuable insights and analytics to empower decision-makers in the private market.About the JobIn this role, you will design and implement web crawlers in Python for extracting data from diverse online platforms....
-
Web Crawling Specialist
5 days ago
India YipitData Full timeJob DescriptionAbout YipitData:YipitData is the leading market research and analytics firm for the disruptive economy and recently raised up to $475M from The Carlyle Group at a valuation over $1B.We analyze billions of alternative data points every day to provide accurate, detailed insights on ridesharing, e-commerce marketplaces, payments, and more. Our...
-
Senior Data Crawling Specialist
7 days ago
India Petras Solutions Private Limited Full timeJob Description:Responsible for designing and developing distributed web crawlers that can independently solve various problems encountered in the actual development process.Responsible for researching and developing web page information extraction technology algorithms to improve the efficiency and quality of data capture.Responsible for analyzing and...
-
Web Scraping Engineer
5 days ago
India YipitData Full timeAbout YipitData:We are a fast-growing technology company backed by The Carlyle Group and Norwest Venture Partners. Our offices are located in various cities around the world, but this role can be fully remote based in India with standard work hours from 11am to 8pm IST.We cultivate a people-centric culture focused on mastery, ownership, and transparency. As...
-
Python Web Scraping Developer
3 hours ago
India Forage AI Full timeWe are seeking a talented Python Web Scraping Developer to join our team at Forage AI. As a Python Web Scraping Developer, you will be responsible for building and maintaining web crawlers, extracting valuable insights from the web, and ensuring data quality.You will develop and refine crawlers using Python-based tools and frameworks such as Scrapy, and...
-
Python Web Scraping Developer
7 days ago
India Petras Solutions Private Limited Full timeAbout the Job : In full growth, particularly internationally, we are looking for new collaborators to join our fabulous team A young but experienced, dynamic and complementary team: a resolutely start-up spirit Real job and career opportunities A friendly atmosphere and a climate of trust that promotes autonomy and challenge" Responsibilities : - Responsible...
-
Data Extraction Engineer
11 hours ago
India Forage AI Full timeWe are seeking a highly skilled Data Extraction Engineer to join our team at Forage AI. In this role, you will be responsible for building and maintaining web crawlers, extracting valuable insights from the web, and ensuring data quality.As a key member of our data acquisition team, you will develop and refine crawlers using Python-based tools and frameworks...
-
Software Engineering Specialist
11 hours ago
India Hindustan Times Full timeAbout Us : Hindustan Times, part of Mosaic Digital, is a pioneering company dedicated to creating innovative products for private market stakeholders. Our platform provides valuable insights, analytics, and solutions to empower decision-makers to navigate the dynamic landscape of the private market.Overview : We are seeking talented Software Developers...
-
Data Engineer
7 days ago
India HT Digital Streams Full timeJob Description : Data Engineer (Python). About Us : VCCEdge, part of Mosaic Digital is an upcoming company dedicated to building innovative products for private market stakeholders. Our platform provides valuable insights, analytics, and solutions to empower decision-makers to navigate the dynamic landscape of the private market. Overview: We are seeking an...
-
Data Engineer
4 days ago
India Forage AI Full timeAs a Data Engineer at Forage AI, you will be responsible for designing, building, and maintaining scalable data pipelines for data ingestion, processing, and storage. You will work closely with cross-functional teams to integrate crawled data from external sources and ensure seamless data flow. Your role will involve implementing robust monitoring and...
-
Software Engineer
7 days ago
India Hindustan Times Full timeWe are Hindustan Times, a renowned media house, and we are seeking a talented Software Engineer to join our team. Our platform provides valuable insights and analytics to empower decision-makers in the private market.About UsWe are an upcoming company dedicated to building innovative products for private market stakeholders. Our mission is to provide...
-
Software Developer
7 days ago
India Hindustan Times Full timeAbout Us : VCCEdge, part of Mosaic Digital is an upcoming company dedicated to building innovative products for private market stakeholders. Our platform provides valuable insights, analytics, and solutions to empower decision-makers to navigate the dynamic landscape of the private market. Overview : We are seeking talented Software Developers proficient in...
-
Software Engineer
4 weeks ago
India Forage AI Full timeRole Summary: Join our dynamic and innovative team dedicated to delivering cutting-edge solutions in web data extraction, AI integration, and software engineering. As a Software Engineer, you will collaborate with a talented group of engineers and data scientists to develop robust, scalable solutions while acting as a crucial link between multiple teams to...
-
Data Scientist
4 days ago
India Adglobal360 Full timeWe are seeking a highly skilled Data Scientist to join our team as a Web Scraping Specialist. In this role, you will be responsible for designing and developing robust web scraping solutions using Python and related libraries.">The ideal candidate will have strong expertise in web scraping and data extraction using Python, proficiency in web scraping...
-
Associate Solution Architect
7 days ago
India MAGELLANIC CLOUD LIMITED Full timeResponsibilities : - Design, deploy, and manage highly scalable and reliable search platforms using Lucidworks Fusion and Apache Solr. - Configure and maintain search clusters on cloud environments (GKE, AWS EC2). - Ensure high availability, fault tolerance, and disaster recovery for search infrastructure. - Analyze search performance metrics and identify...
-
Resilient Data Extractor
4 days ago
India YipitData Full timeYou will be responsible for refactoring and maintaining existing web scrapers, implementing advanced scraping techniques, and collaborating with cross-functional teams. You will also monitor and troubleshoot scraper performance, develop robust monitoring solutions, and propose new tooling and methodologies to enhance our scraping capabilities and...
-
Digital Search Engine Optimizer
19 hours ago
India AviinTech Business Solutions Full timeKey Responsibilities- Perform technical SEO audits and provide recommendations for site improvements- Optimize website structure and internal linking for improved crawlability- Manage XML sitemaps and robots.txt to ensure proper indexing and crawling- Implement structured data markup to enhance search results appearance- Monitor and analyze site performance...
-
Data Engineering Professional
4 days ago
India Forage AI Full timeJob OverviewWe are seeking a highly motivated and detail-oriented Data Engineer Intern to join our team at Forage AI. As a Data Engineer Intern, you will work closely with our internal teams to develop and integrate AI-based data solutions.ResponsibilitiesCollaborate with engineers to build and optimize LLM-based applications for data extraction, XPath...