NewsCatcher (YC S22) | Web Crawling & Scraping Python Engineer | delhi
2 weeks ago
Functions:
Crawling Platform: Design, construct, test and maintain robust, reliable, and scalable crawling pipeline infrastructure.
Add an automatic way of fixing non-working crawlers
Provide metrics on website coverage
Data Pipeline :
Design, construct, test and maintain robust, reliable, and data pipeline infrastructure.
Automation and unit tests
Optimization: Optimize server performance and resource utilization of crawling infrastructure.
Regularly review and improve system performance and scalability.
Collaboration and Documentation: Maintain accurate and up-to-date documentation of server configurations, procedures, and policies.
Provide technical support and training to team members as needed.
Example Tasks:
Introduce a new automatic way of crawling a website that does not work with existing techniques
Come up with an idea on how to verify why a specific crawler stopped working and fix it automatically
Use LLM methods to improve crawling methods
Experience:
Proven experience as a Web Crawling & Scraping Engineer or similar role.
Web Scraping and Web Crawling Techniques
Streaming/batch data processing framework such as RabbitMQ.
Solid knowledge of SQL and NoSQL databases
Kubernetes / Docker is a must have
Strong problem-solving skills
Excellent communication and collaboration skills.
Nice to have:
Experience with ElasticSearch (OpenSearch)
Your KPIs:
Number of non-working crawlers per website (should be small)
The time between a crawler goes down and we come up with a fix (should be small)
Compensation and Perks :
Competitive salary and equity.
Up to 24 days of vacation & 16 days of sick leave/holidays (all fully paid)
Learning and development compensation
One meeting-free day per week
Co-working Budget
Training Budget
We provide all the necessary equipment to work comfortably and efficiently from home.
Yearly company retreats (2024 — Canary Islands, 2023 — French Alpes)
Needed tools:
Scrapy
Crawlee
-
Web Crawling
2 weeks ago
Delhi, India NewsCatcher (YC S22) Full timeWe seek a highly skilled and motivated Web Crawling & Scraping Engineer. We crawl over 100k news websites daily and we are looking for someone who is passioned about Web Crawling the same way we are. Curious to explore new ways of handling difficult website cases and automating our crawling techniques.Functions:Crawling Platform:Design, construct, test and...
-
Web Crawling
2 weeks ago
Delhi, India NewsCatcher (YC S22) Full timeWe seek a highly skilled and motivated Web Crawling & Scraping Engineer. We crawl over 100k news websites daily and we are looking for someone who is passioned about Web Crawling the same way we are. Curious to explore new ways of handling difficult website cases and automating our crawling techniques.Functions:Crawling Platform: Design, construct, test and...
-
Delhi, India NewsCatcher (YC S22) Full timeWe seek a highly skilled and motivated Web Crawling & Scraping Engineer. We crawl over 100k news websites daily and we are looking for someone who is passioned about Web Crawling the same way we are. Curious to explore new ways of handling difficult website cases and automating our crawling techniques.Functions:- Crawling Platform: Design, construct, test...
-
Web crawling
2 weeks ago
Delhi, India NewsCatcher Full timeWe seek a highly skilled and motivated Web Crawling & Scraping Engineer. We crawl over 100k news websites daily and we are looking for someone who is passioned about Web Crawling the same way we are. Curious to explore new ways of handling difficult website cases and automating our crawling techniques.Functions:Crawling Platform: Design, construct, test and...
-
Python + Web Scraping
1 month ago
delhi, India Gravity Infosolutions Full timeLocation : Remote Employment Type : Contract Experience : 3-4 years Job Description : We are seeking an experienced Python Developer specializing in Web Scraping for a contract role. The ideal candidate will have strong skills in Python, database management, and experience with web scraping tools and techniques. Key Responsibilities : Develop, test, and...
-
Python + Web Scraping
4 weeks ago
Delhi, India Gravity Infosolutions Full timeLocation : RemoteEmployment Type : ContractExperience : 3-4 yearsJob Description :We are seeking an experienced Python Developer specializing in Web Scraping for a contract role. The ideal candidate will have strong skills in Python, database management, and experience with web scraping tools and techniques.Key Responsibilities :Develop, test, and deploy...
-
CompUp (YC S22) | Rewards Specialist | delhi
2 weeks ago
delhi, India CompUp (YC S22) Full timeAbout the job- CompUp is a compensation management startup located in Bengaluru. Our platform helps total rewards teams eliminate pay disparities and promote fair pay in organizations. With our software, you can build and share compensation bands, run budget simulations, seek manager recommendations for increments, and generate reward letters. We also...
-
CompUp (YC S22) | Rewards Specialist | delhi
2 weeks ago
delhi, India CompUp (YC S22) Full timeAbout the job-CompUp is a compensation management startup located in Bengaluru. Our platform helps total rewards teams eliminate pay disparities and promote fair pay in organizations. With our software, you can build and share compensation bands, run budget simulations, seek manager recommendations for increments, and generate reward letters. We also provide...
-
Software Engineer
1 month ago
delhi, India ELife Full timeOur Company: A fast-growing start-up headquartered in San Francisco, CA, USA in the heart of Silicon Valley. We recruit worldwide as our customer base is global. Reliable ground transportation provider, any type of vehicle globally. Vision: Reliable ground transportation services globally with all types of vehicles. Mission: Empower high quality local...
-
Web Scraper Python
6 months ago
Delhi, India Raptor Supplies Private Limited Full time**Web Scraper (Python) Job Description: - ** About Raptor Supplies Raptor Supplies is a UK registered ecommerce company operating in the Maintenance, Repair, and Operations (MRO) industry. With a thriving India office in South Delhi, we provide top-quality products globally. Our strong team of over 200 employees serves a diverse customer base all over the...
-
Software Engineer
1 month ago
Delhi, India ELife Full timeOur Company:A fast-growing start-up headquartered in San Francisco, CA, USA in the heart of Silicon Valley. We recruit worldwide as our customer base is global. Reliable ground transportation provider, any type of vehicle globally.Vision:Reliable ground transportation services globally with all types of vehicles.Mission:Empower high quality local...
-
Software Engineer
1 month ago
Delhi, India ELife Full timeOur Company:A fast-growing start-up headquartered in San Francisco, CA, USA in the heart of Silicon Valley. We recruit worldwide as our customer base is global. Reliable ground transportation provider, any type of vehicle globally.Vision:Reliable ground transportation services globally with all types of vehicles.Mission:Empower high quality local...
-
Web Scraping
3 days ago
Delhi, India WIZ Full timeJob role - Web Scraping & Data Extraction SpecialistExperience - 2+yearsLocation - ChennaiRoles & Responsibilities- Develop, test, and maintain custom web scraping scripts and solutions using Python or other programming languages.- Scrape structured and unstructured data from static and dynamic websites.- Handle challenges related to dynamic content,...
-
Software Engineer
1 month ago
delhi, India ELife Full timeOur Company:A fast-growing start-up headquartered in San Francisco, CA, USA in the heart of Silicon Valley. We recruit worldwide as our customer base is global. Reliable ground transportation provider, any type of vehicle globally.Vision: Reliable ground transportation services globally with all types of vehicles.Mission: Empower high quality local...
-
Software Engineer
1 month ago
Delhi, India ELife Full timeOur Company:A fast-growing start-up headquartered in San Francisco, CA, USA in the heart of Silicon Valley. We recruit worldwide as our customer base is global. Reliable ground transportation provider, any type of vehicle globally.Vision: Reliable ground transportation services globally with all types of vehicles.Mission: Empower high quality local...
-
Software Engineer
1 month ago
delhi, India ELife Full timeOur Company:A fast-growing start-up headquartered in San Francisco, CA, USA in the heart of Silicon Valley. We recruit worldwide as our customer base is global. Reliable ground transportation provider, any type of vehicle globally.Vision: Reliable ground transportation services globally with all types of vehicles.Mission: Empower high quality local...
-
Forage AI | Software Engineer | delhi
3 weeks ago
delhi, India Forage AI Full timeSoftware Engineer Role Summary - In this role, you’ll be working with an amazingly passionate and talented team of engineers and data scientists who are working at the bleeding edge of data science and data Automation. Requirements: A bachelor’s degree in Computer Science/Information Technology engineering is preferred. 2-3 years of experience in web...
-
Forage AI | Software Engineer | delhi
2 weeks ago
delhi, India Forage AI Full timeSoftware EngineerRole Summary - In this role, you’ll be working with an amazingly passionate and talented team of engineers and data scientists who are working at the bleeding edge of data science and dataAutomation.Requirements:A bachelor’s degree in Computer Science/Information Technology engineering is preferred.0-2 years of experience in web crawling...
-
delhi, India Black Panda Enterprises Full timeRole SummaryIn this role, you’ll be working with an amazingly passionate and talented team of engineers and data scientists who are working at the bleeding edge of data science and data automation.RequirementsA bachelor’s degree in Computer Science/Information Technology engineering is preferred.0-2 years of experience in web crawling using Python.Must...
-
Python + Web Scrapping
1 month ago
delhi, India Gravity Infosolutions Full timeLocation: Remote Employment Type: Contract Experience: 3-4 years Job Description: We are seeking an experienced Python Developer specializing in Web Scraping for a contract role. The ideal candidate will have strong skills in Python, database management, and experience with web scraping tools and techniques. Key Responsibilities: - Develop, test, and...