NewsCatcher (YC S22) | Web Crawling & Scraping Python Engineer | india
1 week ago
🌟 We seek a highly skilled and motivated Web Crawling & Scraping Engineer. We crawl over 100k news websites daily and we are looking for someone who is passioned about Web Crawling the same way we are. Curious to explore new ways of handling difficult website cases and automating our crawling techniques.
Functions:
- Crawling Platform: Design, construct, test and maintain robust, reliable, and scalable crawling pipeline infrastructure.
- Add an automatic way of fixing non-working crawlers
- Provide metrics on website coverage
- Data Pipeline:
- Design, construct, test and maintain robust, reliable, and data pipeline infrastructure.
- Automation and unit tests
- Optimization: Optimize server performance and resource utilization of crawling infrastructure.
- Regularly review and improve system performance and scalability.
- Collaboration and Documentation: Maintain accurate and up-to-date documentation of server configurations, procedures, and policies.
- Provide technical support and training to team members as needed.
Example Tasks:
- Introduce a new automatic way of crawling a website that does not work with existing techniques
- Come up with an idea on how to verify why a specific crawler stopped working and fix it automatically
- Use LLM methods to improve crawling methods
Experience:
- Proven experience as a Web Crawling & Scraping Engineer or similar role.
- Web Scraping and Web Crawling Techniques
- Streaming/batch data processing framework such as RabbitMQ.
- Solid knowledge of SQL and NoSQL databases
- Kubernetes / Docker is a must have
- Strong problem-solving skills
- Excellent communication and collaboration skills.
Nice to have:
- Experience with ElasticSearch (OpenSearch)
Your KPIs:
- Number of non-working crawlers per website (should be small)
- The time between a crawler goes down and we come up with a fix (should be small)
Compensation and Perks:
- Competitive salary and equity.
- Up to 24 days of vacation & 16 days of sick leave/holidays (all fully paid)
- Learning and development compensation
- One meeting-free day per week
- Co-working Budget
- Training Budget
- We provide all the necessary equipment to work comfortably and efficiently from home.
- Yearly company retreats (2024 — Canary Islands, 2023 — French Alpes)
Needed tools:
- Scrapy
- Crawlee
-
India NewsCatcher (YC S22) Full timeWe are seeking an experienced Crawling Expert to join our team at NewsCatcher (YC S22). This is a fantastic opportunity to work on scalable web data solutions, leveraging your skills in web crawling and scraping engineering.Job OverviewThe successful candidate will be responsible for designing, constructing, testing, and maintaining robust, reliable, and...
-
Web Crawling
1 week ago
India NewsCatcher (YC S22) Full time🌟 We seek a highly skilled and motivated Web Crawling & Scraping Engineer. We crawl over 100k news websites daily and we are looking for someone who is passioned about Web Crawling the same way we are. Curious to explore new ways of handling difficult website cases and automating our crawling techniques.Functions:Crawling Platform: Design, construct,...
-
Web Crawling
1 week ago
India NewsCatcher (YC S22) Full timeWe seek a highly skilled and motivated Web Crawling & Scraping Engineer. We crawl over 100k news websites daily and we are looking for someone who is passioned about Web Crawling the same way we are. Curious to explore new ways of handling difficult website cases and automating our crawling techniques. Functions: Crawling Platform: Design, construct,...
-
Web Crawling
1 week ago
India NewsCatcher (YC S22) Full time We seek a highly skilled and motivated Web Crawling & Scraping Engineer. We crawl over 100k news websites daily and we are looking for someone who is passioned about Web Crawling the same way we are. Curious to explore new ways of handling difficult website cases and automating our crawling techniques. Functions: Crawling Platform: Design, construct,...
-
Web crawling
4 days ago
India NewsCatcher Full timeWe seek a highly skilled and motivated Web Crawling & Scraping Engineer. We crawl over 100k news websites daily and we are looking for someone who is passioned about Web Crawling the same way we are. Curious to explore new ways of handling difficult website cases and automating our crawling techniques. Functions: Crawling Platform: Design, construct,...
-
Python + Web Scraping
4 weeks ago
India Gravity Infosolutions Full timeLocation: RemoteEmployment Type: ContractExperience: 3-4 yearsJob Description:We are seeking an experienced Python Developer specializing in Web Scraping for a contract role. The ideal candidate will have strong skills in Python, database management, and experience with web scraping tools and techniques.Key Responsibilities:Develop, test, and deploy...
-
Python + Web Scraping
4 weeks ago
india Gravity Infosolutions Full timeLocation: RemoteEmployment Type: ContractExperience: 3-4 yearsJob Description:We are seeking an experienced Python Developer specializing in Web Scraping for a contract role. The ideal candidate will have strong skills in Python, database management, and experience with web scraping tools and techniques.Key Responsibilities:Develop, test, and deploy...
-
CompUp (YC S22) | Rewards Specialist | india
7 days ago
india CompUp (YC S22) Full timeAbout the job- CompUp is a compensation management startup located in Bengaluru. Our platform helps total rewards teams eliminate pay disparities and promote fair pay in organizations. With our software, you can build and share compensation bands, run budget simulations, seek manager recommendations for increments, and generate reward letters. We also...
-
CompUp (YC S22) | Rewards Specialist | india
1 week ago
india CompUp (YC S22) Full timeAbout the job-CompUp is a compensation management startup located in Bengaluru. Our platform helps total rewards teams eliminate pay disparities and promote fair pay in organizations. With our software, you can build and share compensation bands, run budget simulations, seek manager recommendations for increments, and generate reward letters. We also provide...
-
Software engineer
4 weeks ago
India ELife Full timeOur Company: A fast-growing start-up headquartered in San Francisco, CA, USA in the heart of Silicon Valley. We recruit worldwide as our customer base is global. Reliable ground transportation provider, any type of vehicle globally. Vision: Reliable ground transportation services globally with all types of vehicles.​ Mission: Empower high...
-
Software Engineer
4 weeks ago
india ELife Full timeOur Company: A fast-growing start-up headquartered in San Francisco, CA, USA in the heart of Silicon Valley. We recruit worldwide as our customer base is global. Reliable ground transportation provider, any type of vehicle globally. Vision: Reliable ground transportation services globally with all types of vehicles.​ Mission: Empower high quality local...
-
Software Engineer
1 month ago
India ELife Full timeOur Company: A fast-growing start-up headquartered in San Francisco, CA, USA in the heart of Silicon Valley. We recruit worldwide as our customer base is global. Reliable ground transportation provider, any type of vehicle globally. Vision: Reliable ground transportation services globally with all types of vehicles.​ Mission: Empower high...
-
Software Engineer
1 month ago
India ELife Full timeOur Company: A fast-growing start-up headquartered in San Francisco, CA, USA in the heart of Silicon Valley. We recruit worldwide as our customer base is global. Reliable ground transportation provider, any type of vehicle globally. Vision: Reliable ground transportation services globally with all types of vehicles.​ Mission: Empower high...
-
Software Engineer
1 month ago
india ELife Full timeOur Company: A fast-growing start-up headquartered in San Francisco, CA, USA in the heart of Silicon Valley. We recruit worldwide as our customer base is global. Reliable ground transportation provider, any type of vehicle globally. Vision: Reliable ground transportation services globally with all types of vehicles.​ Mission: Empower high quality local...
-
Software Engineer
1 month ago
India ELife Full timeOur Company: A fast-growing start-up headquartered in San Francisco, CA, USA in the heart of Silicon Valley. We recruit worldwide as our customer base is global. Reliable ground transportation provider, any type of vehicle globally. Vision: Reliable ground transportation services globally with all types of vehicles.​ Mission: Empower high quality local...
-
Software Engineer
1 month ago
India ELife Full timeOur Company:A fast-growing start-up headquartered in San Francisco, CA, USA in the heart of Silicon Valley. We recruit worldwide as our customer base is global. Reliable ground transportation provider, any type of vehicle globally.Vision: Reliable ground transportation services globally with all types of vehicles.​Mission: Empower high quality local...
-
Software Engineer
1 month ago
india ELife Full timeOur Company:A fast-growing start-up headquartered in San Francisco, CA, USA in the heart of Silicon Valley. We recruit worldwide as our customer base is global. Reliable ground transportation provider, any type of vehicle globally.Vision: Reliable ground transportation services globally with all types of vehicles.​Mission: Empower high quality local...
-
Python + web scrapping
3 weeks ago
India Gravity Infosolutions Full timeLocation : Remote Employment Type : Contract Experience : 3-4 years Job Description : We are seeking an experienced Python Developer specializing in Web Scraping for a contract role. The ideal candidate will have strong skills in Python, database management, and experience with web scraping tools and techniques. Key Responsibilities ...
-
Forage AI | Software Engineer | india
2 weeks ago
india Forage AI Full timeSoftware EngineerRole Summary - In this role, you’ll be working with an amazingly passionate and talented team of engineers and data scientists who are working at the bleeding edge of data science and data Automation.Requirements:A bachelor’s degree in Computer Science/Information Technology engineering is preferred. 2-3 years of experience in web...
-
Forage AI | Software Engineer | india
1 week ago
india Forage AI Full timeSoftware EngineerRole Summary - In this role, you’ll be working with an amazingly passionate and talented team of engineers and data scientists who are working at the bleeding edge of data science and dataAutomation.Requirements:A bachelor’s degree in Computer Science/Information Technology engineering is preferred.0-2 years of experience in web crawling...