Web Scrapper

**Job description**
- Proven experience as Web Scrapper/Crawler or similar role 2-4 years’ experience with a Bachelor's Degree in Computer Science, Engineering, Technology or related field required
- Have strong understanding and working knowledge of web crawlers, web scrapers and other automation tools, to help browse the web content
- Knowledge of web scraping and tools
- Strong knowledge of any of multiple open-source and proprietary scraping frameworks available
- Hands-on-experience with SQL/NO-SQL (MySQL/ Postgres/Cassandra /MongoDB)
- Good knowledge and coding experience in one or more programming languages such as Python, Java, JavaScript
- Experience of creating scrapy spiders for websites with Captcha, IP ban, geolocation ban, Cloudflare / Distil / Imperva firewalls, sites required login to access data, Dynamic websites loading through JS / REST API / Graphql etc.
- Knowledge of Object-oriented programming
- Experience with AWS cloud services (EC2)
- Python Tech stack (Python libraries - scrapy, requests, Urllib, Beautiful soup, splash, Selenium, pandas)

**Responsibilities**
- Design, build web crawlers to scrape data and URLs by using Python modules [scrapy, selenium, requests, Beautiful Soup, splash, etc.]
- Create crawlers for all types of websites irrespective of the technical roadblocks.
- Manage the crawlers to overcome technical challenges like IP ban, geolocation ban, captcha and bot blocking services
- Design scrapy pipelines to connect the crawler output to MySQL database
- Integrate the data crawled and scraped into our databases
- Build and maintain high quality reusable code
- Automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability

**Salary**: Up to ₹1,200,000.00 per year

Schedule:

- Night shift