Data Extractor

As a Data Engineer, you will be responsible for extracting and processing large amounts of data from various sources. This involves designing and implementing scalable web scraping systems to handle high-volume data collection without interruptions.

The ideal candidate will have experience in web scraping, crawling, or data collection, with strong proficiency in Python and familiarity with NoSQL databases and data serialization formats.

Key responsibilities include building and maintaining automated scrapers, developing multi-threaded crawlers, normalizing scraped data, and ensuring consistency before passing it to data pipelines.

You will also implement anti-bot and evasion tactics, such as proxy rotation and request throttling, to handle scraping restrictions.

Integration with pipelines is another key aspect of the role, where you will deliver clean, structured datasets into NoSQL stores and ETL pipelines for further enrichment and graph-based storage.

Additionally, you will ensure data accuracy, deduplicate records, and maintain a trust scoring system for data confidence.

• Technical Skills:
  • Experience in web scraping, crawling, or data collection.
  • Strong proficiency in Python (libraries like BeautifulSoup, Scrapy, Selenium, Playwright, Requests).
  • Familiarity with NoSQL databases (MongoDB, DynamoDB) and data serialization formats (JSON, CSV, Parquet).
  • Experience in handling large-scale scraping with proxy management and rate-limiting.
  • Basic knowledge of ETL processes and integration with data pipelines.
  • Exposure to graph databases (Neo4j) is a plus.
  • Soft Skills:
    • Detail-oriented, ensuring accuracy and reliability of collected data.
    • Strong problem-solving skills, particularly in adapting scrapers to evolving web structures.
    • Curious mindset with a drive to discover new data sources.
    • Comfortable working in a fast-paced environment.

This is an exciting opportunity to leverage technology in the fight against fraud and build something impactful from day one.