Lead Data Crawling Engineer

Position: Lead Data Crawling EngineerLocation: Sector 23 Dwarka, DelhiThis is a Delhi-based position and work from office onlyJob Summary: We are looking for an experienced Lead Data Crawling Engineer with 4–6 years of hands-on expertise in designing, optimizing, and scaling data extraction systems. The ideal candidate should be highly skilled in Python, Selenium, Scrapy, API integrations, SQL, and handling complex, large-scale crawling architectures. You will be responsible for building advanced automated scraping solutions, leading optimization initiatives, and ensuring the highest standards of data quality, efficiency, and compliance.Key Responsibilities:Design, develop, and maintain complex, large-scale web crawling and scraping systems to extract structured and unstructured data.Build high-performance automated pipelines using Selenium, Scrapy, Beautiful Soup, Requests, and Python-based frameworks.Handle dynamic, JavaScript-heavy websites using tools like Selenium, Playwright, or headless browsers.Develop and optimize distributed crawling setups , load balancing, and scalable architectures.Implement API-based data extraction including REST, GraphQL, OAuth, and rate-limit management.Clean, preprocess, validate, and store data efficiently using Pandas, SQL databases , and cloud storage solutions.Monitor, debug, and improve pipeline performance, ensuring high uptime and consistent data quality.Collaborate with analytics, engineering, and product teams to deliver data required for reporting and machine learning needs.Ensure all scraping activities follow ethical standards, legal guidelines, and robots.txt compliance .Mentor junior developers and contribute to coding best practices and architecture decisions.Required Skills & Qualifications:Experience: 4–6 years of professional experience in data crawling, scraping automation, and large-scale data extraction.Strong proficiency in Python , with hands-on experience using:a. Seleniumb. Scrapyc. Beautiful Soupd. RequestsExcellent understanding of HTML, CSS, JavaScript, HTTP protocols , and browser behaviour.Strong experience with data manipulation using Pandas.Expertise in SQL , database design, indexing, and performance optimization.Strong understanding of API integrations , authentication methods, and data exchange formats.Experience with Git , CI/CD workflows, and collaborative development environments.Nice-to-Have Skills:Experience with cloud-based scraping tools or serverless functions (AWS Lambda, GCP Cloud Functions, Azure Functions).Familiarity with distributed crawling , proxy rotation, headless browser automation, and captcha solving techniques.Experience handling large-scale data collection , data warehousing, or pipeline orchestration tools (Airflow, Prefect, Luigi).Knowledge of containerization (Docker) and cloud deployment.Experience with log management and monitoring tools (ELK, Grafana, Prometheus).Soft Skills:Strong problem-solving abilities with a keen eye for detail.Ability to write clean, scalable, and maintainable code .Excellent debugging and performance optimization skills.Strong communication and stakeholder management abilities.Ability to lead small projects and mentor junior team members.Nice to Have (Python & Advanced Technical Skills):Experience with cloud-based scraping tools or serverless services such as AWS Lambda, Google Cloud Functions, or Azure Functions.Familiarity with distributed crawling architectures , parallel scraping, proxy management, and data pipeline orchestration (e.g., Airflow, Prefect, Luigi).Hands-on experience in large-scale data collection, processing, and storage systems (data lakes, warehousing, cloud storage, etc.).Understanding of ethical, compliant, and legally safe web scraping practices , including robots.txt, rate limits, and data protection guidelines.About Nuvoretail ( )Nuvoretail Enlytical Technologies Private Limited is an e-commerce analytics and automation company. Our proprietary digital shelf analytics and automation platform called Enlytical.ai helps e-commerce brands solve the complexities in today’s e-commerce landscape by offering a unified and all- encompassing business view on the various aspects of e-commerce business. Our platform leverages insights drawn from multiple data points that help our clients win in e-commerce by gaining a competitive edge with data-driven insights for sharper decision-making. The insights cover all aspects of e-commerce such as digital product portfolio analysis, supply chain analytics, e-commerce operations automation, pricing, and competitor benchmarking, and Amazon advertising automation using our proprietary algorithms.As a leading e-commerce service provider, we offer the most comprehensive end-to-end e-commerce solutions to brands, both in India and abroad. Right from preparing a road map to writing our client’s e- commerce success story to assisting them In increasing their online sales, we do everything via our diverse e-commerce services and bespoke strategies and technology. Our services span across the brand’s e-commerce enablement including content and digital asset creation for product listing, On Platform, and Off Platform marketing services with deep expertise in Amazon Marketing Services (AMS), Amazon SEO through keyword research, e-Commerce operations across various e-commerce platforms, website development, social media marketing, and AI-enabled e-Commerce MIS Dashboards.Awards & Recognition:Thanks to the faith reposed on us by our clients, NuvoRetail has been featured as "The Most Promising Ecommerce Technology Service Providers in India 2020” by CIOReviewIndia Magazine. Our leadership is often acknowledged by leading e-commerce services, digital marketing, consulting, and other e- commerce programs around the world. We are now one of the very few companies in India that have become an Amazon Ads Advanced partner.