Data engineer/Web scapper

6 days ago

Lucknow, Uttar Pradesh, India QuickBid Systems Pvt. Ltd. Full time ₹ 9,00,000 - ₹ 12,00,000 per year

Job Title: Web Scraper / Data Engineer

Location: Remote | Experience: 1+ years

Responsibilities:

Design, develop, and maintain robust web scrapers for structured and dynamic data sources.

Implement headless browser automation using Playwright, Puppeteer, or Selenium to extract content reliably.

Manage session handling, cookies, CAPTCHAs, and anti-bot mechanisms to ensure uninterrupted data collection.

Transform, clean, and structure extracted data into usable formats such as JSON, CSV, or directly into databases.

Integrate scraper outputs with internal APIs, analytics pipelines, or downstream applications.

Monitor scrapers for failures, performance bottlenecks, and data inconsistencies; implement automated alerting and recovery mechanisms.

Ensure ethical scraping practices, compliance with site policies, and respect for data usage rules.

Optimize scraping performance and minimize resource usage, including efficient handling of large-scale data.

Skills & Experience:

Proficient in or Python with strong programming fundamentals.

Hands-on experience with browser automation frameworks (Playwright, Puppeteer, Selenium).

Deep understanding of HTTP, REST APIs, JSON parsing, and web protocols.

Knowledge of proxy rotation, anti-detection techniques, rate limiting, and IP management.

Experience with relational (PostgreSQL) and NoSQL (MongoDB) databases.

Strong problem-solving skills and ability to debug complex automation scenarios.

Bonus: Experience with cloud platforms (Azure/AWS) and building real-time data pipelines.

Familiarity with containerization (Docker) and CI/CD pipelines is a plus.

Perks:

Fully remote and flexible work environment.

Opportunity to work on high-scale, real-time data automation and analytics projects.

Collaborate with a dynamic team solving complex challenges in data engineering.

  • Data Extraction Analyst

    2 days ago

    Lucknow, Uttar Pradesh, India Ardem Data Services Full time ₹ 2,00,000 - ₹ 4,00,000 per year

    This is a data extraction project focused on researching and collecting information related to Business Lawsuits, Law Firms, Consumer Debt, and Debt Buyers. The work involves extracting structured data from publicly available legal databases.Key Responsibilities:Use the provided website url: to perform data lookups.Extract and record accurate information...

  • Web Development Intern

    1 week ago

    Lucknow, Uttar Pradesh, India Innovative Modern Engineers Pvt. Ltd. Full time ₹ 4,00,000 - ₹ 6,00,000 per year

    Company DescriptionInnovative Modern Engineers Pvt. Ltd. specializes in developing software solutions and online services, particularly in the field taxi/cabs and food catering through online network management. By fostering research and development, the company aims to advance software and online support systems. Innovative Modern Engineers Pvt. Ltd. is...

  • Data Engineer

    1 week ago

    Lucknow, Uttar Pradesh, India CloudKeeper Full time ₹ 5,00,000 - ₹ 12,00,000 per year

    DescriptionAbout CLOUDKEEPER :CloudKeeper is a cloud cost optimization partner that combines the power of group buying & commitments management, expert cloud consulting & support, and an enhanced visibility & analytics platform to reduce cloud cost & help businesses maximize the value from AWS, Microsoft Azure, & Google Cloud. A certified AWS Premier...

  • Full Stack Web Developer Intern

    1 week ago

    Lucknow, Uttar Pradesh, India Predulive Edutech Foundation Full time ₹ 2,00,000 - ₹ 6,00,000 per year

    Location:Lucknow, Uttar Pradesh (On-site/Hybrid)Company:Predulive LabsDuration:Internship (3–6 Months)Stipend:Based on performance and experienceAbout Predulive Labs:Predulive Labs is a leading innovator inDrones and AI-powered solutions, building products that transformagriculture, infrastructure and industrial operations. Supported bySTPI under MeitY,...

  • Data Engineer

    2 weeks ago

    Lucknow, Uttar Pradesh, India Carnation Infotech Full time ₹ 5,00,000 - ₹ 8,00,000 per year

    Job SummaryAs a Data Engineer , you will be responsible for designing, building, and maintaining the infrastructure and tools that enable the organization to collect, store, and analyze large volumes of data. You will work closely with data scientists, analysts, and other engineers to ensure that our data systems are reliable, efficient, and scalable.Key...

  • Data Quality Engineer

    1 week ago

    Lucknow, Uttar Pradesh, India Ditstek Innovation Pvt. Ltd. Full time ₹ 5,00,000 - ₹ 12,00,000 per year

    Job Title : Data Quality Engineer (Data QE) Job Description :We are seeking a skilled and experienced Data Quality Engineer (Data QE) to join our team. The ideal candidate will have a strong background in Quality Engineering, ETL and database testing, and hands-on expertise with Python, PySpark, SQL, and Databricks. You will be responsible for designing...

  • React Frontend Developer

    2 days ago

    Lucknow, Uttar Pradesh, India Ardem Data Services Full time ₹ 12,00,000 - ₹ 24,00,000 per year

    React Frontend Developer Mobile Responsive UI & TestingLocation: RemoteShift: EST hoursCompany: ARDEM IncorporatedPosition Overview:ARDEM is seeking an experienced Frontend Developer to enhance our existing React-based loan amortization desktop application by extending it with full mobile responsiveness and comprehensive testing. This role focuses on...

  • Geographic Information System Developer

    6 days ago

    Lucknow, Uttar Pradesh, India NextUpgrad Web Solutions Full time ₹ 9,00,000 - ₹ 12,00,000 per year

    Job Title:Geographic Information System Developer (GIS Developer)Company:Nextupgrad Web Solutions Pvt. Ltd.Location:Gomti Nagar, LucknowExperience:4 – 5 Years (Government Project Experience Preferred)Job Type:Full-TimeWork Mode:Work from OfficeAbout the RoleWe are looking for an experiencedGIS Developerto join our team in Lucknow. The ideal candidate will...

  • Search Engine Optimization Specialist

    1 week ago

    Lucknow, Uttar Pradesh, India Vicdigit Technologies Pvt Ltd Full time ₹ 9,00,000 - ₹ 12,00,000 per year

    Company DescriptionVICDIGIT Technologies, based in Lucknow, is a leader in providing innovative IT solutions. We offer a wide range of services, including software development, app development, web development, SEO, web design, social media, PPC, email marketing, web hosting, and content writing. Our expert team ensures client satisfaction by delivering...

  • Network Engineering Specialist Advisor

    1 week ago

    Lucknow, Uttar Pradesh, India NTT DATA North America Full time ₹ 1,04,000 - ₹ 1,30,878 per year

    Req ID:331844NTT DATA strives to hire exceptional, innovative and passionate individuals who want to grow with us. If you want to be part of an inclusive, adaptable, and forward-thinking organization, apply now.We are currently seeking a Network Engineering Specialist Advisor to join our team in Greater Noida, Uttar Pradesh (IN-UP), India (IN).Role...