Python Web Scraping Engineer

1 month ago

BangaloreHyderabad, Karnataka, India Whizz HR Full time

We are seeking a highly skilled and experienced Python Web Scraping Engineer with 4 to 6 years of experience to join our team.

The ideal candidate will have a strong background in web scraping, data extraction, and automation using Python.

As a Python Web Scraping Engineer, you will be responsible for developing and maintaining web scraping scripts, extracting data from various sources, and delivering high-quality datasets for analysis and :

Web Scraping Development :


- Design, develop, and maintain web scraping scripts and automation tools using Python.

- Write efficient and scalable code to extract data from websites, APIs, databases, and other online sources.

Data Extraction :


- Implement data extraction techniques to retrieve structured and unstructured data from HTML pages, JSON responses, XML documents, and other formats.

- Handle pagination, AJAX requests, and dynamic content rendering.

Data Transformation :


- Clean, preprocess, and transform raw data into usable formats for analysis and visualization.

- Perform data validation, deduplication, normalization, and enrichment as needed.

Proxy Rotation and CAPTCHA Handling :


- Implement strategies for proxy rotation and CAPTCHA handling to bypass rate limits, IP blocks, and other anti-scraping measures.

- Use proxy servers, user-agent rotation, and CAPTCHA solvers to ensure uninterrupted scraping.

Scalability and Performance Optimization :


- Optimize web scraping scripts for performance, scalability, and efficiency.

- Implement asynchronous and parallel processing techniques to improve speed and throughput.

Error Handling and Logging :


- Handle errors, exceptions, and edge cases gracefully in web scraping scripts.

- Implement logging and monitoring solutions to track scraping activities, detect anomalies, and troubleshoot issues.

Compliance and Legal Considerations :


- Ensure compliance with website terms of service, copyright laws, and data privacy regulations while scraping data.

- Respect robots txt rules and obtain necessary permissions for scraping restricted or sensitive content.

Documentation and Collaboration :


- Document technical specifications, data schemas, and scraping workflows.

- Collaborate with cross-functional teams including data scientists, analysts, and business stakeholders.

Qualifications :

- Educational Qualifications: Bachelor's/Master's

- 4 to 6 years of experience in web scraping and data extraction using Python.

- Proficiency in web scraping libraries and frameworks such as BeautifulSoup, Scrapy, Selenium, or Requests.

- Strong understanding of HTML, CSS, XPath, JSON, and regular expressions for data parsing and extraction.

- Experience with database systems such as MySQL, PostgreSQL, MongoDB, or SQLite for storing and managing scraped data.

- Familiarity with proxy servers, CAPTCHA solving services, and IP rotation techniques for bypassing anti-scraping measures.

- Excellent problem-solving, analytical, and communication skills.

(ref:hirist.tech)

  • GMWARE - Python Developer - Web Scraping & Crawling

    2 weeks ago

    Bangalore, Karnataka, India GM WARE Full time

    Role : Python Engineer - Web Scraping & CrawlingPython Web scraping Engineer :- Web Scraping will be responsible for efficient web scraping/web crawling and parsing. - The candidate should have demonstrated experience in web scraping and data extraction along with the ability to communicate effectively and adhere to set deadlines.Responsibilities :- Develop...

  • GMWARE - Python Developer - Web Scraping & Crawling

    2 weeks ago

    Bangalore, Karnataka, India GM WARE Full time

    Role : Python Engineer - Web Scraping & CrawlingPython Web scraping Engineer :- Web Scraping will be responsible for efficient web scraping/web crawling and parsing. - The candidate should have demonstrated experience in web scraping and data extraction along with the ability to communicate effectively and adhere to set deadlines.Responsibilities :- Develop...

  • GMWARE - Python Developer - Web Scraping & Crawling

    1 week ago

    Bangalore, Karnataka, India GM WARE Full time

    Job Description :Python Lead Engineer - Web Scraping will be responsible for efficient web scraping/web crawling and parsing. The candidate should have demonstrated experience in web scraping and data extraction along with the ability to communicate effectively and adhere to set deadlines. Responsibilities : - Develop and maintain service that extracts...

  • AI Palette

    2 weeks ago

    Bangalore, Karnataka, India AI Palette Full time

    We are Ai Palette : We think the world would be a nicer place to be if Food Companies could create products that the consumers really want. So we are making it happen. We want to be the most preferred Food AI company in the world. We're making it possible by building an AI-powered SaaS platform based on our founders' experience in the Food Industry &...

  • AI Palette

    2 weeks ago

    Bangalore, Karnataka, India AI Palette Full time

    We are Ai Palette : We think the world would be a nicer place to be if Food Companies could create products that the consumers really want. So we are making it happen. We want to be the most preferred Food AI company in the world. We're making it possible by building an AI-powered SaaS platform based on our founders' experience in the Food Industry &...

  • Backend Engineer

    2 weeks ago

    Bangalore, Karnataka, India Freelancer Recruiter Full time

    Skills : Hadoop, Java, Python, Scala, SqlRequirements :- 3-4+ yrs as a backend developer- Worked in a good startup or good engineering colleges- Working with an early-stage startup in a role that involves- Building scalable systems to extract information from 15Mn+ domains.- Being able to extract people information from these domains- Extracting...

  • Backend Engineer

    2 months ago

    Bangalore, Karnataka, India Freelancer Recruiter Full time

    Skills : Hadoop, Java, Python, Scala, SqlRequirements :- 3-4+ yrs as a backend developer- Worked in a good startup or good engineering colleges- Working with an early-stage startup in a role that involves- Building scalable systems to extract information from 15Mn+ domains.- Being able to extract people information from these domains- Extracting...

  • KreditBee - Senior Data Engineer - Python Programming

    2 weeks ago

    Bangalore, Karnataka, India KreditBee Full time

    Job Description :Key Responsibilities :- Develop the Policies & Next Gen. Rule Engine for Digital Lending- Develop various RestFul APIs & Integrate with various data sourcesJob Duties :- Develop the Policies & Next Gen. Rule Engine for Digital Lending- Develop various RestFul APIs & Integrate with various data sources- Develop tools & techniques related to...

  • KreditBee - Senior Data Engineer - Python Programming

    1 month ago

    Bangalore, Karnataka, India KreditBee Full time

    Job Description :Key Responsibilities :- Develop the Policies & Next Gen. Rule Engine for Digital Lending- Develop various RestFul APIs & Integrate with various data sourcesJob Duties :- Develop the Policies & Next Gen. Rule Engine for Digital Lending- Develop various RestFul APIs & Integrate with various data sources- Develop tools & techniques related to...

  • Python Developer

    1 month ago

    Bangalore/Hyderabad, Karnataka, India Innova ESI Full time

    Job Description:We are seeking a Python Developer for an on-site role in Hyderabad. The Python Developer will be responsible for handling day-to-day tasks related to back-end web development, software development, object-oriented programming (OOP), programming, and databases. Working closely with our team, the Python Developer will contribute to the design,...

  • Lead Backend Developer

    2 weeks ago

    Bangalore, Karnataka, India MountTalent Consulting Full time

    Organization Overview:At Hoomanely, we are obsessed with Pet Health and Longevity.Our Vision: To build the world's only "Pet-first" smart ecosystem, including the right conscious products engineered to peak and sustain a pet's well-being and prosperityOur Mission: To transform knowledge systems around Pet Care Management and its conscious...

  • Lead Backend Developer

    2 months ago

    Bangalore, Karnataka, India MountTalent Consulting Full time

    Organization Overview:At Hoomanely, we are obsessed with Pet Health and Longevity.Our Vision: To build the world's only "Pet-first" smart ecosystem, including the right conscious products engineered to peak and sustain a pet's well-being and prosperityOur Mission: To transform knowledge systems around Pet Care Management and its conscious...

  • KreditBee - Senior Python Backend Developer - Django/Flask

    2 weeks ago

    Bangalore, Karnataka, India KreditBee Full time

    About :A highly motivated and passionate individual who has experience in executing end to end application deliveries; bringing them to production with high quality. Passionate about building flexible and scalable solutions with an eye for details and can weigh pros and cons to find the best possible solutions.As a Python Backend Developer, you should be...

  • KreditBee - Senior Python Backend Developer - Django/Flask

    1 month ago

    Bangalore, Karnataka, India KreditBee Full time

    About :A highly motivated and passionate individual who has experience in executing end to end application deliveries; bringing them to production with high quality. Passionate about building flexible and scalable solutions with an eye for details and can weigh pros and cons to find the best possible solutions.As a Python Backend Developer, you should be...

  • Backend Engineer

    2 weeks ago

    Bangalore, Karnataka, India Mopid Full time

    Job Title : Backend Engineer (Python)Location : Bengaluru, IndiaJob Type: Full-timeExperience: 1-6 yearsJob Description :We are seeking a skilled and passionate Backend Engineer to develop and maintain robust backend services and APIs. As part of our engineering team, you will collaborate with cross-functional teams to design, build and enhance applications...

  • Backend Engineer

    1 month ago

    Bangalore, Karnataka, India Mopid Full time

    Job Title : Backend Engineer (Python)Location : Bengaluru, IndiaJob Type: Full-timeExperience: 1-6 yearsJob Description :We are seeking a skilled and passionate Backend Engineer to develop and maintain robust backend services and APIs. As part of our engineering team, you will collaborate with cross-functional teams to design, build and enhance applications...

  • Senior Data Engineer

    2 weeks ago

    Bangalore, Karnataka, India Web Idea Solution LLP Full time

    Role : Senior Data Engineer. Experience : 4-6 years. To be onboarded by : Immediate. Location : Bangalore. Type : Onsite/Full-Time. Responsibilities :- Design and build reusable components, frameworks and libraries at scale to support analytics products.- Design and implement product features in collaboration with business and Technology stakeholders.-...

  • Senior Data Engineer

    2 weeks ago

    Bangalore, Karnataka, India Web Idea Solution LLP Full time

    Role : Senior Data Engineer. Experience : 4-6 years. To be onboarded by : Immediate. Location : Bangalore. Type : Onsite/Full-Time. Responsibilities :- Design and build reusable components, frameworks and libraries at scale to support analytics products.- Design and implement product features in collaboration with business and Technology stakeholders.-...

  • Full Stack Web Developer

    7 days ago

    Bangalore, Karnataka, India Crox Consulting Inc Full time

    Role : Full Stack Web DeveloperThis is an individual contributor role with a deep passion for the field of cybersecurity. Responsibilities : The Full Stack Web Developer will be responsible for designing and developing the new system, which will be used for prioritizing and reporting on infrastructure vulnerabilities. This role will encompass a variety of...

  • Staff Engineer

    2 months ago

    Bangalore, Karnataka, India Talentxo Full time

    Role : Staff Engineer (Full stack-Python, React)Location : BengaluruMode : OnsiteCriteria :- 12+ YOE in Software development- Minimum 8+ YOE in Python and any one of Django / Flask / Pyramid.- Experience in Frontend frameworks like reactjs / Nextjs / Vuejs / Nuxtjs etc.- Must have Experience in any databases - MySQL / PostgreSQL / Postgres / Oracle / SQL...