Data Engineer

22 hours ago

Hyderabad, Telangana, India OSI Digital Full time ₹ 9,00,000 - ₹ 12,00,000 per year

We are looking for a skilled
Web Crawling Engineer
with strong expertise in Python and hands-on experience building scalable crawlers and scrapers. The ideal candidate should have deep knowledge of crawling frameworks, anti-blocking techniques, and proxy management, along with experience solving challenges like captchas and rate-limiting.

Key Responsibilities

  • Design, develop, and maintain scalable and efficient web crawlers for data extraction.
  • Work with Scrapy, Requests, and other Python libraries to implement crawling solutions.
  • Handle anti-scraping measures, including IP blocks, rate limits, and captchas.
  • Implement and manage proxy rotation and session management strategies.
  • Ensure high-quality and structured data extraction, cleaning, and storage.
  • Monitor crawler performance, troubleshoot issues, and optimize for reliability and efficiency.
  • Collaborate with the data engineering team to integrate crawled data into pipelines.
  • Stay updated with the latest tools, techniques, and best practices in web crawling.

Required Skills & Qualifications

  • Strong programming skills in Python.
  • Proven experience with Scrapy, Requests, BeautifulSoup, Selenium (if required).
  • Hands-on experience solving blocking issues, handling
  • Good understanding of proxy usage, rotation, and fingerprinting avoidance techniques.
  • Knowledge of HTTP, cookies, headers, sessions, and request/response cycles.
  • Experience with databases (SQL/NoSQL) for storing crawled data.
  • Strong debugging, problem-solving, and analytical skills.

Good to Have

  • Experience with cloud environments (AWS, GCP, Azure).
  • Knowledge of data pipelines and ETL processes.

Education & Experience

  • Bachelor's degree in Computer Science, Information Technology, or equivalent practical experience.
  • 3+ years of experience in web crawling or related fields

  • Senior Data Engineer

    2 weeks ago

    Hyderabad, Telangana, India Enable Data Full time ₹ 15,00,000 - ₹ 20,00,000 per year

    Experience Required: 8+YearsMode of work: RemoteSkills Required: Azure DataBricks, Eventhub, Kafka, Architecture,Azure Data Factory, Pyspark, Python, SQL, SparkNotice Period : Immediate Joiners/ Permanent/Contract role (Can join within September 15th 2025)Design, develop, and maintain scalable and robust data solutions in the cloud using Apache Spark and...

  • Azure Data Engineer ll

    2 weeks ago

    Hyderabad, Telangana, India Enable Data Full time US$ 90,000 - US$ 1,20,000 per year

    Enable Data Incorporated is currently seeking a skilled and experienced Azure Data Engineer to join our dynamic team. As a leading provider of advanced application, data, and cloud engineering services, Enable Data has developed deep expertise across various industries. We work closely with our customers to leverage modern solutions and technologies to drive...

  • Data Engineer

    2 weeks ago

    Hyderabad, Telangana, India ICE Data Services Full time US$ 1,25,000 - US$ 1,75,000 per year

    Job PurposeAt Intercontinental Exchange (ICE), we engineer technology, exchanges and clearing houses that connect companies around the world to global capital and derivative markets. With a leading-edge approach to developing technology platforms, we have built market infrastructure in all major trading centers, offering customers the ability to manage risk...

  • Senior Data Engineer

    2 weeks ago

    Hyderabad, Telangana, India Enable Data Incorporated Full time ₹ 15,00,000 - ₹ 20,00,000 per year

    Design, develop, and maintain scalable and robust data solutions in the cloud using Apache Spark and Databricks. Gather and analyse data requirements from business stakeholders and identify opportunities for data-driven insights. Build and optimize data pipelines for data ingestion, processing, and integration using Spark and Databricks. Ensure data...

  • Senior Data Engineer

    2 weeks ago

    Hyderabad, Telangana, India Enable Data Incorporated Full time ₹ 15,00,000 - ₹ 20,00,000 per year

    Experience Required: 8+YearsMode of work: RemoteSkills Required: Azure DataBricks, Eventhub, Kafka, Architecture,Azure Data Factory, Pyspark, Python, SQL, SparkNotice Period : Immediate Joiners/ Permanent/Contract role (Can join within September 15th 2025)Design, develop, and maintain scalable and robust data solutions in the cloud using Apache Spark and...

  • Property Data Engineer

    1 week ago

    Hyderabad, Telangana, India ICE Data Services Full time ₹ 15,00,000 - ₹ 28,00,000 per year

    Job PurposeThe Property Data Engineeris responsible for developing and maintaining data conversion programs that transform raw property assessment data into standardized formats based on specifications by Property Data Analyst and Senior Analysts. This role requires not only advanced programming and ETL skills but also a deep understanding of the structure,...

  • Lead/Senior Data Engineer

    2 weeks ago

    Hyderabad, Telangana, India Data Economy Full time US$ 1,50,000 - US$ 2,00,000 per year

    We are seeking a Lead/Senior Data Engineer with 7-12 years of experience to architect, develop, and optimize data solutions in a cloud-native environment. The role requires strong expertise in AWS Glue, PySpark, and Python with a proven ability to design scalable data pipelines and frameworks for large-scale enterprise systems. Prior exposure to financial...

  • Senior Data Engineer

    1 week ago

    Hyderabad, Telangana, India Enable Data Incorporated Full time

    Experience Required: 8+YearsMode of work: RemoteSkills Required: Azure DataBricks, Eventhub, Kafka, Architecture, Azure Data Factory, Pyspark, Python, SQL, SparkNotice Period : Immediate Joiners/ Permanent/Contract role (Can join within September 29th 2025)- Translate business rules into technical specifications and implement scalable data solutions.- Manage...

  • Aws Data Engineer

    2 weeks ago

    Hyderabad, Telangana, India Data Economy Full time US$ 1,50,000 - US$ 2,00,000 per year

    We are seeking a highly skilled and experienced Senior Data Engineer to lead the end-to-end development of complex models for compliance and supervision. The ideal candidate will have deep expertise in cloud-based infrastructure, ETL pipeline development, and financial domains, with a strong focus on creating robust, scalable, and efficient solutions.Key...

  • Senior Data Engineer

    1 week ago

    Hyderabad, Telangana, India Enable Data Incorporated Full time ₹ 15,00,000 - ₹ 28,00,000 per year

    Experience Required: 8+YearsMode of work: RemoteSkills Required: Azure DataBricks, Eventhub, Kafka, Architecture, Azure Data Factory, Pyspark, Python, SQL, SparkNotice Period : Immediate Joiners/ Permanent/Contract role (Can join within September 29th 2025)Translate business rules into technical specifications and implement scalable data solutions. Manage a...