Data Engineering Specialist

20 hours ago

Belgaum, Karnataka, India beBeeEngineer Full time ₹ 1,50,00,000 - ₹ 2,00,00,000

Job Title:

Data Engineering Specialist

Job Description:

We are seeking a highly skilled Data Engineering Specialist to join our team. This is an exciting opportunity for someone who is passionate about building scalable and efficient data pipelines.

  • Design and develop a web crawling architecture that can handle large volumes of data.
  • Implement a lightweight classifier to route HTTP vs Playwright requests.
  • Enforce per-domain throttling and backoff mechanisms to prevent overload.
  • Develop a PDF discovery and download module to extract relevant information.
  • Implement URL normalization and canonicalization to ensure data consistency.
  • Develop a resource budgeting system to optimize browser automation performance.
  • Integrate third-party APIs as first-class sources to enhance data accuracy.
  • Unify API and crawl outputs to provide a unified view of the data.
  • Create per-domain selectors with verification on hold-outs to ensure data quality.
Required Skills and Qualifications:
  • 4+ years of Python experience, including 2+ years of building production web crawlers at scale.
  • Strong expertise in Scrapy or aiohttp/asyncio and Playwright (or Puppeteer) in production.
  • Practical proxy management and polite anti-bot tactics.
  • Hands-on experience with ETag/Last-Modified, retries, backoff, and HTTP caching.
  • Confident with CSS/XPath, schema.org/JSON-LD, and HTML parsing.
  • APIs: consuming REST/GraphQL (auth, pagination, backoff) and building small internal services (FastAPI or similar).
  • Automation/Orchestration: Airflow/Temporal/Celery (or equivalent schedulers/queues) for scheduled runs and monitoring.
  • PDF handling (requests/HEAD, hashing, size limits) and file integrity checks.
  • Queues (Redis/Kafka), Docker, Linux basics; comfort with logs/metrics.
  • Clear, pragmatic communication and strong ownership.
Benefits:
  • Competitive compensation package.
  • Opportunity to work with a talented team of engineers.
  • Collaborative and dynamic work environment.
Others:
  • Ship in small, measurable increments.
  • Track coverage and freshness as north-star metrics.
  • Prefer simple designs that are easy to operate at scale.

  • Data Engineering Specialist

    4 days ago

    Belgaum, Karnataka, India beBeeDataEngineering Full time ₹ 1,50,00,000 - ₹ 2,00,00,000

    Data Engineering SpecialistJob Summary:We are seeking an experienced data engineering specialist to join our team. The ideal candidate will have a strong background in data engineering and experience with Google Cloud Platform.The successful candidate will be responsible for developing, constructing, testing and maintaining large-scale data acquisition...

  • Data Engineering Specialist

    1 week ago

    Belgaum, Karnataka, India beBeeDataEngineering Full time US$ 15,00,000 - US$ 20,00,000

    Job DescriptionWe are seeking a seasoned Data Engineering Specialist to join our team. As a key member of our data engineering group, you will play a pivotal role in designing, building, and maintaining our data pipeline architecture.Our ideal candidate is an experienced professional with a passion for data engineering, a strong track record of delivering...

  • Data Engineering Solutions Specialist

    2 days ago

    Belgaum, Karnataka, India beBeeDataEngineering Full time ₹ 18,00,000 - ₹ 25,00,000

    Data Engineering Solutions SpecialistJob Summary:We are seeking an experienced Data Engineering Solutions Specialist to design, develop and maintain large-scale data engineering solutions using Azure Databricks.Main Responsibilities:Develop and implement ETL (Extract, Transform, Load) processes to integrate data from various sources.Design and implement data...

  • Data Engineering Specialist

    5 days ago

    Belgaum, Karnataka, India beBeeDataEngineering Full time ₹ 30,00,000 - ₹ 50,00,000

    **Job Overview**We are seeking a skilled Data Engineering Specialist to join our team. In this role, you will be responsible for designing and implementing data engineering solutions that support our business needs.**Required Skills and Qualifications*** Bachelor's or Master's degree in Computer Science, Information Systems, Data Science, or related field.*...

  • Data Engineering Specialist

    3 days ago

    Belgaum, Karnataka, India beBeeDataEngineering Full time ₹ 20,00,000 - ₹ 25,00,000

    Job Title: Data Engineering Specialist">Key ResponsibilitiesDesign and implement scalable data processing solutions using Python, PySpark, and distributed systems like HDFS, Hive, and Spark architecture.Develop ETL workflows for handling large datasets, leveraging Big Data technologies to optimize data extraction, transformation, and loading...

  • Data Engineering Specialist

    1 week ago

    Belgaum, Karnataka, India beBeeDataEngineering Full time ₹ 15,00,000 - ₹ 25,00,000

    Job Title: Data Engineering SpecialistJob Description:We are seeking an experienced Data Engineering Specialist to develop, optimize and maintain automated data pipelines supporting business intelligence initiatives. This role will focus on building production-ready workflows for data ingestion, transformation, quality checks and metadata integration,...

  • Data Insights Specialist

    1 week ago

    Belgaum, Karnataka, India beBeeDataInsights Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

    Job Title: Data Insights SpecialistAt our organization, we have a unique opportunity for a skilled Data Insights Specialist to join our team of professionals. This role will play a pivotal part in helping us unlock the full potential of our data and drive business growth.About People Prime Worldwide:We are a global IT services company that specializes in...

  • Data Engineer

    1 week ago

    Belgaum, Karnataka, India beBeeData Full time ₹ 15,00,000 - ₹ 20,00,000

    Cloud Data SpecialistWe are seeking a highly skilled professional with 4+ years of experience in data engineering to lead the design, development, and maintenance of scalable ETL pipelines for batch data ingestion and egress using Azure Data Factory, Databricks, and Spark. This role requires strong programming skills in Python and Scala, as well as hands-on...

  • Data Engineering Expert

    2 days ago

    Belgaum, Karnataka, India beBeeDataEngineering Full time ₹ 1,20,00,000 - ₹ 2,00,00,000

    Job Title: Data Engineering ExpertWe are seeking an exceptional data engineering specialist to design, build, and optimize large-scale data pipelines.The ideal candidate will have hands-on experience with distributed data processing frameworks, Python programming, and emerging technologies like IoT and immersive analytics.You will be responsible for...

  • Data Engineering Specialist

    2 days ago

    Belgaum, Karnataka, India beBeeData Full time ₹ 15,00,000 - ₹ 20,00,000

    Data Engineer RoleWe're looking for an experienced Data Engineer to play a pivotal role in building and optimizing data pipelines. This challenging yet rewarding position offers the chance to work closely with talented teams, solve real-world data challenges, and make a direct impact on business decisions.