Data Engineer

3 days ago

Bengaluru, Karnataka, India Talent Vision Services Full time ₹ 80,00,000 - ₹ 1,50,00,000 per year

Job description

We are seeking a highly skilled and motivated Full Stack Python Developer to join our team . The ideal candidate will be flexible, thrive in a demanding startup environment, and report directly to the Lead Engineer. This is a work from office' position at our office in Bangalore.

Key Responsibilities

  • Design and develop web scraping frameworks to extract structured and unstructured data from a variety of sources (government, regulatory, financial, and open-web).
  • Build and maintain ETL pipelines for large-scale ingestion, transformation, and loading of data into data warehouses or document stores.
  • Write efficient, scalable, and resilient Python code, including retry logic, job orchestration, and error handling.
  • Work with Celery / Airflow / Dagster (or equivalent) to automate and schedule scraping and ETL jobs.
  • Implement anti-bot bypass mechanisms including CAPTCHA solving integrations, headless browsers (e.g., Playwright), and session management.
  • Ensure data quality, deduplication, and validation at every stage of the pipeline.
  • Monitor and optimize scraping & ETL performance for speed, cost, and reliability.
  • Collaborate with DevOps and Infra teams to deploy and scale workloads on AWS (ECS/Lambda/Batch) or similar cloud environments.

Must have

  • 3+ years hands-on experience in Python with a strong focus on data engineering.
  • Proven experience with web scraping frameworks (Playwright, Requests, BeautifulSoup, Selenium, Scrapy).
  • Strong understanding of ETL design patterns and best practices.
  • Experience with task orchestration tools (Celery, Airflow, Dagster).
  • Proficiency with databases and data stores (MySQL, PostgreSQL, MongoDB, S3, or Data Lakes).
  • Working knowledge of Docker and containerized deployments.
  • Familiarity with cloud environments (AWS preferred).
  • Strong debugging, logging, and monitoring skills.
  • Uses Git for source code management.
  • Continuous Deployment - Writing new application features so they can be deployed with zero downtime.
  • Strong ability to articulate architectures and problem statements.
  • Experience with Kubernetes deployment.

Good to have

  • Experience with OCR tools (e.g., EasyOCR, Tesseract) for scraping complex sources.
  • Knowledge of distributed scraping or proxy rotation strategies.
  • Familiarity with data modeling and schema evolution.
  • Exposure to message queues (Redis, RabbitMQ, Kafka).
  • Experience with API integrations and data ingestion from third-party services.

Plus Points

  • Prior experience working in a fast-paced startup environment.
  • Domain knowledge in the financial/data space.
  • Any external, relevant certifications.
  • Hands-on knowledge of deployment and monitoring tools.

The process

  • Candidates who get shortlisted will need to attend a scheduled coding challenge in our office.
  • Those who qualify will need to go through a second round of interview
  • Suitable candidates will be given an offer as per the company policy.

Role:

Software Development - Other

Industry Type:

FinTech / Payments

Department:

Engineering - Software & QA

Employment Type:

Full Time, Permanent

Role Category:

Software Development

Education

UG:

B.Tech/B.E. in Any Specialization, B.Sc in Any Specialization, Bachelor of Artificial Intelligence in Any Specialization

PG:

MCA in Any Specialization, M.Tech in Any Specialization

Key Skills

Skills highlighted with '' are preferred keyskills

PandasDjango FrameworkNumpyPython

DockerPython DevelopmentKubernetes DeploymentWeb ScrapingSQLKubernetesFlask

  • Data Entry Operator

    3 days ago

    Bengaluru, Karnataka, India Design Engineer Full time ₹ 2,64,000 per year

    We are seeking a dedicated and detail-oriented Data Entry Operator (DEO) to support R&D projects. The role involves accurate data entry, documentation, and maintenance of research records in secure systems. The DEO will assist scientists, engineers, and administrative staff by ensuring timely and error-free handling of project information.Key...

  • Data Engineer

    1 week ago

    Bengaluru, Karnataka, India NTT DATA Full time ₹ 15,00,000 - ₹ 25,00,000 per year

    Design and implement tailored data solutions to meet customer needs and use cases, spanning from streaming to data lakes, analytics, and beyond within a dynamically evolving technical stack. Provide thought leadership by recommending the most appropriate technologies and solutions for a given use case, covering the entire spectrum from the application layer...

  • Sr. Data Engineer Data Science

    1 week ago

    Bengaluru, Karnataka, India NTT DATA Full time ₹ 12,00,000 - ₹ 24,00,000 per year

    Req ID: 342869NTT DATA strives to hire exceptional, innovative and passionate individuals who want to grow with us. If you want to be part of an inclusive, adaptable, and forward-thinking organization, apply now. We are currently seeking a Sr. Data Engineer Data Science to join our team in Bangalore, Karnātaka (IN-KA), India (IN). ...

  • Data Engineer

    2 weeks ago

    Bengaluru, Karnataka, India NTT DATA, Inc. Full time ₹ 1,04,000 - ₹ 1,30,878 per year

    Req ID:321800We are currently seeking a Data Engineer (Talend &Pyspark) to join our team in Bangalore, Karntaka (IN-KA), India (IN)."Job Duties: Key Responsibilities: Design and implement tailored data solutions to meet customer needs and use cases, spanning from streaming to data lakes, analytics, and beyond within a dynamically evolving technical...

  • Data Engineer

    1 week ago

    Bengaluru, Karnataka, India NTT DATA, Inc. Full time ₹ 15,00,000 - ₹ 25,00,000 per year

    Key Responsibilities:Design and implement tailored data solutions to meet customer needs and use cases, spanning from streaming to data lakes, analytics, and beyond within a dynamically evolving technical stack.Provide thought leadership by recommending the most appropriate technologies and solutions for a given use case, covering the entire spectrum from...

  • Technical Lead- Data Engineer

    3 days ago

    Bengaluru, Karnataka, India Astar Data Full time ₹ 12,00,000 - ₹ 36,00,000 per year

    Kindly find the Job Description Below.Job Title:Technical Lead- Data EngineerLocation: BangaloreYears of Experience: 8+ years of experienceSigmoidworks with a variety of clients from start-ups to fortune 500 companies. We are looking for a detailed oriented self-starter to assist our engineering and analytics teams in various roles as a Software...

  • Data & AI Engineer Lead

    3 days ago

    Bengaluru, Karnataka, India NTT DATA Full time ₹ 12,00,000 - ₹ 36,00,000 per year

    Framework Design & Architecture Architect a metadata-driven, Python/Spark-based framework for automated data validation across high-volume production datasets. Define DQ rule templates for completeness, integrity, conformity, accuracy, and timeliness. Establish data quality thresholds, escalation protocols, and exception workflows. Automation & Integration...

  • Data Engineer

    2 weeks ago

    Bengaluru, Karnataka, India NTT DATA Full time ₹ 4,00,000 - ₹ 8,00,000 per year

    Req ID: 344005NTT DATA strives to hire exceptional, innovative and passionate individuals who want to grow with us. If you want to be part of an inclusive, adaptable, and forward-thinking organization, apply now. We are currently seeking a Data Engineer - ETL Developer to join our team in Bangalore, Karnātaka (IN-KA), India (IN). "Job Duties: Design,...

  • Data Engineer

    2 weeks ago

    Bengaluru, Karnataka, India NTT DATA Full time ₹ 5,00,000 - ₹ 15,00,000 per year

    Req ID: 343998NTT DATA strives to hire exceptional, innovative and passionate individuals who want to grow with us. If you want to be part of an inclusive, adaptable, and forward-thinking organization, apply now. We are currently seeking a Data Engineer - Sr ETL Developer (Architect) to join our team in Bangalore, Karnātaka (IN-KA), India (IN). "Job...

  • Data Engineer

    3 days ago

    Bengaluru, Karnataka, India NTT DATA, Inc. Full time ₹ 10,00,000 - ₹ 25,00,000 per year

    Job Duties:Migrate ETL workflows from SAP BODS to AWS Glue/dbt/Talend. Develop and maintain scalable ETL pipelines in AWS. Write PySpark scripts for large-scale data processing. Optimize SQL queries and transformations for AWS PostgreSQL. Work with Cloud Engineers to ensure smooth deployment and performance tuning. Integrate data pipelines with existing Unix...