Data Pipeline Specialist

3 days ago


Bengaluru, Karnataka, India beBeeBackend Full time ₹ 1,50,00,000 - ₹ 2,00,00,000
Backend Developer Job Description

We are seeking a skilled Backend Developer to join our team. The ideal candidate will have a strong background in developing scalable data collection pipelines and implementing automated web data extraction solutions.

The successful candidate will be responsible for designing, developing, and maintaining large-scale data pipeline systems that process financial documents from diverse web sources. They will also work closely with cross-functional teams to ensure technical decisions align with business goals.

  • Main Responsibilities:
  • Design and develop scalable data collection pipelines
  • Implement automated web data extraction solutions
  • Collaborate with cross-functional teams
  • Monitor and troubleshoot data pipelines

Required Skills:

  • Technical Skills:
  • Python programming language
  • Experience with web scraping technologies (Selenium, BeautifulSoup, Scrapy)
  • AWS services (Lambda, EC2, ECS, S3)
  • ETL processes, data pipeline management, workflow tools (Apache Airflow)
  • Docker for backend services and web data extraction scripts
  • Proxy services

Benefits:

  • Competitive salary package
  • Top-notch equipment of your choice
  • Wellness allowance
  • Flexible work setup: work remotely or from our offices

Key Performance Indicators (KPIs):

  • Data quality and integrity
  • Pipeline efficiency and scalability
  • Timely data delivery and availability


  • Bengaluru, Karnataka, India beBeeDataPipeline Full time ₹ 12,00,000 - ₹ 25,00,000

    Job Opportunity: Data Pipeline SpecialistWe are seeking a skilled data pipeline expert to validate and optimize our data transformations.ETL testing methodologies and best practicesDebugging data issues and resolving discrepanciesWorking with large datasets in cloud environmentsThe ideal candidate should have hands-on experience in Databricks, Python, SQL,...


  • Bengaluru, Karnataka, India beBeeDataPipeline Full time

    Job Title: Data Pipeline SpecialistA leading IT consulting and software solutions provider is seeking a highly skilled professional to design, build, and maintain scalable and reliable data pipelines.This role involves developing ETL processes to ensure data quality, integrity, and consistency. The ideal candidate will collaborate with analysts, scientists,...


  • Bengaluru, Karnataka, India beBeeDataPipeline Full time US$ 1,00,000 - US$ 1,50,000

    Job Overview- We are seeking a highly skilled Data Pipeline Specialist to join our team. The ideal candidate will have experience building and productionizing data pipelines using industry-leading technologies.- A strong background in working with complex SQL datasets is required, as well as expertise in building efficient SQL scripts using BigQuery Syntax-...


  • Bengaluru, Karnataka, India beBeeData Full time ₹ 15,00,000 - ₹ 25,00,000

    We are seeking a highly skilled Senior Data Pipeline Specialist to join our team.As a Senior Data Pipeline Specialist, you will be responsible for designing and developing real-time and batch data pipelines using tools like Apache NiFi, Apache Kafka, and Apache Airflow.Design and develop scalable data pipelines that meet the needs of various business...


  • Bengaluru, Karnataka, India beBeeAzure Full time ₹ 15,00,000 - ₹ 25,00,000

    Freelance Azure Data Factory SpecialistWe are seeking an experienced Freelance Azure Data Factory (ADF) Specialist to support our data integration and ETL needs.The ideal consultant will have strong hands-on expertise in Azure ADF pipelines, data orchestration, and transformations to ensure smooth, scalable, and secure data flows across multiple sources and...


  • Bengaluru, Karnataka, India beBeeData Full time ₹ 1,00,00,000 - ₹ 2,00,00,000

    The Role of Senior Data Integration SpecialistJob Overview:We are seeking an experienced Senior Data Integration Specialist to join our team. This is a key position that will play a vital role in the smooth operation of our enterprise data integration platform.As a Senior Data Integration Specialist, you will be responsible for designing, implementing, and...


  • Bengaluru, Karnataka, India beBeeDataEngineer Full time

    Job Title: Azure Databricks EngineerAs a skilled Data Engineer, you will play a pivotal role in our dynamic team.High Impact Required: Immediate joiners only. Please note that candidates with less than 12 months of experience will not be considered.Key Responsibilities:Our ideal candidate will have strong expertise in building and optimizing data pipelines...


  • Bengaluru, Karnataka, India beBeeDataEngineering Full time ₹ 15,00,000 - ₹ 25,00,000

    About Our Data Engineering PositionAt our company, we are seeking an experienced data engineer to join our team. The ideal candidate will have a strong background in building scalable data pipelines and working on modern cloud & big data platforms.Key Responsibilities:Design, develop, and maintain large-scale data systems using Python, PySpark, and SQL.Work...


  • Bengaluru, Karnataka, India beBeedata Full time ₹ 10,00,000 - ₹ 15,00,000

    Job Opportunity    Description:A Data Engineer Associate/Jr Data Engineer position is available in our organization. The jobholder will be responsible for designing, building and managing scalable data pipelines to support business needs. They will also perform complex data transformations using SQL and Spark, conduct performance tuning to improve the...


  • Bengaluru, Karnataka, India beBeeDataEngineer Full time US$ 1,60,000 - US$ 2,00,000

    Job Title:Big Data EngineerWe are seeking a skilled and experienced Spark Scala Developer to fill this key role.Key Responsibilities:Design and develop scalable data pipelines using Apache Spark and ScalaOptimize and troubleshoot Spark jobs for performance (e.g. memory management, shuffles, skew)Work with massive datasets in on-prem Hadoop clusters or cloud...