Senior Pipeline Developer

4 days ago

Tiruppur, Tamil Nadu, India beBeeDataEngineer Full time ₹ 1,20,00,000 - ₹ 2,00,00,000

As a Data Engineer, you will play a pivotal role in building scalable pipelines and working with streaming data to deliver high-quality solutions.

About the Role
  • This position involves designing, building, and optimizing data pipelines and ETL workflows.
  • You will manage and process large datasets using Spark, PySpark, and SQL.
  • Experience with real-time streaming applications and Spark Streaming/Kafka is highly desirable.
  • Collaboration with data scientists and product teams to integrate AI/ML models into production is essential.
Key Responsibilities:
  • Design, build, and maintain efficient data pipelines.
  • Manage and process large datasets using Spark, PySpark, and SQL.
  • Build and maintain real-time streaming applications with Spark Streaming/Kafka.
  • Collaborate with data scientists and product teams to integrate AI/ML models.
  • Ensure data quality, scalability, and performance in all pipelines.
  • Deploy and manage Spark workloads on cloud platforms (AWS, Azure, GCP, Databricks).
  • Automate testing and deployment of Spark jobs via CI/CD pipelines.
Requirements:
  • Bachelor's/Master's degree in Computer Science, Computer Engineering, or related field.
  • Minimum 4 years of professional experience as a Data Engineer.
  • Strong expertise in Python, Spark, PySpark.
  • Hands-on experience with Spark SQL, DataFrames, UDFs, DAG execution.
  • Knowledge of data ingestion tools (Kafka, Flume, Kinesis) and data formats (Parquet, Avro, ORC).
  • Proficiency in Airflow/Prefect for scheduling and orchestration.
  • Familiarity with performance tuning in Spark (partitioning, caching, broadcast joins).
  • Experience deploying on Databricks, AWS EMR, Azure HDInsight, or GCP Dataproc.
  • Exposure to testing and CI/CD for data pipelines (pytest, Jenkins, GitHub Actions).

  • Senior Data Pipeline Developer

    2 weeks ago

    Tiruppur, Tamil Nadu, India beBeeDataEngineer Full time ₹ 18,00,000 - ₹ 25,00,000

    We are seeking a skilled Data Engineer to join our team.The successful candidate will be responsible for designing and developing scalable data pipelines using IBM DataStage and AWS Glue.They will work closely with architects, business analysts, and data modelers to ensure timely and accurate delivery of critical data assets.Key ResponsibilitiesDesign,...

  • Senior Data Pipeline Specialist

    2 days ago

    Tiruppur, Tamil Nadu, India beBeeDataEngineer Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

    Job TitleBig Data ArchitectRole Description:As a senior data engineer, you will lead the design, development and deployment of scalable data pipelines using Apache Spark.Developing robust data processing workflows in PySpark is essential for delivering insights to our stakeholders.You will collaborate with cross-functional teams to ensure alignment on data...

  • Senior Data Engineer

    7 days ago

    Tiruppur, Tamil Nadu, India beBeeetl Full time ₹ 1,93,77,296 - ₹ 2,13,95,228

    About the RoleWe are seeking a skilled ETL Developer to join our data engineering team. The successful candidate will work on building scalable and efficient data pipelines using IBM DataStage (on Cloud Pak for Data), AWS Glue, and Snowflake.The role requires collaboration with architects, business analysts, and data modelers to ensure timely and accurate...

  • Senior Data Pipeline Developer

    1 week ago

    Tiruppur, Tamil Nadu, India beBeeData Full time ₹ 15,00,000 - ₹ 25,00,000

    Key Role: Building End-to-End Data Solutions on AzureWe are seeking an expert professional to design, develop, and run comprehensive data pipelines. This position involves handling data validation, transformation, and warehousing.Main Responsibilities:Develop scalable data solutions using Azure technologiesCollaborate with stakeholders for business insights...

  • Data Pipeline Architect

    1 day ago

    Tiruppur, Tamil Nadu, India beBeeData Full time ₹ 18,00,000 - ₹ 23,10,000

    Senior Data Engineer Job DescriptionWe are seeking a seasoned data engineer to design and optimize complex data pipelines. The ideal candidate will have a strong background in ETL, Cloud, Data Warehousing, and Big Data ecosystems.Key Responsibilities:Design and Optimize Data Pipelines: Build ETL/ELT workflows using AWS Glue, Airflow, Databricks, StreamSets,...

  • Senior Data Pipeline Validator

    2 days ago

    Tiruppur, Tamil Nadu, India beBeeDataQuality Full time ₹ 18,00,000 - ₹ 22,00,000

    ETL Data Quality EngineerWe are seeking a meticulous ETL Data Quality Engineer with expertise in testing large-scale data pipelines and ensuring data accuracy, consistency, and throughput in event-driven systems.Key Responsibilities:Test ETL processes and perform data reconciliation across systems to guarantee seamless data flow.Validate Kafka/Redpanda...

  • Senior Cloud Developer

    3 days ago

    Tiruppur, Tamil Nadu, India beBeeExpert Full time ₹ 80,00,000 - ₹ 1,50,00,000

    Job DescriptionAs a senior developer, you will play a crucial role in accelerating the delivery of next-generation AI workflows. You will work closely with our architect and developers to improve API performance, RCM human workflow automation, and drive AI adoption across our platform.Achieve significant reductions in cloud spend while scaling usage and...

  • Data Pipeline Engineer

    1 week ago

    Tiruppur, Tamil Nadu, India beBeeDeveloper Full time ₹ 1,50,00,000 - ₹ 2,00,00,000

    Job Title: Talend ETL DeveloperRequired Skills and Qualifications:Design and optimize complex data pipelines using TMC.Proficiency in BigQuery, including partitioning, clustering, federated queries, and advanced SQL.Expertise in Python for ETL scripting, automation, and integration with cloud data pipelines.Hands-on experience with PostgreSQL, including...

  • Senior ETL Developer

    2 days ago

    Tiruppur, Tamil Nadu, India Hyrhub Full time

    Role – Senior ETL Developer with 6+ Years of experienceNotice Period – Immediate JoinersLocation – Bangalore/Mangalore (Hybrid)Communication should be GoodJob Description:• Overall 6+ years of experience in development and maintenance in a data engineering project.• Collaborate with stakeholders to understand data requirements and translate them...

  • Senior Software Developer

    1 week ago

    Tiruppur, Tamil Nadu, India beBeeBackend Full time ₹ 1,50,00,000 - ₹ 2,00,00,000

    Develop high-quality web solutions and collaborate closely with our development team as a senior backend developer.The Websites team is responsible for building CMS systems that receive over 2 million hits weekly.You will engage with agile methods and adaptive frameworks to meet deadlines and follow best practices.Key ResponsibilitiesDesign and implement...