Big Data Pipeline Developer

2 weeks ago


India DataE2E Technologies Full time
Job Requirements

We are seeking an experienced Big Data Pipeline Developer to join our team. The ideal candidate will have 5+ years of hands-on experience with Azure Databricks and big data processing frameworks. Strong expertise in PySpark, Spark SQL, and distributed data processing is required. Good understanding of Azure Data Lake Storage (ADLS Gen2), Azure Data Factory, and Azure Synapse is also necessary. Proficient in writing complex SQL queries and optimizing them for performance.

Responsibilities:
  • Design, develop, and maintain scalable data pipelines using Azure Databricks, PySpark, and SQL.
  • Implement data ingestion, transformation, and integration processes from various data sources (structured and unstructured).
  • Optimize big data workflows for performance and cost-efficiency in Azure environments.
  • Work with Azure Data Lake, Azure Synapse Analytics, Azure Data Factory, and other services for end-to-end data management.


  • India beBee Careers Full time

    Job Requirements:We require an experienced Data Engineer to design, implement, and test data pipelines and products using cutting-edge technologies.Key ResponsibilitiesThis role involves designing and developing scalable data architectures and implementing data pipelines using various technologies.Key Qualifications:Bachelor's or Master's degree in Computer...


  • India beBee Careers Full time

    Job OverviewAs a Data Engineer II, you will be responsible for designing, building and maintaining the complete lifecycle of big data infrastructure from collection and storage to analysis and implementation. You will work with cutting-edge technologies to develop ETL processes, maintain data architecture, and collaborate with data science teams to solve...


  • India beBee Careers Full time

    Job Description:We need an experienced engineer to help us build and maintain our big data pipeline infrastructure. The successful candidate will have hands-on experience with data engineering, cloud platforms, and big data tools.Main Responsibilities:Design and develop efficient data pipelines using ETL/ELT techniques.Collaborate with data scientists and...


  • India beBee Careers Full time

    Data Engineering Role:We are seeking an experienced Data Engineer to lead the design, development, and optimization of large-scale data pipelines using Apache Spark, PySpark, and Airflow. The successful candidate will have hands-on experience with SQL skills, NoSQL databases, and data storage and retrieval processes.The ideal candidate will also have...

  • Big Data Developer

    3 days ago


    India beBee Careers Full time

    Job Overview">We are seeking a highly skilled Big Data Engineer with expertise in Java and Apache Spark to join our dynamic data engineering team. The ideal candidate will be responsible for designing, developing, and optimizing large-scale data processing systems and pipelines for enterprise-level applications.">Key Responsibilities">">Design and develop...


  • India beBee Careers Full time

    Role Overview:Sr Big Data Engineer (GCP)About the Role:This is an exciting opportunity to join our team as a Sr Big Data Engineer (GCP). The successful candidate will be responsible for designing, building, and maintaining robust data pipelines that support large-scale data processing and analytics.Responsibilities:Design and develop Big Data pipelines and...


  • India beBee Careers Full time

    Job DescriptionThe ideal candidate will play a critical role in building the data infrastructure that supports our business intelligence, analytics, and machine learning initiatives.About the RoleDevelop and optimize cloud data solutions using cloud-based data platforms (AWS, Azure, GCP).Collaborate with cross-functional teams to design, develop, and...


  • India beBee Careers Full time

    ResponsibilitiesAs a Data Engineer, you will be responsible for:Designing and implementing scalable ETL pipelines using Snowflake, Databricks, and AWSCurate and deliver reliable, validated datasets across multiple departmentsPartner with cross-functional teams to ensure seamless data integration and analyticsSolve pipeline failures and implement anomaly...


  • India beBee Careers Full time

    Big Data Solutions Developer Job DescriptionWe are seeking a highly skilled Big Data Solutions Developer to join our team. As a key member of our big data team, you will be responsible for designing and developing scalable big data solutions using Databricks.Main Responsibilities:Design and implement big data pipelines using DatabricksDevelop and maintain...


  • India beBee Careers Full time

    This is a challenging role that requires strong technical skills and experience in designing and implementing cloud-native data processing frameworks.Main TasksDesign and implement a highly scalable data processing framework using GCP and Spark.Develop efficient data pipelines using Databricks and Airflow for transforming raw data into valuable...