GCP & PySpark with ETL - Lead

3 days ago

Kochi Thiruvananthapuram, India UST Full time ₹ 12,00,000 - ₹ 36,00,000 per year

Key Responsibilities

  • Design, develop, and optimize ETL pipelines using PySpark on Google Cloud Platform (GCP).
  • Work with BigQuery, Cloud Dataflow, Cloud Composer (Apache Airflow), and Cloud Storage for data transformation and orchestration.
  • Develop and optimize Spark-based ETL processes for large-scale data processing.
  • Implement best practices for data governance, security, and monitoring in a cloud environment.
  • Collaborate with data engineers, analysts, and business stakeholders to understand data requirements.
  • Troubleshoot performance bottlenecks and optimize Spark jobs for efficient execution.
  • Automate data workflows using Apache Airflow or Cloud Composer.
  • Ensure data quality, validation, and consistency across pipelines.
  • 5+ years of experience in ETL development with a focus on PySpark.
  • Strong hands-on experience with Google Cloud Platform (GCP) services, including:
  • BigQuery
  • Cloud Dataflow / Apache Beam
  • Cloud Composer (Apache Airflow)
  • Cloud Storage
  • Proficiency in Python and PySpark for big data processing.
  • Experience with data lake architectures and data warehousing concepts.
  • Knowledge of SQL for data querying and transformation.
  • Experience with CI/CD pipelines for data pipeline automation.
  • Strong debugging and problem-solving skills.
  • Experience with Kafka or Pub/Sub for real-time data processing.
  • Knowledge of Terraform for infrastructure automation on GCP.
  • Experience with containerization (Docker, Kubernetes).
  • Familiarity with DevOps and monitoring tools like Prometheus, Stackdriver, or Datadog.

Required Skills

Gcp,Pyspark,Etl

  • Lead GCP Data Engineer

    23 hours ago

    Thiruvananthapuram, India People Prime Worldwide Full time

    Important Note (Please Read Before Applying)🚫 Do NOT apply if: • You have less than 10 years of relevant IT experience • You lack hands-on Python + GCP project experience • You have no real-time experience with Dataflow, BigQuery, Airflow, or Spark • You are not experienced in architecting or designing data pipelines • You cannot work in the...

  • ETL Lead

    1 week ago

    Thiruvananthapuram, India CONNECTING 2 WORK Full time

    Job Description ETL Lead Desired Profile Receives assignments from technology team lead in the form of objectives and technical designs to implement technical solution Manages his deliverables as well as team deliverables and raises the bar by implementing latest technologies. Engages with team members business analysts, client service teams and data...

  • ETL Lead

    2 hours ago

    Thiruvananthapuram, India CONNECTING 2 WORK Full time

    Job Description ETL Lead Desired Profile Receives assignments from technology team lead in the form of objectives and technical designs to implement technical solution Manages his deliverables as well as team deliverables and raises the bar by implementing latest technologies. Engages with team members business analysts, client service teams and data...

  • Data Engineering Lead

    1 week ago

    Kochi, India Velodata Global Pvt Ltd Full time

    About the job :Location : Kochi / Bangalore (initially in Kochi)Role Summary :We are seeking a Data Engineering Lead to architect, build, and optimize scalable data pipelines and data platforms that power critical business decisions. The ideal candidate will have deep expertise in PySpark, ETL frameworks, SQL Stored Procedures, and Data Modeling, along with...

  • Data Architect/Senior Data Engineer

    2 weeks ago

    Thiruvananthapuram, Thiruvananthapuram / Trivandrum, India Quest Global Full time

    Job Description Job Requirements Job Title: AWS Data Engineer Company: Quest Global Job Category: Multi Industry - MedTech Quest Global is seeking a highly skilled AWS Data Engineer with 6+ years of experience in AWS, Databricks, Pyspark and S3. As an AWS Data Engineer, you will be responsible for designing, developing, and maintaining data pipelines,...

  • Senior ETL Test Engineer

    1 week ago

    thiruvananthapuram, India Quantiphi Full time

    Role & Responsibilities: ● Develop and execute comprehensive test plans and test cases for ETL processes, data warehouses, and data marts, ensuring data integrity and accuracy. This includes validating source-to-target mappings, transformation logic, and data loading. ● Write and execute complex SQL queries to validate data at various stages of the ETL...

  • Senior ETL Test Engineer

    1 week ago

    Thiruvananthapuram, India Quantiphi Full time

    Role & Responsibilities: ● Develop and execute comprehensive test plans and test cases for ETL processes, data warehouses, and data marts, ensuring data integrity and accuracy. This includes validating source-to-target mappings, transformation logic, and data loading.● Write and execute complex SQL queries to validate data at various stages of the ETL...

  • Senior ETL Test Engineer

    1 week ago

    Thiruvananthapuram, India Quantiphi Full time

    Role & Responsibilities: ● Develop and execute comprehensive test plans and test cases for ETL processes, data warehouses, and data marts, ensuring data integrity and accuracy. This includes validating source-to-target mappings, transformation logic, and data loading. ● Write and execute complex SQL queries to validate data at various stages of the ETL...

  • Senior ETL Test Engineer

    1 week ago

    thiruvananthapuram, India Quantiphi Full time

    Role & Responsibilities: ● Develop and execute comprehensive test plans and test cases for ETL processes, data warehouses, and data marts, ensuring data integrity and accuracy. This includes validating source-to-target mappings, transformation logic, and data loading.● Write and execute complex SQL queries to validate data at various stages of the ETL...

  • Lead Ii

    4 weeks ago

    Thiruvananthapuram, Kerala, India UST Full time

    Job Summary We are looking for a highly skilled Senior Data Engineer with over 6 years of experience in data engineering and a strong focus on data migration projects The ideal candidate will have deep expertise in Azure Data Factory ADF ETL ELT processes and Python PySpark with proven experience building scalable and efficient data pipelines across diverse...