GCP & PySpark with ETL - Lead

7 days ago

Kochi Thiruvananthapuram, India UST Full time ₹ 12,00,000 - ₹ 36,00,000 per year

Key Responsibilities

  • Design, develop, and optimize ETL pipelines using PySpark on Google Cloud Platform (GCP).
  • Work with BigQuery, Cloud Dataflow, Cloud Composer (Apache Airflow), and Cloud Storage for data transformation and orchestration.
  • Develop and optimize Spark-based ETL processes for large-scale data processing.
  • Implement best practices for data governance, security, and monitoring in a cloud environment.
  • Collaborate with data engineers, analysts, and business stakeholders to understand data requirements.
  • Troubleshoot performance bottlenecks and optimize Spark jobs for efficient execution.
  • Automate data workflows using Apache Airflow or Cloud Composer.
  • Ensure data quality, validation, and consistency across pipelines.
  • 5+ years of experience in ETL development with a focus on PySpark.
  • Strong hands-on experience with Google Cloud Platform (GCP) services, including:
  • BigQuery
  • Cloud Dataflow / Apache Beam
  • Cloud Composer (Apache Airflow)
  • Cloud Storage
  • Proficiency in Python and PySpark for big data processing.
  • Experience with data lake architectures and data warehousing concepts.
  • Knowledge of SQL for data querying and transformation.
  • Experience with CI/CD pipelines for data pipeline automation.
  • Strong debugging and problem-solving skills.
  • Experience with Kafka or Pub/Sub for real-time data processing.
  • Knowledge of Terraform for infrastructure automation on GCP.
  • Experience with containerization (Docker, Kubernetes).
  • Familiarity with DevOps and monitoring tools like Prometheus, Stackdriver, or Datadog.

Required Skills

Gcp,Pyspark,Etl

  • Lead GCP Data Engineer

    4 weeks ago

    Thiruvananthapuram, India People Prime Worldwide Full time

    Important Note (Please Read Before Applying)🚫 Do NOT apply if: • You have less than 10 years of relevant IT experience • You lack hands-on Python + GCP project experience • You have no real-time experience with Dataflow, BigQuery, Airflow, or Spark • You are not experienced in architecting or designing data pipelines • You cannot work in the...

  • Aws Pyspark Architect

    2 weeks ago

    Thiruvananthapuram, Kerala, India UST Global Full time

    9 - 12 Years- 1 Opening- Thiruvananthapuram**Role description**: **Responsibilities**: - AWS Data Architecture Design, implement, and maintain scalable and robust data architectures on AWS. - Utilize AWS services such as S3, Glue, and Redshift for data storage, processing, and analytics. - ETL Development with PySpark Develop and implement ETL processes...

  • Azure with Pyspark

    2 weeks ago

    Thiruvananthapuram, Kerala, India UST Global Full time

    9 - 12 Years- 4 Openings- Thiruvananthapuram**Role description**: **Responsibilities**: **Data Pipeline Development**: - Design, implement, and optimize end-to-end data pipelines on Azure, focusing on scalability and performance. - Develop and maintain ETL workflows for seamless data processing. **Azure Cloud Expertise**: - Utilize Azure services such as...

  • Etl Tester

    2 weeks ago

    Kochi, India Ambc Technologies Full time

    **Position**: ETL Tester **Location**: Cochin, Kerala **Experience**: 4 to 6 Years **For Data Testing key points**: - Technical Skills - Should be expertise in **SQL**. An expert level knowledge/problem solving is needed for the position. - Test Process - Good knowledge on test process. - ETL testing - Should be expertise on **ETL process**: - BI...

  • GCP Data Engineer

    2 weeks ago

    Kochi, India E-Solutions Full time

    Job Role: GCP Data EngineerLocation: Pan IndiaExperience: 5+ yearsType of Hire: PermanentRequired Technical Skill Set: GCP, PySpark, Python, HDFS, Hadoop, SQLMust Have Skills:• Good Handson Experience on GCP• Should have worked on Data Migration projects, from On-prem to Cloud• Should have Cloud Storage Knowledge, Big Query, Cluster Knowledge• Sound...

  • GCP Data Engineer

    2 weeks ago

    Kochi, India E-Solutions Full time

    Job Role: GCP Data EngineerLocation: Pan IndiaExperience: 5+ yearsType of Hire: PermanentRequired Technical Skill Set: GCP, PySpark, Python, HDFS, Hadoop, SQLMust Have Skills:• Good Handson Experience on GCP• Should have worked on Data Migration projects, from On-prem to Cloud• Should have Cloud Storage Knowledge, Big Query, Cluster Knowledge• Sound...

  • ETL Lead

    4 weeks ago

    Thiruvananthapuram, India CONNECTING 2 WORK Full time

    Job Description ETL Lead Desired Profile Receives assignments from technology team lead in the form of objectives and technical designs to implement technical solution Manages his deliverables as well as team deliverables and raises the bar by implementing latest technologies. Engages with team members business analysts, client service teams and data...

  • ETL Lead

    2 weeks ago

    Thiruvananthapuram, India CONNECTING 2 WORK Full time

    Job Description ETL Lead Desired Profile Receives assignments from technology team lead in the form of objectives and technical designs to implement technical solution Manages his deliverables as well as team deliverables and raises the bar by implementing latest technologies. Engages with team members business analysts, client service teams and data...

  • Data Warehousing Solutions Architect

    4 days ago

    thiruvananthapuram, India beBeeData Full time

    Job Title: Data Science and Engineering ExpertWe are seeking an experienced professional with a strong track record in designing, building, and optimizing scalable data pipelines using Snowflake and Databricks. The ideal candidate will have expertise in cloud-based data engineering frameworks, data warehousing solutions, and distributed data processing.The...

  • Cloud Data Systems Architect

    4 days ago

    thiruvananthapuram, India beBeeData Full time

    Job Title: Cloud Data Systems ArchitectAs a seasoned professional with 5+ years of experience, you will be responsible for designing, building, and maintaining large-scale data processing systems on the Google Cloud Platform. Your expertise in GCP's data side, storage classes, Dataflow, Big Query, PySpark/Python, and Airflow will be crucial in driving...