Sr. Python Developer

Job Description: Python Developer / Data Engineer – Talend to Python Migration

About the Role

We are seeking experienced Python Developers/Data Engineers to join our team for a strategic migration project, moving enterprise ETL workloads from Talend to a modern Python-based ingestion framework on Google Cloud Platform (GCP). You will work closely with architects, platform engineers, and automation teams to design, build, and optimize scalable data pipelines, leveraging Airflow, Cloud Run, Dataproc, and BigQuery.

Key Responsibilities

* ETL Migration & Development

* Analyze existing Talend jobs/modules and design equivalent Python-based ETL workflows.

* Develop, test, and deploy Python scripts/functions for ingestion, preprocessing, file management, and data loading (SFTP to GCS, file unzip, preprocessing, raw/harmonized table loads, DQ checks, view creation).

* Implement orchestration using Airflow DAGs and Cloud Run jobs, ensuring modular, maintainable, and scalable solutions.

* Integrate with automation tools (Automic) for job triggering, monitoring, and error handling.

* Platform Configuration & Optimization

* Configure and tune serverless workloads (Cloud Run, Dataproc, BigQuery) for performance, stability, and cost efficiency (e.g., retry strategies, timeout settings, CPU/memory sizing, autoscaling).

* Apply best practices for resource allocation, parallelism, and batch processing in Python and PySpark jobs.

* Data Quality & Observability

* Implement data quality checks, logging, and monitoring for traceability and operational excellence.

* Ensure robust error handling, retry logic, and alerting for ETL pipelines.

* Security & Compliance

* Work with IAM, service accounts, and Secret Manager for secure credential management and access control.

* Adhere to data governance, privacy, and compliance requirements.

* Collaboration & Documentation

* Collaborate with cross-functional teams (platform, DevOps, data governance) to align on requirements and deliverables.

* Document migration processes, configuration parameters, and operational procedures.

Required Skills & Experience

* 5+ years of experience in Python development, with a focus on data engineering and ETL pipeline design.

* Hands-on experience with GCP services: Cloud Run, Dataproc, BigQuery, Cloud Composer (Airflow), Cloud Storage.

* Strong understanding of Talend ETL concepts and experience migrating Talend jobs to Python is highly desirable.

* Experience with orchestration tools (Airflow, Automic) and automation of data workflows.

* Proficiency in containerization (Docker), CI/CD, and artifact management.

* Familiarity with data quality frameworks, logging, and monitoring tools.

* Solid grasp of IAM, service accounts, and cloud security best practices.

* Excellent problem-solving, communication, and documentation skills.

Preferred Qualifications

* Experience with PySpark, serverless Spark jobs, and advanced resource tuning.

* Exposure to enterprise data platforms, large-scale migrations, and cloud-native architectures.

* Knowledge of GCP networking, firewall rules, and hybrid connectivity.

Sample Roles & Responsibilities for Job Portal Posting

Role/Responsibility Description

ETL Migration Migrate Talend ETL jobs/modules to Python-based workflows on GCP.

Pipeline Development Build scalable, modular data pipelines using Python, Airflow, Cloud Run, and Dataproc.

Platform Configuration Tune serverless workloads for optimal performance and cost efficiency.

Data Quality & Monitoring Implement DQ checks, logging, and monitoring for all ETL processes.

Security & IAM Manage credentials, service accounts, and access policies in GCP.

Collaboration Work with architects, DevOps, and data governance teams to deliver migration objectives.

Documentation Maintain clear documentation of migration steps, configurations, and operational guides.

Job Type: Full-time

Pay: ₹338, ₹1,509,899.56 per year

Work Location: In person