Gcp Data Engineer
2 weeks ago
Hyderabad Pune Chennai, TG, India
Elabs Infotech
Remote
Full-time
Free with email or Google
Save this job and keep your search organized
Create a free account to save jobs, create alerts and return to this listing from your dashboard.
Free with email or Google
By continuing, you agree to our Terms & Privacy Policy.
## Job highlights
- Expertise in GCP including IAM strategy, advanced Python and GoogleSQL, Cloud Composer/Airflow, Dataflow (Apache Beam), Dataproc (Spark), and BigQuery optimization
- Design and implement complex data pipelines, lead Cloud Composer orchestration, build reusable Dataflow templates, perform BigQuery tuning, troubleshoot pipeline issues, review code, and mentor engineers
Responsibilities:
- Design and implement complex data pipelines using Dataflow (Apache Beam) and Dataproc (Spark).
- Lead Cloud Composer (Airflow) orchestration design, optimizing DAG structure, dependencies, and retry strategies.
- Drive advanced Python and SQL implementations, including reusable frameworks and utility components.
- Build and standardize custom Dataflow / Beam templates and reusable pipeline patterns.
- Perform deep performance tuning for BigQuery (query optimization, partitioning strategies, cost optimization).
- Troubleshoot complex pipeline failures and performance bottlenecks across processing and orchestration layers.
- Review code and design artifacts produced by engineers; provide technical guidance and mentoring.
- Own technical quality for assigned modules, including non-functional aspects (performance, reliability, scalability).
- Act as the first point of escalation for technical issues before involving architects.
- Collaborate with architects to understand design intent and ensure correct implementation. Mandatory Technical
Skills:
- Advanced GCP Knowledge
- Strong understanding of IAM strategy, service isolation, and project-level design considerations.
- Programming & Querying
- Advanced proficiency in Python and complex GoogleSQL.
- Orchestration
- Strong hands-on expertise with Cloud Composer / Airflow, including best practices and optimization.
- Data Processing
- Deep hands-on experience with Dataflow and Apache Beam.
- Strong working knowledge of Dataproc / Spark internals.
- Analytics Platform
- Advanced experience with BigQuery, including performance and cost optimization techniques.
- Expertise in GCP including IAM strategy, advanced Python and GoogleSQL, Cloud Composer/Airflow, Dataflow (Apache Beam), Dataproc (Spark), and BigQuery optimization
- Design and implement complex data pipelines, lead Cloud Composer orchestration, build reusable Dataflow templates, perform BigQuery tuning, troubleshoot pipeline issues, review code, and mentor engineers
Responsibilities:
- Design and implement complex data pipelines using Dataflow (Apache Beam) and Dataproc (Spark).
- Lead Cloud Composer (Airflow) orchestration design, optimizing DAG structure, dependencies, and retry strategies.
- Drive advanced Python and SQL implementations, including reusable frameworks and utility components.
- Build and standardize custom Dataflow / Beam templates and reusable pipeline patterns.
- Perform deep performance tuning for BigQuery (query optimization, partitioning strategies, cost optimization).
- Troubleshoot complex pipeline failures and performance bottlenecks across processing and orchestration layers.
- Review code and design artifacts produced by engineers; provide technical guidance and mentoring.
- Own technical quality for assigned modules, including non-functional aspects (performance, reliability, scalability).
- Act as the first point of escalation for technical issues before involving architects.
- Collaborate with architects to understand design intent and ensure correct implementation. Mandatory Technical
Skills:
- Advanced GCP Knowledge
- Strong understanding of IAM strategy, service isolation, and project-level design considerations.
- Programming & Querying
- Advanced proficiency in Python and complex GoogleSQL.
- Orchestration
- Strong hands-on expertise with Cloud Composer / Airflow, including best practices and optimization.
- Data Processing
- Deep hands-on experience with Dataflow and Apache Beam.
- Strong working knowledge of Dataproc / Spark internals.
- Analytics Platform
- Advanced experience with BigQuery, including performance and cost optimization techniques.