Urgent ETL Developer

Job Description

Key Responsibilities:

1. ETL Pipeline Development:

- Design, develop, and maintainscalable ETL processesto extract, transform, and load data from various structured and unstructured sources intoGCP-based data warehouses (BigQuery, Cloud SQL, Cloud Storage, etc.).
- Develop efficient SQL queries and scripts to supportdata transformation, aggregation, and validation.
- Optimize ETL workflows to ensurelow-latency data processingand high performance.

2. Google Cloud Dataform & Data Transformation:

- UtilizeGoogle Cloud Dataformto implementSQL-based data transformations in BigQueryfollowingbest practices in data modeling, version control, and dependency management.
- Developmodular SQL workflowsusing Dataform to simplify transformation logic and enhance reusability.
- Integrate Dataform into existingETL/ELT pipelinesto streamlinedata engineering and analytics workflows.
- LeverageDataform's automated testing, scheduling, and Git-based version controlforcollaborative development and data quality assurance.

3. Data Integration & Management:

- Work with diverse data sources (databases, APIs, streaming data, and cloud storage) tointegrate data into centralized repositories.
- Ensuredata consistency, integrity, and accuracythrough rigorous testing and validation.
- Implementincremental data loads, change data capture (CDC), and batch/real-time ETL strategies.
- LeverageGCP services like Dataflow, Dataproc, Cloud Functions, and Pub/Subto handle data ingestion and transformation.

4. Database & SQL Development:

- Writecomplex SQL queries, stored procedures, and functionsto support analytical and operational data needs.
- Optimize SQL queries forperformance tuning and cost efficiencyinBigQuery, Cloud SQL, and other relational databases.
- Ensure proper indexing, partitioning, and clustering strategies for optimal query performance.

5. Cloud & DevOps Integration:

- Deploy and monitor ETL workflows usingGCP-native tools (Cloud Composer/Airflow, Dataform, Dataflow, Dataprep, etc.).
- ImplementCI/CD pipelinesfor ETL jobs usingTerraform, Cloud Build, GitHub Actions, or Jenkins.
- Work with Infrastructure and DevOps teams to ensuresecure and reliable deployment of ETL solutionsin a cloud environment.

6. Data Quality & Governance:

- Implementdata validation, data cleansing, and error-handling mechanismsin ETL pipelines.
- Monitor data pipeline performance and ensuretimely resolution of issues and failures.
- Work with stakeholders to definedata governance policies, metadata management, and access controls.

7. Documentation & Collaboration:

- Maintaincomprehensive documentationfor ETL workflows, data transformations, and technical design.
- Collaborate withdata engineers, data analysts, and business teamsto understand data needs and optimize data processing workflows.
- Conductcode reviews and provide mentorshipto junior developers when necessary.

Required Skills & Qualifications:

1. Technical Skills:

ETL Development:

- Hands-on experience in designing and implementing ETL pipelines.
- Proficiency inETL toolssuch asApache Airflow (Cloud Composer), Dataflow, or Informatica.

SQL & Database Management:

- Strong expertise inSQL (DDL, DML, performance tuning, indexing, partitioning, stored procedures, etc.).
- Experience working withrelational (Cloud SQL, PostgreSQL, MySQL) and NoSQL databases (Bigtable, Firestore, MongoDB, etc.).

Cloud (GCP) Expertise:

- Strong hands-on experience withGoogle Cloud Platform (GCP)services:
- BigQuery(data warehousing & analytics)
- Cloud Storage(data lake storage)
- Cloud Composer (Apache Airflow)(workflow orchestration)
- Cloud Functions(serverless ETL tasks)
- Cloud Dataflow(Apache Beam-based data processing)
- Pub/Sub(real-time streaming)
- Dataproc(Hadoop/Spark-based processing)
- Google Cloud Dataform(SQL-based transformations for BigQuery)

Programming & Scripting:

- Experience withPython, SQL scripting, and Shell scriptingfor ETL automation.
- Knowledge ofPySpark or Apache Beamis a plus.

CI/CD & DevOps:

- Experience indeploying ETL workflowsusingTerraform, Cloud Build, or Jenkins.
- Familiarity withGit/GitHub for version control.

---