Senior Data Engineer

About the RoleWe are seeking a seasoned Data Integration Specialist with expertise in designing and developing scalable data pipelines. The ideal candidate will have a deep understanding of cloud-based data integration platforms, specifically Google Cloud Platform (GCP).The successful Data Integration Specialist will be responsible for architecting, building, and maintaining robust data pipelines that integrate with various data sources and destinations. This includes leveraging GCP services such as BigQuery, AlloyDB, Cloud Storage, Cloud Composer, and Pub/Sub.Design and develop data pipelines using GCP native tools and Python-based orchestration frameworks.Leverage Cloud Composer (Airflow) to orchestrate and automate data workflows.Implement data quality and lineage frameworks to ensure data integrity and accountability.This is a hands-on role requiring strong analytical and problem-solving skills. The ideal candidate will be able to collaborate with cross-functional teams to ensure seamless data flow across GCP components.Key Responsibilities1. Data Orchestration and Scheduling:Create and manage DAGs in Cloud Composer 2 to schedule and monitor data workflows.Develop asynchronous or parallel execution strategies to optimize pipeline performance under GCP constraints.2. Architecture & Design Enhancement:Collaborate with solution architects to review and refine pipeline architecture.Make design updates and code refactoring based on evolving data requirements, schema changes, or performance improvements.3. Performance Optimization:Tune BigQuery queries and AlloyDB import strategies for large datasets.Implement partitioning, batching, and retry mechanisms for high throughput and reliability.4. Monitoring & Logging:Implement detailed logging, alerting, and monitoring using Cloud Logging, Cloud Monitoring, and Stackdriver.Set up job-level and table-level audit trails for pipeline observability and troubleshooting.5. Security & Compliance:Use Service Accounts, VPC Service Controls, IAM roles, and CMEK encryption to ensure data security and governance compliance.Adhere to enterprise security policies and guardrails for data movement across GCP projects.Required Skills and Experience• 5+ years of experience in data engineering or cloud data integration roles.• Strong expertise in Google Cloud Platform (GCP), including:BigQuery (SQL, views, partitioning), AlloyDB / PostgreSQL (import/export, COPY operations), Cloud Storage (buckets, lifecycle policies), Cloud Composer (Airflow), Cloud Functions, Pub/Sub, and EventArc.• Proven experience building ETL/ELT data pipelines and automated workflows using GCP native tools or Python-based orchestration.• Proficiency in Python (Airflow operators, GCP SDK, REST API integration).• Strong SQL and database schema design skills.• Familiarity with asynchronous processing, retry handling, and GCP APIs for data import/export.• Understanding of data quality, lineage, and audit frameworks.• Excellent analytical and problem-solving abilities.• Strong documentation and communication skills.• Ability to adapt to changing requirements and propose alternative designs.• Collaborative mindset with cross-functional teams (architecture, security, operations).BenefitsThis role offers a unique opportunity to work with cutting-edge technologies and contribute to the development of innovative data integration solutions. As a member of our team, you can expect:a dynamic work environment, opportunities for growth and professional development, collaborative teamwork, and competitive compensation and benefits package.],