Senior Data Engineer

Job Description

Experience: 6+ Years

Role: Senior Data Engineer

Location: Coimbatore- Work from Office

Employment Type: Full-time

Availability: Immediate Joiner Preferred

Role Summary

We are seeking an experienced Data Engineer with 6+ years of experience in designing, developing, and maintaining scalable data pipelines and data warehouse solutions. The ideal candidate should have strong expertise in ETL/ELT, Python, dbt, Snowflake, workflow orchestration, job scheduling, and Git.

Key Responsibilities

* Design, develop, test, and maintain scalable ETL/ELT data pipelines.

* Build and manage enterprise data warehouse solutions using Snowflake.

* Develop reusable data transformation models using dbt.

* Write efficient and maintainable Python programs for data ingestion, transformation, validation, and automation.

* Extract and integrate data from databases, APIs, flat files, and other structured or semi-structured sources.

* Develop dimensional data models using concepts such as:

   * Star and snowflake schemas

   * Fact and dimension tables

   * Slowly Changing Dimensions

   * Surrogate keys

   * Incremental and historical data loads

* Create and maintain ETL jobs, schedules, dependencies, alerts, and recovery procedures.

* Work with orchestration and scheduling tools such as Apache Airflow, Control-M, Azure Data Factory, AWS Glue, or similar platforms.

* Implement data-quality checks, reconciliation processes, audit controls, and exception handling.

* Monitor pipeline performance and troubleshoot failed or delayed data-processing jobs.

* Optimize Snowflake queries, storage, warehouses, and overall pipeline performance.

* Use Git for source-code management, branching, merging, pull requests, code reviews, and release management.

* Support CI/CD processes for deploying data pipelines and dbt models across environments.

* Collaborate with business analysts, data analysts, reporting teams, architects, and other stakeholders to understand data requirements.

* Prepare and maintain technical documentation, data mappings, lineage, and operational support procedures.

* Participate in Agile ceremonies, sprint planning, estimation, peer reviews, and production-support activities.

Required Skills and Qualifications

* 6+ years of professional experience in data engineering, ETL development, or data warehousing.

* Strong understanding of ETL/ELT processes and data warehousing concepts.

* Hands-on experience with Python for data processing and automation.

* Strong hands-on experience with dbt, including:

   * Models and materializations

   * Sources and seeds

   * Tests and documentation

   * Macros and Jinja

   * Incremental models

   * Snapshots

   * Hooks and packages

* Strong experience with Snowflake, including:

   * Databases, schemas, tables, and views

   * Virtual warehouses

   * Stages and file formats

   * Snowpipe

   * Streams and Tasks

   * Time Travel and zero-copy cloning

   * Query and warehouse performance optimization

   * Role-based access control

* Advanced SQL skills, including joins, CTEs, window functions, stored procedures, and query optimization.

* Experience creating, scheduling, monitoring, and troubleshooting ETL jobs.

* Experience with workflow orchestration tools such as Apache Airflow or an equivalent platform.

* Strong working knowledge of Git, including branching strategies, merge conflict resolution, pull requests, and code reviews.

* Experience working with structured and semi-structured data formats such as CSV, JSON, XML, Avro, and Parquet.

* Knowledge of data-quality frameworks, logging, monitoring, reconciliation, and error-handling practices.

* Good understanding of software development life cycle, Agile methodologies, and production-support processes.

* Strong analytical, troubleshooting, communication, and problem-solving skills.

Preferred Qualifications

* Experience with cloud platforms such as AWS, Microsoft Azure, or Google Cloud Platform.

* Experience implementing CI/CD pipelines using GitHub Actions, GitLab CI, Jenkins, Azure DevOps, or similar tools.

* Familiarity with BI and reporting tools such as Power BI, Tableau, or Looker.

* Understanding of data governance, metadata management, lineage, security, and access control.

* Relevant Snowflake, dbt, cloud, or data-engineering certifications would be an advantage.

Expected Competencies <