Senior Data Engineer

Job Description

Job Description

This is a remote position.

Job Summary

We are looking for a Senior Data Quality & Governance Engineer who will take ownership of enforcing data contracts, quality, and metadata standards across intelligence and analytics platforms. This role involves designing lineage models, implementing statistical validations, and establishing reusable quality dashboardsall using AWS-native and lightweight open-source tools.

Responsibilities

- Design and enforce data contracts across Raw Clean Modeled zones.
- Define schema versioning policies, KPI logic, and metadata rules across business-critical datasets.
- Implement row-level and statistical validations using Great Expectations or Deequ.
- Create audit-ready QA tables to track failed checks, schema mismatches, and data regressions.
- Track end-to-end lineage and KPI evolution using OpenMetadata (with Glue/S3/Athena).
- Auto-classify columns as PII, derived, or forecast-driving fields using AWS Glue Tags/Scripts.
- Provide CTAS-based Athena queries for building QA dashboards and UAT verifications.
- Build BI-ready, QA-approved datasets for downstream tools like Superset and Power BI.
- Establish reusable profiling and validation dashboards for data quality and business teams.
- Collaborate with engineers, QA, and business SMEs to finalize data validation logic.

Requirements

Essential Skills:

Job

- Hands-on experience with AWS Glue (Jobs, Crawlers, Catalog), S3, and Athena.
- Strong foundation in data contracts, quality enforcement, and schema versioning.
- Expertise in using Deequ or Great Expectations for anomaly detection and data validation.
- Familiarity with OpenMetadata, Amundsen, or custom metadata tracking solutions.
- Ability to tag and manage sensitive data fields (e.g., PII, model inputs, derived KPIs).
- Strong SQL with Athena (CTEs, CTAS, filters, aggregations).
- Experience building QA dashboards in Superset, Streamlit, or similar BI tools.

Personal

- Excellent collaboration and communication with QA, architects, and business teams.
- Self-driven with attention to detail in schema accuracy and metadata enrichment.
- Ability to translate KPIs and quality rules into validation logic.
- Proactive in surfacing data regressions and audit issues before they reach production.
- High ownership mindset with a strong data compliance and governance attitude.

Preferred Skills

Job

- Implementation of statistical QA techniques like z-score anomalies or entropy thresholds.
- Experience handling rejected record logs, schema drift validations, and data reconciliation.
- Awareness of AWS cost optimization techniques in Glue and Athena.

Personal

- Proactive, ownership-driven mindset with a collaborative approach.
- Strong communication and collaboration skills.
- Strong problem-solving skills with attention to detail.
- Have the ability to work under stringent deadlines and demanding client conditions.
- Strong analytical and problem-solving skills.
- Ability to work in fast-paced, delivery-focused environments.
- Should have strong mentoring and documentation skills.
- Ability to take end-to-end ownership of QA validation modules.

Other Relevant Information

- Bachelors degree in Computer Science, Information Technology, or a related field.
- Minimum 9+ years of experience in data engineering & architecture.

Benefits

- This role offers the flexibility of working remotely in India.

LeewayHertz is an equal opportunity employer and does not discriminate based on race, color, religion, sex, age, disability, national origin, sexual orientation, gender identity, or any other protected status. We encourage a diverse range of applicants.

check(event) ; career-website-detail-template-2 => apply(record.id,meta)' mousedown='lyte-button => check(event)' final-style='background-color:#6875E2;border-color:#6875E2;color:white;' final-class='lyte-button lyteBackgroundColorBtn lyteSuccess' lyte-rendered=''>

---