Senior Data Developer

Job Opportunity:

Azure Data Engineer Role

We are seeking a skilled data engineer to design, build, and maintain automated data pipelines supporting analytics initiatives. The successful candidate will focus on building production-ready workflows for ingestion, transformation, quality checks, lineage capture, access auditing, cost usage analysis, retention tracking, and metadata integration primarily using Azure Databricks, Azure Data Lake, and Microsoft Purview.

This role requires a strong background in data engineering with a focus on Azure and Databricks. Proficiency in PySpark, SQL, and ETL/ELT pipeline design is essential. The ideal candidate should have demonstrated experience building granular data quality checks and integrating governance logic into pipelines.

About the Role:

1. Pipeline Development: Design, build, and deploy robust ETL/ELT pipelines in Databricks (PySpark, SQL, Delta Lake) to ingest, transform, and curate governance and operational metadata from multiple sources landed in Databricks.
2. Data Quality Capture: Implement profiling logic to capture issue-level metadata (source table, column, timestamp, severity, rule type) to support drill-down from dashboards into specific records and enable targeted remediation.
3. Governance Metrics Automation: Develop data pipelines to generate metrics for dashboards covering data quality, lineage, job monitoring, access & permissions, query cost, usage & consumption, retention & lifecycle, policy enforcement, sensitive data mapping, and governance KPIs.
4. Microsoft Purview Integration: Automate asset onboarding, metadata enrichment, classification tagging, and lineage extraction for integration into governance reporting.
5. Data Retention & Policy Enforcement: Implement logic for retention tracking and policy compliance monitoring (masking, RLS, exceptions).
6. Job & Query Monitoring: Build pipelines to track job performance, SLA adherence, and query costs for cost and performance optimization.
7. Metadata Storage & Optimization: Maintain curated Delta tables for governance metrics, structured for efficient dashboard consumption.
8. Testing & Troubleshooting: Monitor pipeline execution, optimize performance, and resolve issues quickly.
9. Collaboration: Work closely with the lead engineer, QA, and reporting teams to validate metrics and resolve data quality issues.
10. Security & Compliance: Ensure all pipelines meet organizational governance, privacy, and security standards.

Required Skills and Qualifications:

• Bachelor's degree in Computer Science, Engineering, Information Systems, or related field.
• 4+ years of hands-on data engineering experience, with Azure Databricks and Azure Data Lake.
• Proficiency in PySpark, SQL, and ETL/ELT pipeline design.
• Demonstrated experience building granular data quality checks and integrating governance logic into pipelines.
• Working knowledge of Microsoft Purview for metadata management, lineage capture, and classification.
• Experience with Azure Data Factory or equivalent orchestration tools.
• Understanding of data modeling, metadata structures, and data cataloging concepts.
• Strong debugging, performance tuning, and problem-solving skills.
• Ability to document pipeline logic and collaborate with cross-functional teams.

Desirable Skills and Qualifications:

• Microsoft certification in Azure Data Engineering.
• Experience in governance-heavy or regulated environments (e.g., finance, healthcare, hospitality).
• Exposure to Power BI or other BI tools as a data source consumer.
• Familiarity with DevOps/CI-CD for data pipelines in Azure.
• Experience integrating both cloud and on-premises data sources into Azure.