Senior Azure Data Engineer Exp

Role SnapshotTitle: Senior Azure Data EngineerExperience: 6–8+ years in Data Engineering (minimum 4+ years on Azure, and ( Or ) 6 months to 1+ year with Microsoft Fabric)Tech Focus: Microsoft Fabric, Azure Data Factory (ADF), Databricks (Python, PySpark, Spark SQL), Delta Lake, Power BI (DAX), Azure Storage, Lakehouse, WarehouseEngagement: Client-facing, hands-on, design-to-deliveryLocation: Any Accion Labs offices in India - Bangalore, Pune (Preferred), Mumbai, Hyderabad, Indore, Noida - THIS IS A HYBRID Work Model, NOT Remote.Notice period: Preferred Immediate joiners or candidates who can join within 20 days are neededCore ResponsibilitiesEnd-to-End EngineeringDesign, implement, and deliver batch & streaming data pipelines into Fabric Lakehouse/Warehouse using ADF and Databricks with Delta Lake.Data Architecture UnderstandingStrong grasp of Bronze–Silver–Gold layering, incremental ingestion, watermarking, and best practices for scalable pipelines.Medallion ArchitectureApply Bronze/Silver/Gold patterns, enforce schema evolution, handle late/dirty data, and implement SCD (Type 1/2) and late-arriving dimensions.Fabric Platform & SecurityBuild solutions on Microsoft Fabric (OneLake, Lakehouse, Warehouse, Pipelines, Dataflows Gen2, Notebooks).Implement security layers: workspace & item permissions, RLS/OLS in Warehouse/Lakehouse SQL endpoints, credentialed connections/shortcuts to external storage, environments & capacities alignment.ADF Orchestration & ReusabilityCreate parameterized, template-driven pipelines with reusable activities (ForEach, Lookup, Mapping Data Flows).Ensure robust dependency management with retry/alert patterns.Databricks Engineering ExcellenceAuthor complex & nested notebooks (via %run/dbutils.notebook.run) in Python, PySpark, and Spark SQL for ETL/ELT.Debug & troubleshoot jobs and clusters; resolve skew, shuffle spills, checkpoint failures, schema drift, streaming backlogs.Apply performance optimizations: partitioning & clustering, Z-ORDER, OPTIMIZE/VACUUM, file size tuning, AQE, broadcast joins, caching, checkpoint & trigger strategies for Structured Streaming.Data Quality, Observability & ReliabilityImplement data quality checks (validations, expectations), idempotency, exactly-once/at-least-once semantics, and dead-letter flows.Set up monitoring & logging (Azure Monitor/Log Analytics, Databricks system tables, Fabric monitoring), with alerting & dashboards.SQLStrong understanding of MS SQL concepts; hands-on experience in writing functions and stored procedures, along with DDL/DML operations.Design, Documentation & GovernanceContribute to data models (star/snowflake), semantic layers, dimensional design, and documentation (solution design docs, runbooks).CI/CD & ADO Versioning ManagementImplement branching strategy (Git/ADO), perform PR reviews, manage environment promotion (Dev/Test/Prod), and support Fabric CI/CD process.Leadership & Client EngagementMentor junior engineers; enforce reusable & scalable patterns.Run client demos and brainstorming discussions.Be self-driven and innovative in solution delivery.Must-Have Skills (Strong, Hands-On)Microsoft Fabric (2024+)OneLake, Lakehouse, Warehouse, Pipelines, Dataflows Gen2, Notebooks, capacities, workspace & item security, RLS/OLS.Azure Data Factory (ADF)Reusable, parameterized pipelines; high-level orchestration; robust scheduling, logging, retries, and alerts.Databricks (5+ years on Azure)Python, PySpark, Spark SQL: complex transformations, joins, window functions, UDFs/UDAs.Complex & nested notebooks; modular code with %run / dbutils.notebook.run.Structured Streaming: watermarks, triggers, checkpointing, foreachBatch, schema evolution.Delta Lake: Z-ORDER, OPTIMIZE/VACUUM, MERGE for SCD, Auto Optimize, compaction, time travel.Performance tuning: partitioning, file sizing, broadcast hints, caching, Photon (where available), cluster sizing/pools.Medallion ArchitectureBronze/Silver/Gold patterns, SCD (Type 1/2), handling late-arriving dimensions.Azure StorageADLS Gen2 (hierarchical namespace), tiering (Hot/Cool/Archive), lifecycle & cost optimization, shortcuts into OneLake.Data WarehousingDimensional modeling, fact/aggregate design, query performance tuning in Fabric Warehouse & Lakehouse SQL endpoint.SQLExcellent SQL development; advanced joins, windowing, CTEs, performance tuning/indexing where applicable.Power BI (DAX)Awareness of Power BI and DAX; RLS alignment with Warehouse/Lakehouse.Security & ComplianceRBAC, item-level permissions, credentials for data sources, RLS/OLS, secret management (Key Vault), PII handling.ETL/ELT MethodologiesRobust, testable pipelines; idempotency; error handling; data quality gates.Ways of WorkingAgile delivery, client-facing communication, crisp demos, documentation, and best-practice advocacy.If interested or know anyone, Kindly write to me at shruti.saboo@accionlabs.com along with your latest CV.Thank You,Shruti Saboo