Senior Azure Data Engineer

Role Snapshot

• Title:
  Senior Microsoft Fabric Data Engineer
• Experience:
  7–8+ years in Data Engineering (
  minimum 4+ years on Azure
  , and ( Or )
  6 months
  to
  1+ year with Microsoft Fabric
  )
• Tech Focus:
  Microsoft Fabric, Azure Data Factory (ADF), Databricks (Python, PySpark, Spark SQL), Delta Lake, Power BI (DAX), Azure Storage, Lakehouse, Warehouse
• Engagement:
  Client-facing, hands-on, design-to-delivery

Location:
Any Accion Labs offices in India - Bangalore, Pune (Preferred), Mumbai, Hyderabad, Indore, Noida -
THIS IS A HYBRID Work Model
, NOT Remote.

Notice period:
Preferred Immediate joiners or candidates who can join within 20 days are needed

Core Responsibilities

1. End-to-End Engineering

2. Design, implement, and deliver
   batch & streaming
   data pipelines into
   Fabric Lakehouse/Warehouse
   using
   ADF
   and
   Databricks with Delta Lake
   .

3. Data Architecture Understanding

4. Strong grasp of
   Bronze–Silver–Gold layering
   ,
   incremental ingestion
   ,
   watermarking
   , and
   best practices
   for scalable pipelines.

5. Medallion Architecture

6. Apply
   Bronze/Silver/Gold patterns
   , enforce
   schema evolution
   , handle
   late/dirty data
   , and implement
   SCD (Type 1/2)
   and
   late-arriving dimensions
   .

7. Fabric Platform & Security

8. Build solutions on
   Microsoft Fabric (OneLake, Lakehouse, Warehouse, Pipelines, Dataflows Gen2, Notebooks)
   .

9. Implement
   security layers
   : workspace & item permissions,
   RLS/OLS
   in Warehouse/Lakehouse SQL endpoints, credentialed connections/shortcuts to external storage, environments & capacities alignment.

10. ADF Orchestration & Reusability

11. Create
    parameterized, template-driven pipelines
    with reusable activities (ForEach, Lookup, Mapping Data Flows).

12. Ensure
    robust dependency management
    with retry/alert patterns.

13. Databricks Engineering Excellence

14. Author
    complex & nested notebooks
    (via ) in Python, PySpark, and Spark SQL for ETL/ELT.

15. Debug & troubleshoot jobs and clusters; resolve
    skew, shuffle spills, checkpoint failures, schema drift, streaming backlogs
    .

16. Apply
    performance optimizations
    : partitioning & clustering, Z-ORDER, OPTIMIZE/VACUUM, file size tuning, AQE, broadcast joins, caching, checkpoint & trigger strategies for Structured Streaming.

17. Data Quality, Observability & Reliability

18. Implement
    data quality checks
    (validations, expectations), idempotency, exactly-once/at-least-once semantics, and
    dead-letter flows
    .

19. Set up
    monitoring & logging
    (Azure Monitor/Log Analytics, Databricks system tables, Fabric monitoring), with alerting & dashboards.

20. SQL

21. Strong understanding of
    MS SQL concepts
    ; hands-on experience in writing
    functions and stored procedures
    , along with
    DDL/DML operations
    .

22. Design, Documentation & Governance

23. Contribute to
    data models
    (star/snowflake), semantic layers, dimensional design, and
    documentation
    (solution design docs, runbooks).

24. CI/CD & ADO Versioning Management

25. Implement
    branching strategy (Git/ADO)
    , perform
    PR reviews
    , manage
    environment promotion
    (Dev/Test/Prod), and support
    Fabric CI/CD process
    .

26. Leadership & Client Engagement

27. Mentor junior engineers; enforce
    reusable & scalable patterns
    .

28. Run
    client demos and brainstorming discussions
    .
29. Be
    self-driven
    and
    innovative
    in solution delivery.

Must-Have Skills (Strong, Hands-On)

• Microsoft Fabric
• OneLake, Lakehouse, Warehouse, Pipelines, Dataflows Gen2, Notebooks, capacities, workspace & item security, RLS/OLS.
• Azure Data Factory (ADF)
• Reusable, parameterized pipelines; high-level orchestration; robust scheduling, logging, retries, and alerts.
• Databricks (5+ years on Azure)
• Python, PySpark, Spark SQL: complex transformations, joins, window functions, UDFs/UDAs.
• Complex & nested notebooks; modular code with %run /
• Structured Streaming: watermarks, triggers, checkpointing, foreachBatch, schema evolution.
• Delta Lake: Z-ORDER, OPTIMIZE/VACUUM, MERGE for SCD, Auto Optimize, compaction, time travel.
• Performance tuning: partitioning, file sizing, broadcast hints, caching, Photon (where available), cluster sizing/pools.
• Medallion Architecture
• Bronze/Silver/Gold patterns, SCD (Type 1/2), handling late-arriving dimensions.
• Azure Storage
• ADLS Gen2 (hierarchical namespace), tiering (Hot/Cool/Archive), lifecycle & cost optimization, shortcuts into OneLake.
• Data Warehousing
• Dimensional modeling, fact/aggregate design, query performance tuning in Fabric Warehouse & Lakehouse SQL endpoint.
• SQL
• Excellent SQL development; advanced joins, windowing, CTEs, performance tuning/indexing where applicable.
• Power BI (DAX)
• Awareness of Power BI and DAX; RLS alignment with Warehouse/Lakehouse.
• Security & Compliance
• RBAC, item-level permissions, credentials for data sources, RLS/OLS, secret management (Key Vault), PII handling.
• ETL/ELT Methodologies
• Robust, testable pipelines; idempotency; error handling; data quality gates.
• Ways of Working
• Agile delivery, client-facing communication, crisp demos, documentation, and best-practice advocacy.

If interested or know anyone, Kindly write to me at along with your latest CV.

Thank You,

Shruti Saboo