Azure Data Engineer

Module Lead Experience Range 6 Years Primary Must have skills - To be Screened by TA Team 3 years of experience in Azure Databricks with PySpark 2 years of experience in Databricks workflow Unity catalog 2 years of experience in ADF Azure Data Factory 2 years of experience in ADLS Gen 22 years of experience in Azure SQL 3 years of experience in Azure Cloud platform 2 years of experience in Python programming package builds of Role RNR - To be Evaluated by Technical Panel Define it to give more clarity Key technical skills Hands-on experience in designing and building scalable data pipelines using Databricks with PySpark supporting batch and near-real-time ingestion transformation and processing Ability to optimize Spark jobs and manage large-scale data processing using RDD DataFrame APIs Demonstrated expertise in partitioning strategies file format optimization Parquet Delta and Spark SQL tuning Skilled in governing and manage data access for Azure Data lakehouse with Unity Catalog Experience in configuring data permissions object lineage and access policies with Unity Catalog Understanding of integrating Unity Catalog with Azure AD external metastores and audit trails Experience in building efficient orchestration solutions using Azure data factory Databricks Workflows Ability to design modular reusable workflows using tasks triggers and dependencies Skilled in using dynamic expressions parameterized pipelines custom activities and triggers Familiarity with integration runtime configurations pipeline performance tuning and error handling strategies Good experience in implementing secure hierarchical namespace-based data lake storage for structured semi-structured data aligned to bronze-silver-gold layers with ADLS Gen2 Hands-on experience with lifecycle policies access control RBAC ACLs and folder-level security Understanding of best practices in file partitioning retention management and storage performance optimization Capable of developing T-SQL queries stored procedures and managing metadata layers on Azure SQL Comprehensive experience working across the Azure Cloud ecosystem including networking security monitoring and cost management relevant to data engineering workloads Experience in writing modular testable Python code used in data transformations utility functions and packaging reusable components Familiarity with Python environments dependency management pip Poetry Conda and packaging libraries Ability to write unit tests using PyTest unittest and integrate with CI CD pipelines Able to prepare Design documents for development adopting code analyzers and unit testing frameworks Write clear and concise documentation leveraging industry standard practices guidelines Soft skills other skills - To be Evaluated by Hiring Manager To define how this will be evaluated Communication Skills Communicate effectively with internal and customer stakeholders Communication approach verbal emails and instant messagesInterpersonal Skills Strong interpersonal skills to build and maintain productive relationships with team membersProvide constructive feedback during code reviews and be open to receiving feedback on your own code Problem-Solving and Analytical Thinking Capability to troubleshoot and resolve issues efficiently Analytical mindset Task Work Updates Prior experience in working on Agile Scrum projects with exposure to tools like Jira Azure DevOps Provides regular updates proactive and due diligent to carry out responsibilities Expected Outcome We are seeking a highly skilled Data Engineering specialist with above mentioned Primary Skills to join our dynamic team who are at the forefront of enabling enterprises in Healthcare sectors The ideal candidate should be passionate about working on Data Engineering on Azure cloud with strong focus on DevOps practices in building product for our customers Effectively Communicate and Collaborate with internal teams and customer to build code leveraging or building low level design documents aligning to standard coding principles and guidelines