Azure Data Engineer

Mandatory Skills

• Bachelor's or higher degree in Computer Science or a related discipline; or equivalent (minimum 4+ years work experience). 
• At least 6+ years of consulting or client service delivery experience on Azure Microsoft data engineering.
• At least 4+ years of experience in developing data ingestion, data processing and analytical pipelines for big data, relational databases such as SQL server and data warehouse solutions such as Synapse/Azure Databricks, Microsoft Fabric
• Hands-on experience implementing data ingestion, ETL and data processing using Azure services: Fabric, onelake, ADLS, Azure Data Factory, Azure Functions, services in Microsoft Fabric etc.
• Minimum of 5+ years of hands-on experience in Azure and Big Data technologies such as Fabric, databricks, Python, SQL, ADLS/Blob, pyspark/SparkSQL.
• Minimum of 3+ years of RDBMS experience
• Experience in using Big Data File Formats and compression techniques.
• Experience working with Developer tools such as Azure DevOps, Visual Studio Team Server, Git, etc.

Primary Roles and Responsibilities 

An Azure Data Engineer is responsible for designing, building, and maintaining the data infrastructure for an organization using Azure cloud services. This includes creating data pipelines, integrating data from various sources, and implementing data security and privacy measures. The Azure Data Engineer will also be responsible for monitoring and troubleshooting data flows and optimizing data storage and processing for performance and cost efficiency. 

Preferred Skills

• Technical Leadership & Demo Delivery:
  • Provide technical leadership to the data engineering team, guiding the design and implementation of data solutions.
  • Deliver compelling and clear demonstrations of data engineering solutions to stakeholders and clients, showcasing functionality and business value.
  • Communicate fluently in English with clients, translating complex technical concepts into business-friendly language during presentations, meetings, and consultations.
• ETL Development & Deployment on Azure Cloud:
  • Design, develop, and deploy robust ETL (Extract, Transform, Load) pipelines using Azure Data Factory (ADF), Azure Synapse Analytics, Azure Notebooks, Azure Functions, and other Azure services.
  • Ensure scalable, efficient, and secure data integration workflows that meet business requirements.
• Microsoft Fabric Expertise:
  • Develop and deploy ETL solutions using Microsoft Fabric, leveraging its capabilities for end-to-end data integration and analytics.
  • Build and maintain medallion architecture (Bronze, Silver, Gold layers) in Microsoft Fabric to ensure data quality, scalability, and performance.
  • Implement metadata-driven ingestion frameworks to streamline data ingestion processes.
  • Design and develop data quality frameworks to validate, cleanse, and monitor data integrity.
  • Perform advanced data transformations, including Slowly Changing Dimensions (SCD Type 1 and Type 2), using Fabric Notebooks or Databricks.
• Generative AI & Microsoft Copilot:
  • Exposure to Microsoft Copilot and Generative AI fundamentals, with the ability to identify and implement industry-relevant use cases (e.g., automated insights generation, data summarization, or predictive analytics).
  • Leverage Gen AI tools to enhance data engineering workflows, such as automating data pipeline creation or improving data quality checks.
• Microsoft Certifications:
  • Hold relevant role-based Microsoft certifications, such as:
    • DP-600: Implementing Analytics Solutions Using Microsoft Fabric.
    • DP-203: Data Engineering on Microsoft Azure.
    • DP-900: Microsoft Azure Data Fundamentals.
    • AI-102: Designing and Implementing a Microsoft Azure AI Solution.
    • AI-900: Microsoft Azure AI Fundamentals.
  • Additional certifications in related areas (e.g., PL-300 for Power BI) are a plus.
• Power BI & Business Insights:
  • Proficient in Microsoft Power BI for designing and developing interactive reports and dashboards.
  • Generate actionable insights for business users by analyzing data trends and patterns, ensuring alignment with business objectives.
  • Collaborate with stakeholders to define KPIs and visualize data effectively.
• Azure Security & Access Management:
  • Strong knowledge of Azure Role-Based Access Control (RBAC) and Identity and Access Management (IAM).
  • Implement and manage access controls, ensuring data security and compliance with organizational and regulatory standards on Azure Cloud.
• Alignment with Microsoft Vision:
  • Stay updated with Microsoft's vision, roadmap, and latest tools/technologies in the data and AI ecosystem (e.g., Azure Synapse, Fabric, Power Platform, and AI advancements).
  • Proactively explore and adopt emerging tools to enhance team capabilities and deliver cutting-edge solutions.
• Additional Responsibilities & Skills:
  • Team Collaboration: Mentor junior engineers, fostering a culture of continuous learning and knowledge sharing within the team.
  • Project Management: Oversee data engineering projects, ensuring timely delivery within scope and budget, while coordinating with cross-functional teams.
  • Data Governance: Implement data governance practices, including data lineage, cataloging, and compliance with standards like GDPR or CCPA.
  • Performance Optimization: Optimize ETL pipelines and data workflows for performance, cost-efficiency, and scalability on Azure platforms.
  • Cross-Platform Knowledge: Familiarity with integrating Azure services with other cloud platforms (e.g., AWS, GCP) or hybrid environments is an added advantage.
• Soft Skills & Client Engagement:
  • Exceptional problem-solving skills with a proactive approach to addressing technical challenges.
  • Strong interpersonal skills to build trusted relationships with clients and stakeholders.
  • Ability to manage multiple priorities in a fast-paced environment, ensuring high-quality deliverables.