Senior Data Engineering Lead

Description :

Position : Senior Data Engineering Team Leader (Azure Focus)

Work Level : Senior Leadership

Industry Type : IT Services & Consulting

Job Summary :

We are seeking a Disciplined and innovation-focused Senior Data Engineering Team Leader to lead a team in designing and implementing advanced data solutions on the Microsoft Azure cloud platform. This leadership role requires a minimum of three years of hands-on Azure Data Engineering experience, coupled with deep proficiency in Databricks, PySpark, and complex SQL. The leader will be responsible for driving technical innovation, mentoring a team of 5 members, and delivering high-performance, scalable data pipelines for our IT Services and Consulting clients.

Job Description :

Technical Leadership and Innovation :

- Provide Senior Leadership and technical oversight for a team of 5 or more Data Engineers, fostering a culture of technical excellence, accountability, and continuous improvement.

- Drive Innovation within the data engineering function by evaluating and integrating cutting-edge Azure services, advanced Databricks features, and modern data processing methodologies to improve pipeline efficiency and capability.

- Lead the technical design sessions and architectural reviews, ensuring all data solutions are scalable, resilient, and adhere to enterprise-level security and governance standards on Azure.

- Instill Disciplined engineering practices, including robust code review processes, documentation standards, and continuous adherence to CI/CD pipelines for all data assets.

Azure Data Platform Engineering :

- Possess 3 years of mandatory, hands-on experience leading and delivering complex data engineering development projects within the Microsoft Azure cloud ecosystem.

- Design and implement high-volume, low-latency ETL/ELT data pipelines using core Azure services, including Azure Data Factory (ADF), Azure Data Lake Storage (ADLS Gen2), and Azure Key Vault.

- Demonstrate 2 years of deep, proven proficiency in developing, optimizing, and deploying complex data processing jobs utilizing Databricks notebooks and clusters.

- Master the use of PySpark (Python with Spark) to perform advanced data transformations, aggregations, and feature engineering at scale within the Databricks environment.

Data Modeling and Team Management :

- Develop and optimize complex SQL queries, stored procedures, and data models for consumption by reporting tools and analytical applications, managing data integration with services like Azure SQL Database or Synapse Analytics.

- Provide expert-level guidance on performance tuning for Spark jobs, Databricks clusters, and SQL execution plans to ensure optimal query response times and resource utilization.

- Demonstrate 2 years of mandatory experience effectively leading and managing the day-to-day work, career development, and technical delivery of a data engineering team of 5 or more members.

- Act as the primary technical point of contact, leveraging strong communication skills to articulate complex technical issues and architectural decisions to clients, product owners, and senior stakeholders.

Required Skills & Qualifications :

- Experience : Mandatory 5 years total experience in data engineering, including 3 years of dedicated experience in Azure development projects.

- Leadership : Mandatory 2 years experience successfully leading and mentoring a data engineering team of 5 or more members.

- Cloud & Platform : Deep technical expertise in the Microsoft Azure data ecosystem.

- Big Data Tools : Mandatory proficiency (2 years) with Databricks, PySpark for scalable data processing, and expert-level SQL for data manipulation and modeling.

- Core Skills : Exceptional technical communication, stakeholder management, and problem-solving abilities within a consulting environment.

- Education : Mandatory Graduate degree.

Preferred Skills :

- Advanced Azure Services : Hands-on experience with Azure Synapse Analytics (SQL Pool, Spark Pool) for integrated data warehousing and big data workloads.

- Data Visualization : Experience integrating data pipelines with leading business intelligence tools such as Tableau and Power BI for final data consumption.

- Programming Languages : Proficiency in Scala or Java in addition to Python for high-performance data processing tasks.

- DevOps for Data : Experience with specific CI/CD tools (e.g., Azure DevOps, GitHub Actions) for automating Databricks and Azure Data Factory deployments.

- Data Governance : Familiarity with Azure Purview or other data governance tools for metadata management and data lineage tracking.