Immediate Start Lead Data Engineer

Job Description

- Design and Develop Data Pipelines:Hands on with development, and optimisation of scalable and reusable - data pipelines inAzure Microsoft Fabric Synapse Data Engineering, leveraging bothbatchandreal-time processing techniques. Ensure smooth integration withAzure Data Factoryfor orchestration and workflow management.
- Cloud Data Architecture:Collaborate with theData Architecture teamto design and implement robust data architectures in theAzure environment, ensuring they align with business needs while optimising performance, scalability, and cost-efficiency.
- Pipeline Optimisation :Continuously monitor and optimise the performance, cost, and reliability of data pipelines, ensuring efficient processing, storage, and management of large datasets.
- Cross-functional Collaboration:Work closely withdata engineering teams,analysts, andbusiness stakeholdersto understand data requirements, developing solutions that enable self-service analytics and support the decision-making process.
- Documentation Knowledge Sharing:Contribute to internal documentation, fostering a culture of knowledge-sharing. Providementorshipand guidance to junior engineers, helping to elevate team skills and improve overall team performance.
- Microsoft Fabric Experience:Apply your knowledge ofAzure Tech Stackon Data Engineering (or your willingness to learn)Fabric-based developmentto manage end-to-end data orchestration, governance, and security across cloud and on-premises systems, ensuring seamless data movement and integration across hybrid environments.
- Data Modelling Expertise:Leverage your deep expertise inAzureto design and implementdata models, create processing pipelines, and integrate with otherAzure serviceslikeData LakeandSynapseto support data storage and analytics needs.

Required Skills and Qualifications:

- Experience with Azure Ecosystem (Preferably Synapse):5+ years of hands-on experience withAzure Ecosystem, includingSynapse,Spark,OneLake, and other Fabric tools. Expertise in optimisingFabric notebooksand efficiently managing large-scale data workloads.
- Proficiency in Azure Data Factory:Strong experience with designing and orchestrating complex data pipelines usingAzure Data Factory, with an emphasis on seamless data flow integration across various Azure services.
- Familiarity with Microsoft Fabric:A working knowledge or eagerness to learnAzure Data Fabric, focusing on cross-platform data orchestration, governance, and security.
- Advanced Data Engineering Skills:Extensive experience in data engineering, including the design and implementation ofETL processesand working with large datasets. Proven expertise indata quality, monitoring, and testing practices.
- Cloud Architecture Design Expertise:Experience designing and implementingdata architecturesin theAzure ecosystem, including tools such asData Lake,Synapse, andAzure Storage.
- SQL and Data Modelling Expertise:Strong skills inSQLanddata modelling, with the ability to design optimised data structures, tables, and views. Knowledge of both transactional and analytical data modelling.
- Collaboration and Communication Skills:Strong ability to work cross-functionally with teams from various domains. Ability to communicate complex technical concepts to both technical and non-technical stakeholders.
- Cost Optimisation:Proven experience optimisingdata engineering processesandAzure resourcesfor both performance and cost, particularly in large-scale cloud environments.

Preferred Skills:

- Data Lakehouse Experience:Familiarity withData Lakehousearchitectures, particularly with tools likeDelta Lake,OneLake, etc.
- Azure Ecosystem Familiarity:Knowledge ofAzure s full ecosystemfor end-to-end data integration and ETL processes.
- Proficiency in PySpark and Python:Expertise inPySparkfor data processing tasks, with a solid foundation inPython.
- Fabric Integration:Familiarity withFabricand how it integrates with other services within theAzure ecosystem.
- Databricks Experience:Experience withDatabricksis a plus.