Lead Data Engineer

About BRIT: Brit India GCC is the heart of Brit Insurance’s innovation engine. Their team of highly skilled professionals specializes in Data Engineering, Platform Engineering, Full-Stack Web Application Development, Data Management, and Testing. Born in 1995 in London (UK), Brit Ltd. is a leading international general insurance and reinsurance group. Boasting one of the largest and most diverse portfolios in Syndicate 2987, we hold an influential and respected presence at Lloyd's of London. Renowned for our innovative approach to commercial insurance and reinsurance, we at Brit are committed to securing an uncertain future with a hallmark of certainty. A legacy of excellence, driving innovation and personalized service to create exceptional customer experiences. Brit has a leadership position at Lloyd’s, the most well-known insurance market in the world. At Brit, we believe that the uncertainty of the future should never stand in the way of progress. That’s why we exist. To help people and businesses face the future and thrive. We are change-makers enabled by a global workforce who collaborate to deliver a risk service. Our goal is to attract and retain the best talent in the market and empower those individuals to innovate, and champion the strategic objectives within the business. Our technology and data teams have led some of the most noteworthy developments in the London market, by delivering digital, data and AI driven capabilities: Data is a key capability to drive Brit’s ambition for digital, innovation, growth and efficiencies. Brit leads the market with respect to our data modernisation journey, delivering on our vision to build a central, trusted, cost efficient, flexible, and scalable cloud native data platform that enables us to harness maximum value from internal and external data for Brit. Brit is active in advancing digital trading approaches, most notably with respect to Broker APIs. At Brit, we are designing a multi class of business headless API which connects Brits pricing and document generation to multiple global Broker platforms, Market Hubs, and enables bulk submission ingestion. Brit has launched its Underwriting Transformation journey advancing our capabilities deploying digital foundations for pricing and underwriter workbench. Brit’s new modern Underwriting Platform is at the heart of this transformation agenda, with a focus on engineering, upskilling in python, exploring ingestion and analytics, we are actively delivering our digital future. Our market leading proprietary machine learning algorithm designed to accelerate the identification of post catastrophe property damage, based on the use of ultra-high-resolution imagery, allows Brit to accelerate service to its customers when they need it most, noted as the first enabler for virtual claims adjusting in the London market. We value continuous learning & exploration, encouraging our people to constantly evolve and take advantage of the best-in-class technology & tooling. If you have the drive to work in a flexible, fun, passionate Technology and Data team we would love to hear from you Purpose of the Job: The Lead Data Engineer leads a team to deliver an effective Azure Data Lake platform, utilising Delta Lake, Azure Synapse Analytics, Databricks, and PowerBI. The role plays a pivotal part in the architectural design and implementation of these solutions, ensuring they meet a broad range of financial and underwriting reporting and analytics needs. Leveraging prior AI expertise, this role embeds machine learning and automation into data engineering workflows, accelerating innovation and enhancing efficiency across the platform. The position combines technical leadership, team management, and strategic stakeholder engagement to support Brit’s next-generation data platform. The Lead Data Engineer will: Lead the data engineering team across BAU and strategic projects, defining and delivering tasks. Own the design, development, and delivery of core data engineering services, from ingestion to reporting. Oversee CI/CD, code reviews, data integrity, and security for the Brit Data Platform. Develop ETL/ELT processes, data lake architecture, data warehouses, marts, reports, and dashboards. Build relational and dimensional models to support reporting needs. Ensure solutions follow standards and best practices. Assess impacts of system changes. Apply AI expertise to embed machine learning and automation in workflows, driving innovation and efficiency. Work proactively in an agile environment to deliver high-quality solutions on time. Mentor and support junior team members. Principal Accountabilities: Serve as a senior custodian of the Brit Data Platform, prioritising performance, security, and data integrity in all decisions, and ensuring robust data protection. Collaborate with Data Architects and contribute to the design, build, and management of the platform, taking ownership of data engineering activities and ensuring new requirements are met with optimal architecture and practices. Lead both senior and junior data engineers. Own incident management and remediation planning, ensuring production issues are resolved in line with Brit’s RTO expectations. Enforce SDLC best practices, coding standards, and design principles across the data lifecycle and BI/MI solutions. Assess business requirements and develop effective data solutions. Support the development of Brit-specific data management policies, processes, and standards. Stay current with technology trends, platforms, and data engineering techniques, providing input to Brit’s strategic roadmap. Proactively identify opportunities for automation and continuous improvement, leveraging AI and machine learning to drive innovation and efficiency. Develop Proof of Concepts to demonstrate the viability of technical solutions. Regulatory Conduct Rules: Act with integrity. Act with due skill, care and diligence. Be open and co-operative with Lloyd’s, the FCA, the PRA, and other regulators. Pay due regard to the interests of customers and treat them fairly. Observe proper standards of market conduct. Act to deliver good outcomes for retail customers. Education, Qualifications, Knowledge, Skills and Experience: Required Skills & Knowledge: Solid hands-on experience with Azure Data Lake (including Blob Storage and Delta), Azure Synapse Analytics, and Databricks Strong background in data engineering activities such as release management, environment controls, and CI/CD pipeline orchestration Proficiency with Python and Spark Demonstrated ability in Microsoft BI software development, including database and reporting development across the full lifecycle Strong skills in data modelling, problem solving, information analysis, attention to detail, and adaptability Ability to work directly with business functions and translate requirements into effective data and analytics solutions Experience designing and developing complex BI/MI solutions in line with architectural principles and toolsets Capable of delivering complex requirements to tight timescales in close collaboration with business stakeholders Strong interest in the latest data engineering technologies, best practices, and industry trends Experience with container apps (e.g., Azure Container Apps, Docker, or Kubernetes) for deploying and managing scalable data and AI solutions Proven expertise in applying AI and machine learning techniques to data engineering challenges, with a track record of delivering automation and innovation through AI-driven solutions Highly Desirable Skills and Knowledge: Knowledge of London market insurance Extensive experience in designing and building data warehouses and data lakes Experience working with financial systems and business processes Experience with Infrastructure-as-Code (e.g., Terraform) Agile development techniques (SCRUM, Kanban, etc.) Required Toolsets: Azure Data Lake Storage, Azure Synapse Synapse, Azure Databricks, PySpark, Azure Container Apps, Docker, Python Kimball data modelling Experience with AI and machine learning toolsets and frameworks relevant to data engineering, such as Azure Machine Learning, Databricks ML, or similar platforms Required Qualifications: Degree Level or above (Science, Maths, Computing, Engineering or related subject preferred)