Azure Data Engineer

Data Engineer (Property and Casualty Insurance - Commercial Insurance) - SQL, Azure, ADF with 7+ years experienceA Data Engineer with 7+ years of experience in Commercial Insurance and expertise in SQL, Azure, and Azure Data Factory (ADF) is responsible for designing, building, and maintaining the data infrastructure that supports complex analytical and reporting needs. They must ensure data from diverse sources is reliably ingested, transformed, and made accessible for business insights within a heavily regulated industry.Core roles and responsibilitiesData pipeline development: Design, build, and maintain robust and scalable ETL (Extract, Transform, Load) and ELT processes using Azure Data Factory (ADF). This includes orchestrating data movement and transformation from various sources into centralized data storage.Commercial insurance data expertise: Apply a deep understanding of commercial insurance data, including policies, claims, premiums, and reinsurance, to design data models that meet business requirements.SQL development and optimization: Write and optimize complex SQL queries, stored procedures, and scripts for data transformation, aggregation, and analysis within Azure SQL databases.Azure cloud architecture: Implement and manage scalable data solutions on the Azure platform, leveraging services such as Azure Data Lake Storage (ADLS) for efficient storage of large datasets.Data modeling and warehousing: Create and manage logical and physical data models for data warehouses and data lakes, ensuring the design supports efficient reporting and analytics.Collaboration and communication: Work closely with data analysts, data scientists, and business stakeholders to understand data requirements and assist with technical issues.Data quality and governance: Implement data validation, quality checks, and monitoring to ensure data integrity, accuracy, and adherence to security and compliance standards (e.g., GDPR).Performance tuning and monitoring: Identify and resolve performance bottlenecks in data pipelines and optimize database and query performance. Use monitoring tools to ensure high availability and minimal downtime.Essential skills and qualifications:Technical skillsAzure: Strong, hands-on experience with Azure Data Factory (ADF) and other Azure services like Data Lake Storage, Azure SQL Database, and Azure Synapse Analytics.SQL: Expert-level proficiency in Structured Query Language (SQL) and extensive experience with relational databases.Programming: Strong programming skills in languages such as Python or PySpark for data processing tasks, especially when using Azure Databricks.ETL/ELT: In-depth knowledge of ETL/ELT methodologies and tools, with practical experience designing and managing complex data workflows.Data Warehousing: Solid understanding of data warehousing concepts, including star/snowflake schemas, and experience with data warehousing platforms.CI/CD and DevOps: Familiarity with DevOps practices and CI/CD pipelines, using tools like Azure DevOps, for automated deployments.Industry and domain knowledge:Commercial Insurance: Proven experience working with data specific to the commercial insurance domain, including familiarity with industry data standards and reporting needs.Soft skills:Problem-solving: Strong analytical and critical thinking skills to troubleshoot data issues and find innovative solutions.Communication: Excellent verbal and written communication skills to collaborate effectively with both technical and non-technical teams.Leadership: Experience leading projects or mentoring junior engineers is expected for a candidate with over five years of experience.