Senior Data Engineer

Job Description:

We are seeking a highly skilled and experienced Senior Data Engineer with a strong background in Microsoft Azure data platforms. The ideal candidate will have a proven track record of designing, implementing, and managing large-scale data solutions using Azure technologies. This role requires a deep understanding of data engineering best practices, cloud architecture, and the ability to work collaboratively with cross-functional teams.

The Sr. Data Engineer will design, implement, and maintain data integration projects focused primarily on the use of Azure SQL, Azure Databricks, Data Factory, Logic Apps, APIs, and Blob storage. This individual will work with business and technical staff to understand integration requirements and develop agile user stories to resolve them. They will also resolve customer complaints and respond to suggestions for improvements and enhancements. They will act as team leader for all data integration projects; coordinating the work of co-located and distributed team members, including vendor resources.

Essential Duties and Responsibilities:

• Lead the design, implementation, testing, and maintenance of complex system integration processes focused primarily on the use of Azure SQL, Azure Databricks, Data Factory, Logic Apps, APIs, and Blob storage.

• Coordinate the work of other systems integration technical resources, both onshore and offshore

• Become an expert on Microsoft data integration and reporting products

• Participate in requirement analysis and definition

• Create technical specifications and system integration test plans

• Document data flows using data dictionaries and other data governance tools

• Collaborate with different technical teams on best practices

• Work as part of an Agile development team to solve problems and develop projects in a fast-paced environment

• Follow pre-established guidelines as well as create new guidelines to perform the functions of the job

• Demonstrate a high degree of creativity and problem-solving skills

• Mature the software development life cycle within your area of responsibility

• Support applications using software development methodologies including structured programming, documentation, design, and code review and ensure consistent use of Azure DevOps to manage the full lifecycle

• Collaborate with database resources to ensure robust and complete data access and manipulation

• Collaborate with IT Operations to ensure hardware and software are aligned to deliver business requirements

• Mentor junior and other new developers

• Stay up to date with the latest Azure data services and best practices to continuously improve our data infrastructure.

Education, Experience and Knowledge:

• Bachelor’s / master’s degree in computer science, engineering, or a related field, or in-lieu of degree, equivalent education, training, and work-related experience

• 10+ years of experience in IT with an emphasis on ETL, Reporting, and systems integration.

• Strong expertise in Azure Data Factory, Azure Databricks, Azure Synapse Analytics, Azure SQL Database, Azure Cosmos DB, Azure Data Lake, and Azure Databricks.

• Proficiency in programming languages such as Python, SQL, Spark and Scala.

• Experience with data modeling, ETL processes, and data warehousing / data lakes.

• Experience designing and building relational databases

• Expert knowledge of ETL concepts and techniques.

• Proficient in SQL, stored procedures, triggers, indexes.

• Experience integrating with SOAP and Restful Web Services

• Experience applying enterprise design and architectural patterns

• Experience in end-to-end Software Development Life Cycle (SDLC) projects

• Experience with code review and evaluation

• Good at understanding requirements and estimation

• Team player, have ability and willingness to mentor junior developers

• Have passion for learning new technologies and enhancing existing skills

• Excellent problem-solving skills.

• Strong communication skills, with the ability to convey complex technical concepts to non-technical stakeholders.

•Self-starter with ability to work independently.

• Excellent Team player

• Proactive issue resolution with a positive attitude

• Ability to adapt and respond in a rapidly evolving business environment

• Understands solution at the project level

• Proficient in Microsoft Office (Word, Excel, Outlook, PowerPoint)

Preferred

• Possess experience in the insurance industry and/ or basic knowledge of insurance or financial services.

• Experience with Azure DevOps, CI/CD Pipelines and GIT source control

• Experience with Azure Data Factory, Databricks, Azure SQL, API Management, Power BI

• Experience with Azure virtual networks and private endpoints

• Experience working with Agile/Scrum methodologies

• Experience working with Denodo or any other Data Virtualization Tools

• Exposure to Data science, AI/ML