Data Engineer

Position: Data Engineer (Sr. Analyst) JOB SUMMARY The Data Engineer will have responsibility to use various methods to transform raw data into useful data systems and should have strong analytical skills and the ability to combine data from different sources. The person must have familiarity with several programming languages and knowledge of learning machine methods. ESSENTIAL JOB FUNCTIONS The primary responsibilities of this position include:  Development of high-quality code (efficient and maintainable) in Python, PySpark, SQL to ingest, process and transform large datasets  Design and implement scalable data pipelines using ETL and ELT processes in Azure Data Factory, Databricks and Snowflake  Work closely with cross-functional teams to understand data requirements and deliver solutions that meet business needs  Design, develop and optimize data models in Databricks and Snowflake environments  Ensure data quality and integrity by establishing and enforcing data modelling standards and best practices  Support and optimize data tools and associated cloud environments for consumption by downstream systems, data analysts and data scientists  Ensure code, configuration and other technology artifacts are delivered within agreed time schedules and any potential delays are escalated in advance  Collaborate across developers as part of a SCRUM team ensuring collective team productivity  Implement CI/CD pipelines and DevOps practices using Azure DevOps  Ensure that 100% of code is well documented and maintained in source code repository TECHNICAL SKILLS AND EXPERIENCE  5-7 years professional experience as a Data engineer/Data Modeler  Strong experience working with Microsoft Azure Data platform services like: Azure Data Lake Gen2, Azure Data Factory, Azure Databricks  Advanced in building ETL and data pipelines on Databricks using data engineering languages like Python, PySpark.  Hands-on experience with Snowflake  Proficiency in SQL  Proven experience with all aspects of the ETL processes (Data Sourcing, Transformations, Data Quality, etc.…)  Familiarity with Azure DevOps and CI/CD pipelines  Advanced knowledge of data warehousing concepts and best practices  Experience with relational and dimensional database modelling (Relational, Kimball, Data Vault, Star Schemas) NON-TECHNICAL SKILLS AND EXPERIENCE  Creative thinking, problem solving and decision making. Proactive and self-starter.  Has high degree of initiative, dependability and ability to work with little supervision.  Responds to and handles unfamiliar situations with little supervisory guidance.  Ability to solve problems that are complex, varied and only mildly related to those seen before. Simply determining what the problem is requires major individual effort and/or consultation with others within the department.  A high degree of analytical ability and inductive thinking may be required to solve highly intricate, technically complex problems. Must be able to develop new and nonstandard approaches.  Ability to complete work that requires developing imaginative and complex methods, procedures, products or systems.  Strong understanding of the Business Intelligence lifecycle, from requirements gathering to development and acceptance.  Has excellent written and verbal communication skills, interpersonal and collaborative skills, and the ability to communicate technical concepts to both technical and nontechnical audiences.  Requires regular external contacts to carry out organization programs and to explain specialized matters. Position requires contacts with customers or the public involving persons of some importance on matters requiring explanation and discussion and where the effort to influence becomes important.  Strong communication and interpersonal skills to work within a team environment.  Excellent organizational and troubleshooting skills with attention to detail.  Strong presentation and documentation skills.  Must have high proficiency in spoken and written English. EDUCATION  Bachelor’s degree required in Mathematics, Statistics, Computer Sciences, or related field.