Senior Data Engineer

JOB SUMMARY :The primary responsibility of the Senior Data Engineer is to design, develop, implement, and support core systems that enable the acquisition, curation, modeling, and consumption of data for data integration, analytics, and data science. This ensures seamless flow, availability, and reliability of data across systems.This role requires a passionate, committed, and organized individual who is a fast learner, adaptable, and open to new approaches in technology JOB FUNCTIONS :The primary responsibilities of this position are : - Development of high-quality code for the core data stack including data integration, data warehousing, data analytics and reporting. - Design and implement data models and architectures to support analytics and reporting needs.- Develop and optimize robust Extract, Load, Transform (ELT) processes for the efficient acquisition, transformation, and integration of data from various sources.- Build data pipelines using both batch and streaming paradigms- Implement data quality and governance processes to ensure accuracy, consistency, and security.- Work closely with teams and stakeholders to understand requirements and deliver solutions.- Ensure code, configuration and other technology artifacts are delivered in a timely manner.- Monitor and troubleshoot data workflows to ensure optimal performance and reliability.- Maintain comprehensive documentation of data processes, architectures, and SKILLS AND EXPERIENCE :Most important :- 7+ years professional experience as a data engineer, with at least 4 utilizing cloud technologies.- Proven experience building ETL or ETL data pipelines with Databricks either in Azure or AWS using PySpark language.- Strong experience with the Microsoft Azure Data Stack (Databricks, Data Lake Gen2, ADF etc.)- Strong SQL skills and proficiency in Python adhering to standards such as PEP- Proven experience with unit testing and applying appropriate testing methodologies using libraries such as Pytest, Great Expectations, or similar.- Demonstrable experience with CICD including release and test automation tools and processes such as Azure Devops, Terraform, Powershell and Bash scripting or similar.- Strong understanding of data modeling, data warehousing, and OLAP concepts.- Excellent technical documentation skills.Desirable, but not required : - Experience with data visualization tools such as Tableau, Power BI, or Looker.- Experience in event driven architectures- Familiarity with big data technologies such as Hadoop, Spark, or Kafka.- Experience with Snowflake or other analytics platforms- Knowledge of full stack technology concepts from hardware, software platforms to UI layers- Experience with Object oriented design and modelling- Experience with visual modelling tools including SKILLS :Most important :- High level of attention to detail, ensuring accuracy and quality in all work outputs.- Strong analytical and problem-solving skills with the ability to think critically and troubleshooting complex issues.- Excellent verbal and written communication skills, capable of conveying technical concepts to non-technical stakeholders.- Ability to adapt to rapidly changing environments and learn new technologies quickly.- Commitment to delivering high-quality solutions that meet customer needs and expectations.- Strong organizational skills with the ability to manage multiple priorities and projects simultaneously.Desirable, but not required : - Experience with appropriate organization complexity- Experience in Real :- Bachelors degree in computer science or equivalent work experience.- Relevant industry certifications such as Microsoft Certified : Azure Data Engineer Associate, AWS Certified Data Analytics or Google Professional Data Engineer. (ref:hirist.tech)