Senior Data Engineer with MS Fabric

Microsoft Fabric & Azure Expertise

• Design and implement comprehensive data solutions using Microsoft Fabric, including Data Pipelines, Lakehouses, and Real-Time Analytics.
• Build and manage scalable OLAP architectures, including data layer design, dimensional modeling, and semantic models.
• Develop and optimize data pipelines within Fabric Workspaces for efficient ETL/ELT processing.
• Lead implementation of data governance practices, including IAM policies, RBAC, RLS, data masking, and encryption.
• Manage and optimize Fabric resources, including Fabric SKUs, OneLake, and Workspace configurations.
• Set up and manage Power BI apps integrated with Fabric for reporting and dashboard distribution.
• Develop and maintain Fabric Notebooks using PySpark, Python, and SQL for advanced data engineering and analytics use cases.
• Oversee Azure services integration such as App Registrations, IAM, and other platform services to ensure secure and efficient operation.

Python Development

• Build robust, scalable Python scripts and applications with a strong focus on performance, reliability, and maintainability.
• Integrate external systems using RESTful APIs and automate workflows through Python-based orchestration.
• Utilize advanced Python libraries such as pandas, numpy, and others for data manipulation and transformation.
• Develop database integration scripts to connect with relational databases (e.g., SQL Server, MySQL, PostgreSQL, Oracle) and execute complex SQL and PL/SQL queries.
• Handle and transform complex JSON data structures for ingestion and processing.

Collaboration & Leadership

• Work closely with cross-functional stakeholdersincluding data scientists, analysts, product managers, and business leadersto understand requirements and deliver scalable solutions.
• Translate business needs into technical specifications and implement solutions that align with architectural best practices.
• Provide technical mentorship and guidance to junior team members and contribute to architectural decision-making.

Qualifications

• Experience in data engineering, analytics, or platform engineering roles.
• Proven expertise in Microsoft Fabric and Azure Data Services.
• Strong command of Python programming and experience working with large-scale data systems.
• Experience designing and implementing OLAP systems, including semantic models and multidimensional cubes.
• Solid understanding of data governance, security, and access control in enterprise environments.
• Exceptional problem-solving, communication, and collaboration skills.

Regular Tasks Required:

• Build and maintain MS Fabric data import, standardization, and governance procedures.
• Write effective Python code to tackle complex issues, but also use your business sense and analytical abilities to glean valuable insights from public databases
• Communicate clearly with team and help the organization in realizing its objectives
• Clearly express the reasoning and logic when writing code
• Fix bugs in the code and create thorough documentation
• Utilize your data analysis skills to develop and respond to important business queries using available datasets
• Effectively communicate with the team to comprehend the needs and provide the results
• Useful but non-mandatory skills:
• Hands-on experience with Generative AI tools, including LangChain, RAG, and vector databases.
• Experience with web scraping, browser automation, and workflow automation.
• Proficiency in traditional ML libraries (scikit-learn, TensorFlow, PyTorch, XGBoost).

Experience and Educational requirements:

• Bachelors in Engineering or Masters in Computer Science (or equivalent experience)
• At least 2+ years of relevant experience as a data scientist
• 2+ years of Data analysis experience and a desire to have a significant impact on the field of artificial intelligence
• 2+ years of experience working extensively with Python programming
• Experience with MS Fabric, OneLake, and the Microsoft data analytics stack
• Experience with PySpark and Spark SQL
• Extensive experience working with Data Science/Analysis
• Familiarity with SQL, Rest API, Tableau and related technologies is desirable
• Excellent communication abilities to work with stakeholders and researchers successfully
• Strong data analytic abilities and business sense are required to draw the appropriate conclusions from the dataset, respond to those conclusions, and clearly convey the key findings
• Excellent problem-solving and analytical skills
• Fluent in conversational and written English communication skills

Physical requirements:

• Normal, corrective vision range; ability to see color and to distinguish letters, numbers and symbols
• Frequently required to sit, stand, walk, talk, hear, bend and reach
• Ability to reach with hands and arms
• Occasionally lift and/or move up to 30lbs