Senior Data Engineer

Job Title: Senior Data Engineer

Basic Function

The Senior Data Engineer should be an expert familiar with all areas of data warehousing technical components (e.g., ETL, Reporting, Data Model), connected infrastructure, and their integrations. The ideal candidate will be responsible for developing the overall architecture and high-level design of the data schema environment. The candidate must have extensive experience with Star Schemas, Dimensional Models, and Data Marts. The individual is expected to build efficient, flexible, extensible, and scalable ETL design and mappings. Excellent written and verbal communication skills are required as the candidate will work very closely with diverse teams. A wide degree of creativity and latitude is expected. This position reports to the Manager of Data Services.

Typical Requirements

Requires strong technical and analytical skills, data management expertise, and business acumen to achieve results. The ideal candidate should be able to deep dive into data, perform advanced analysis, discover root causes, and design scalable long-term solutions using Databricks, Spark, and related technologies to address business questions. A strong understanding of business data needs and alignment with strategic goals will significantly enhance effectiveness. The role requires the ability to prepare high-level architectural frameworks for data services and present them to business leadership. Additionally, the candidate must work well in a collaborative environment while performing a variety of detailed tasks daily. Strong oral and written communication skills are essential, along with expertise in application design and a deep understanding of distributed computing, data lake architecture, and relational database concepts. This position requires the ability to leverage both business and technical capabilities regularly.

Essential Functions

1. Gather, structure, and process data from various sources (e.g., transactional systems, third-party applications, cloud-based financial systems, customer feedback, etc.) using Databricks and Apache Spark to enhance business insights.
2. Develop and enforce standards, procedures, and quality control measures for data analytics in compliance with enterprise policies and best practices.
3. Partner with business stakeholders to build scalable data models and infrastructure, leveraging Databricks' Delta Lake, MLflow, and Unity Catalog.
4. Identify, analyze, and interpret complex data sets to develop insightful analytics and predictive models.
5. Utilize Databricks to design and optimize data processing pipelines for large-scale data ingestion, transformation, and storage.
6. Ensure data infrastructure completeness and compatibility to support system performance, availability, and reliability requirements.
7. Architect and implement robust data pipelines using PySpark, SQL, and Databricks Workflows for automation.
8. Provide input on technical challenges and recommend best practices for data engineering solutions within Databricks.
9. Design and optimize data models for analytical and operational use cases.
10. Develop and implement monitoring, alerting, and logging frameworks for data pipelines.
11. Lead the architecture and implementation of next-generation cloud-based data solutions.
12. Build scalable and reliable data integration pipelines using Databricks, SQL, Python, and Spark.
13. Mentor and develop junior team members, fostering a data-driven culture within the organization.
14. Develop high-quality, scalable data solutions to support business intelligence, analytics, and data science initiatives.
15. Interface with technology teams to extract, transform, and load (ETL) data from diverse data sources into Databricks.
16. Continuously improve data processes, automating and simplifying workflows for self-service analytics.
17. Work with large, complex data sets to solve non-routine analysis problems, applying advanced machine learning and data processing techniques as needed.
18. Prototype, iterate, and scale data analysis pipelines, advocating for improvements in Databricks data structures and governance.
19. Collaborate cross-functionally to present findings effectively through data visualizations and executive-level presentations.
20. Research and implement advanced analytics, forecasting, and optimization methods to drive business outcomes.
21. Stay up to date with industry trends and emerging Databricks technologies to enhance data-driven capabilities.

Specialized Skills or Technical Knowledge

• Bachelor's degree or higher in a quantitative/technical field (e.g., Computer Science, Statistics, Engineering). A Master's degree in Computer Science, Mathematics, Statistics, or Economics is preferred.
• 5+ years of experience in data engineering, business intelligence, or data analytics, with a focus on Databricks and Apache Spark.
• Extensive experience with SQL and Python for developing optimized queries and data transformations.
• Expertise in Databricks ecosystem, including Delta Lake, ML flow, and Databricks SQL.
• Experience in designing and implementing ETL/ELT pipelines on Databricks using Spark and cloud-based data platforms (Azure Data Lake, AWS S3, or Google Cloud Storage).
• Strong data modeling, data warehousing, and data governance knowledge.
• Experience in working with structured and unstructured data, including real-time and batch processing solutions.
• Familiarity with data visualization tools such as Power BI, Tableau, or Looker.
• Deep understanding of distributed computing, scalable data architecture, and cloud computing frameworks.
• Hands-on experience with CI/CD pipelines, Infrastructure as Code (IaC), and DevOps practices in data engineering.
• Proven track record of working with cross-functional teams, stakeholders, and senior management to deliver high-impact data solutions.
• Knowledge of machine learning and AI-driven analytics is a plus.
• Strong problem-solving skills and the ability to work independently and in a team-oriented environment.
• Excellent communication skills and ability to convey complex data concepts to non-technical stakeholders.
• Experience in a franchised organization is a plus.