Senior Data Systems Developer

We seek a skilled Data Engineer to play a key role in our team's data engineering operations. As part of our team, you will be responsible for designing and implementing large-scale data pipelines using modern DevOps tools.

• Design and implement CI/CD pipelines using GitHub Actions or equivalent tools to ensure smooth code integration and deployment.
• Configure and maintain branch protection rules and workflows in GitHub to guarantee reliable code integration and deployment.
• Bid on complex data processing tasks with Spark jobs in Databricks for large-scale data processing.
• Automate Databricks deployments using Databricks Asset Bundles (DABs) or similar frameworks.
• Develop, test, and maintain robust data pipelines using Python and SQL.
• Implement unit testing, validation, and quality checks for data workflows to ensure high-quality data delivery.
• Manage multi-environment deployments (Dev, Stage, Prod) ensuring reliability, consistency and high data integrity.
• Write and maintain configuration files (YAML, JSON) for workflows, pipelines, and automation.
• Troubleshoot and resolve configuration or syntax issues in YAML and related files.
• Keep critical Databricks notebooks running reliably to ensure business continuity.
• Migrate legacy reports into modern, self-serve dashboards (Snowflake, Tableau, Power BI).

Key Responsibilities:

1. Data Pipeline Design and Implementation: Develop, deploy, and maintain efficient data pipelines that enable scalable data processing and analytics.
2. CI/CD Pipeline Automation: Implement automated CI/CD pipelines using GitHub Actions or equivalent tools to streamline code deployment and reduce errors.
3. Spark Job Management: Manage complex Spark jobs in Databricks for large-scale data processing and optimize job performance.
4. Databricks Deployment Automation: Automate Databricks deployments using Databricks Asset Bundles (DABs) or similar frameworks to enhance efficiency and scalability.
5. Python and SQL Development: Develop, test, and maintain robust data pipelines using Python and SQL to ensure high-quality data delivery.
6. Quality Assurance and Testing: Implement unit testing, validation, and quality checks for data workflows to ensure high-quality data delivery.
7. Multi-Environment Deployments: Manage multi-environment deployments (Dev, Stage, Prod) ensuring reliability, consistency and high data integrity.
8. Configuration File Maintenance: Write and maintain configuration files (YAML, JSON) for workflows, pipelines, and automation.
9. Troubleshooting and Support: Troubleshoot and resolve configuration or syntax issues in YAML and related files to ensure system reliability.
10. Critical Notebooks Maintenance: Keep critical Databricks notebooks running reliably to ensure business continuity.
11. Legacy Report Migration: Migrate legacy reports into modern, self-serve dashboards (Snowflake, Tableau, Power BI) to enhance user experience and insights.

Required Skills and Qualifications:

• Programming Languages: Strong skills in SQL, Databricks (Spark), and Python/R programming languages.
• GitHub and CI/CD Practices: Familiarity with GitHub and CI/CD practices (branch protection, GitHub Actions workflows).
• Unit Testing and Quality Checks: Experience in implementing unit tests and quality checks in data workflows.
• Databricks Asset Bundles: Knowledge of Databricks Asset Bundles (DABs) or similar automation tools.
• YAML Configuration Management: Familiarity with YAML configuration management, debugging, and error resolution.
• Problem-Solving and Debugging: Strong problem-solving and debugging skills in data engineering workflows.
• Cloud Data Warehouse: Experience with Snowflake or another cloud data warehouse.
• Dashboards and Usability: Hands-on experience with Tableau/Power BI dashboards and a good sense for usability.

What We Offer:

We offer a competitive salary and benefits package, as well as opportunities for professional growth and development.

About Our Team:

We are a dynamic and innovative team dedicated to delivering high-quality data solutions. We value collaboration, creativity, and continuous learning.