Sonatype - Senior Databricks Administrator

Description :

Key Responsibilities :

User Provisioning and Management :

- Onboard and offboard users, groups, and service principals within Databricks, including integration with identity providers (IdPs) like Azure Active Directory or Okta via SCIM.

- Manage user roles and entitlements at both the account and workspace levels (Account Admins, Workspace Admins, Metastore Admins, etc.).

- Implement and maintain role-based access control (RBAC) and attribute-based access control (ABAC) to ensure appropriate data and resource access.

Data Lake Governance (Unity Catalog focus) :

- Configure and manage Unity Catalog metastores, catalogs, schemas, and tables.

- Define and enforce data access policies (e.g., table-level, column-level, row-level security) using Unity Catalog.

- Manage data lineage and auditing capabilities to track data flow and usage.

- Collaborate with data owners and stakeholders to define data quality standards and ensure data integrity.

- Implement data retention and lifecycle management policies.

- Aligning Data Sensitivity Policy to Enforceable Data Governance :

- Translate organizational data classification and sensitivity policies into technical controls within Databricks.

- Utilize features like data masking and encryption to protect sensitive information.

- Ensure compliance with regulatory requirements (e.g., GDPR, HIPAA, CCPA) by implementing appropriate security measures.

- Conduct regular security audits and vulnerability assessments.

Managing Cluster and Budget Policies :

- Define and implement compute policies to control cluster creation, configuration, and resource usage, ensuring cost optimization.

- Monitor and manage serverless budget policies to attribute usage to specific teams or projects.

- Optimize cluster configurations for performance and cost-effectiveness, leveraging features like auto-scaling and auto-termination.

- Manage cluster pools to reduce startup times and improve resource allocation.

Reporting and Addressing Discrepancies :

- Monitor Databricks platform health, performance, and resource utilization.

- Identify and troubleshoot issues related to user access, data availability, cluster performance, and policy violations.

- Generate reports on platform usage, costs, security incidents, and compliance.

- Investigate and resolve discrepancies in data, reports, or system behavior in collaboration with data engineers, data scientists, and other teams.

- Develop and maintain comprehensive documentation of configurations, procedures, and best practices.

Collaboration and Support :

- Provide technical support and guidance to Databricks users, data engineers, and data scientists.

- Collaborate with cloud infrastructure teams (AWS, Azure, GCP) to manage underlying cloud resources.

- Stay up-to-date with the latest Databricks features, best practices, and industry trends.

Primary Skills for a Databricks Administrator:

Technical Skills :

Databricks Platform Expertise :

- Deep understanding of Databricks architecture, workspaces, and key components (Unity Catalog, Delta Lake, Spark, SQL Analytics).

- Proficiency in Databricks administration console and APIs.

- Experience with Databricks Workflows, Jobs, and Delta Live Tables (DLT) for orchestration and pipeline management.

- Cloud Platform Knowledge : Strong experience with AWS and its relevant services.

Data Governance & Security :

- Solid understanding of data governance principles, data classification, and data lifecycle management.

- Experience implementing security controls, access policies (RBAC), and encryption.

- Familiarity with compliance standards (GDPR, HIPAA, CCPA) and auditing practices.

Programming & Scripting :

- Proficiency in SQL for data querying and access control.

- Deep expertise in Terraform is essential, extending beyond basic knowledge to managing complex, multi-project infrastructure. This includes hands-on experience with custom Terraform modules crucial for Data Mesh orchestration.

- Scripting skills (e.g., Python, Terraform) for automation and administrative tasks.

- Familiarity with Spark and PySpark concepts for troubleshooting and optimization.

- Identity and Access Management (IAM) : Experience with enterprise identity providers (e.g., Azure AD, Okta, Active Directory) and SCIM provisioning.

- Networking Concepts : Understanding of network security, VPNs, VPCS, private links, VPC peering, and connectivity within cloud environments.

- Monitoring & Logging Tools : Experience with monitoring tools (e.g., Datadog, Observe, cloud-native monitoring) for platform health and performance.

Soft Skills :

- Problem-Solving and Troubleshooting : Ability to diagnose and resolve complex technical issues efficiently.

- Communication : Excellent verbal and written communication skills to interact with technical and non-technical stakeholders.

- Attention to Detail : Meticulous in configuring policies, managing access, and ensuring data integrity.

- Proactive and Self-Driven : Ability to anticipate issues, recommend solutions, and continuously improve the platform.

- Collaboration : Work effectively with cross-functional teams (data engineers, data scientists, security teams).

- Analytical Thinking : Ability to analyze data and system logs to identify trends and discrepancies.

Why Youll Love Working Here :

- Data with purpose : Work on problems that directly impact how the world builds secure software

- Modern tooling : Leverage the best of open-source and cloud-native technologies

- Collaborative culture : Join a passionate team that values learning, autonomy, and impact