Senior Databricks Administrator

Description
Key Responsibilities :

User Provisioning And Management

• Onboard and offboard users, groups, and service principals within Databricks, including integration with identity providers (IdPs) like Azure Active Directory or Okta via SCIM.
• Manage user roles and entitlements at both the account and workspace levels (Account Admins, Workspace Admins, Metastore Admins, etc.).
• Implement and maintain role-based access control (RBAC) and attribute-based access control (ABAC) to ensure appropriate data and resource access.

Data Lake Governance (Unity Catalog Focus)

• Configure and manage Unity Catalog metastores, catalogs, schemas, and tables.
• Define and enforce data access policies (e.g., table-level, column-level, row-level security) using Unity Catalog.
• Manage data lineage and auditing capabilities to track data flow and usage.
• Collaborate with data owners and stakeholders to define data quality standards and ensure data integrity.
• Implement data retention and lifecycle management policies.
• Aligning Data Sensitivity Policy to Enforceable Data Governance :
• Translate organizational data classification and sensitivity policies into technical controls within Databricks.
• Utilize features like data masking and encryption to protect sensitive information.
• Ensure compliance with regulatory requirements (e.g., GDPR, HIPAA, CCPA) by implementing appropriate security measures.
• Conduct regular security audits and vulnerability assessments.

Managing Cluster And Budget Policies

• Define and implement compute policies to control cluster creation, configuration, and resource usage, ensuring cost optimization.
• Monitor and manage serverless budget policies to attribute usage to specific teams or projects.
• Optimize cluster configurations for performance and cost-effectiveness, leveraging features like auto-scaling and auto-termination.
• Manage cluster pools to reduce startup times and improve resource allocation.

Reporting And Addressing Discrepancies

• Monitor Databricks platform health, performance, and resource utilization.
• Identify and troubleshoot issues related to user access, data availability, cluster performance, and policy violations.
• Generate reports on platform usage, costs, security incidents, and compliance.
• Investigate and resolve discrepancies in data, reports, or system behavior in collaboration with data engineers, data scientists, and other teams.
• Develop and maintain comprehensive documentation of configurations, procedures, and best practices.

Collaboration And Support

• Provide technical support and guidance to Databricks users, data engineers, and data scientists.
• Collaborate with cloud infrastructure teams (AWS, Azure, GCP) to manage underlying cloud resources.
• Stay up-to-date with the latest Databricks features, best practices, and industry trends.

Technical Skills
Primary Skills for a Databricks Administrator:
Databricks Platform Expertise

• Deep understanding of Databricks architecture, workspaces, and key components (Unity Catalog, Delta Lake, Spark, SQL Analytics).
• Proficiency in Databricks administration console and APIs.
• Experience with Databricks Workflows, Jobs, and Delta Live Tables (DLT) for orchestration and pipeline management.
• Cloud Platform Knowledge : Strong experience with AWS and its relevant services.

Data Governance & Security

• Solid understanding of data governance principles, data classification, and data lifecycle management.
• Experience implementing security controls, access policies (RBAC), and encryption.
• Familiarity with compliance standards (GDPR, HIPAA, CCPA) and auditing practices.

Programming & Scripting

• Proficiency in SQL for data querying and access control.
• Deep expertise in Terraform is essential, extending beyond basic knowledge to managing complex, multi-project infrastructure. This includes hands-on experience with custom Terraform modules crucial for Data Mesh orchestration.
• Scripting skills (e.g., Python, Terraform) for automation and administrative tasks.
• Familiarity with Spark and PySpark concepts for troubleshooting and optimization.
• Identity and Access Management (IAM) : Experience with enterprise identity providers (e.g., Azure AD, Okta, Active Directory) and SCIM provisioning.
• Networking Concepts : Understanding of network security, VPNs, VPCS, private links, VPC peering, and connectivity within cloud environments.
• Monitoring & Logging Tools : Experience with monitoring tools (e.g., Datadog, Observe, cloud-native monitoring) for platform health and performance.

Soft Skills

• Problem-Solving and Troubleshooting : Ability to diagnose and resolve complex technical issues efficiently.
• Communication : Excellent verbal and written communication skills to interact with technical and non-technical stakeholders.
• Attention to Detail : Meticulous in configuring policies, managing access, and ensuring data integrity.
• Proactive and Self-Driven : Ability to anticipate issues, recommend solutions, and continuously improve the platform.
• Collaboration : Work effectively with cross-functional teams (data engineers, data scientists, security teams).
• Analytical Thinking : Ability to analyze data and system logs to identify trends and discrepancies.

Why Youll Love Working Here

• Data with purpose : Work on problems that directly impact how the world builds secure software
• Modern tooling : Leverage the best of open-source and cloud-native technologies
• Collaborative culture : Join a passionate team that values learning, autonomy, and impact

)