Databricks Administrator

Job Description

The Databricks Administrator/Engineer will serve as an important member of LabCorp's Analytics team focused on supporting Tier 1 Business Intelligence (BI) analytic server administration. This person is part of a team of application administrators and will be responsible for the administration and support of internal-facing Databricks servers. Additionally, the selected candidate will be responsible for supporting Databricks developers/development, application integration and to end-user setup and administration.

Primary Responsibilities

• The Databricks Administrator will be responsible for providing technical expertise in deploying applications on AWS/Azure Databricks platform.
• The Databricks Administrator must be self-motivated and apply knowledge of Azure to drive solutions, support the development team, and create documentation to support and describe technical solutions.
• Strong experience with Databricks administration, managing cloud-based big data platforms.
• Hands-on experience with managing permissions, AD group integration, and service principals within Databricks.
• Experience in setting up and managing permissions for Databricks catalogs, schemas, and jobs.
• Familiarity with cloud platforms (AWS, Azure, GCP) and integration with Databricks environments
• Install, configure, and maintain Databricks clusters and workspaces.
• Monitor cluster performance, resource utilization, and troubleshoot issues to ensure optimal performance.
• Implement and manage access controls and security policies to protect sensitive data.
• Perform regular software updates and patches.
• Provide technical support to Databricks users, including troubleshooting and issue resolution.
• Conduct training sessions to educate users on platform best practices and capabilities.
• Optimize cluster configurations for performance and cost-effectiveness.
• Monitor and manage auto-scaling to ensure resources are allocated efficiently.
• Implement and enforce security policies, access controls, and encryption mechanisms.
• Stay up-to-date with security best practices and compliance requirements.
• Develop and maintain backup and disaster recovery strategies to ensure data integrity and availability.
• Collaborate with data engineers to integrate Databricks with other data sources, data warehouses, and data lakes.
• Monitor and manage platform costs, including optimizing resource utilization to control expenses.
• Maintain detailed documentation of configurations, procedures, and best practices.
• Collaborate with cross-functional teams, including data scientists, data engineers, and business analysts,
• To understand their requirements and provide technical solutions

Required Skills/Experience

• Bachelor's degree in computer science or an analytical discipline or equivalent experience
• Proven experience with Databricks server administration
• 7+ years overall IT experience
• 5+ years Databricks Administration preferably in a high availability architecture
• Must be able to demonstrate ability to lead system and application upgrades
• Proficient with troubleshooting techniques to determine root cause analysis and formulate solution recommendations
• Experience in Memory Management, Handle Scheduling queues & Messages, Performance Management, Supervise Clusters Health check, Regular Back up & recovery, Node Commissioning & Decommissioning, Hive connections, Kafka, Spark, Map Reduce, Tez, User Group Administration, Privilege Checks, Code Deployment to Production, Server Log Analysis and Scheduled Report Monitoring & Back up Monitoring
• Excellent written and verbal communication skills
• Works under minimal supervision with a high degree of self-motivation, commitment and integrity
• Willingness to learn new technologies at an accelerated pace
• Provides clear communication to senior management on issue identification and resolution
• Leverage in-depth business knowledge to translate complex business analysis and reporting requirements into technical designs
• Practical skills in deploying large scale environment processing millions of records per day in a high-availability environment.