Staff Backline Engineer

Staff Backline Engineer (Apache Spark)

At Databricks, we are passionate about enabling data teams to solve the world's toughest problems — from making the next mode of transportation a reality to accelerating the development of medical breakthroughs. We do this by building and running the world's best data and AI infrastructure platform so our customers can use deep data insights to improve their business. Founded by engineers, we leap at every opportunity to tackle technical challenges, from designing next-gen UI/UX for interfacing with data to scaling our services and infrastructure across millions of virtual machines. And we're only getting started.

The Backline Engineering Team serves as the critical bridge between Engineering and Frontline Support. We handle complex technical issues and escalations across the Apache Spark ecosystem. With a strong focus on customer success, we are committed to delivering exceptional customer satisfaction by providing deep technical expertise, proactive issue resolution, and continuous improvements to the platform. We emphasise automation and tooling to enhance troubleshooting efficiency, reduce manual efforts, and improve the overall supportability of the platform. By developing smart solutions and streamlining workflows, we drive operational excellence and ensure a seamless experience for both customers and internal teams.

• Troubleshoot and resolve complex customer issues related to Apache Spark core internals, Spark SQL, Structured Streaming, Databricks Delta, and the Databricks AI product stack.
• Perform in-depth code-level analysis of customer applications to identify root causes and deliver actionable solutions.
• Conduct a comprehensive performance analysis of Spark workloads, identifying opportunities to improve latency, throughput, and resource utilisation.
• Optimise Spark applications by tuning configurations, refining caching strategies, and adjusting execution parameters.
• Review the Spark job code for adherence to best practices in performance, scalability, and maintainability, and provide recommendations.
• Resolve performance issues in Spark applications, including slow queries, failed jobs, memory leaks, and other anomalies.
• Build troubleshooting guides and runbooks to support the team.
• Develop notebook-based repro scenarios and Data Engineering pipelines to demonstrate feature capabilities.
• Collaborate with the Spark Engineering team to raise awareness of upcoming features and releases, and identify bugs with potential workarounds.
• Coordinate with engineering and escalation teams to ensure the timely resolution of customer issues.
• Participate in both weekend and weekday on-call rotations.

• 10+ years of industry experience developing, testing, and sustaining Python or Java or Scala-based applications.
• Comfortable with compiling, building and navigating the Apache Spark source code.
• Comfortable with identifying and applying patches/bug fixes to the Apache Spark source code.
• Experience in Big Data/Hadoop/Spark/Kafka/Elasticsearch data pipelines.
• Hands-on experience with SQL-based database systems.
• Experience with performing complex troubleshooting steps like profiling services, analysing various metrics, and processing dumps to identify and resolve issues.
• Ability to review the source code and identify the root cause of the issues in one or more distributed environments.
• Strong understanding of distributed systems and their internal implementation.
• Experience in JVM, GC, and Thread dump-based troubleshooting is required.
• Experience with AWS or Azure-related services.
• A bachelor's degree in Computer Science or a related field is required.

About Databricks

Databricks is the data and AI company. More than 10,000 organizations worldwide — including Comcast, Condé Nast, Grammarly, and over 50% of the Fortune 500 — rely on the Databricks Data Intelligence Platform to unify and democratize data, analytics and AI. Databricks is headquartered in San Francisco, with offices around the globe and was founded by the original creators of Lakehouse, Apache Spark, Delta Lake and MLflow. To learn more, follow Databricks on Twitter, LinkedIn and Facebook.

Benefits

At Databricks, we strive to provide comprehensive benefits and perks that meet the needs of all of our employees. For specific details on the benefits offered in your region, please visit

Our Commitment to Diversity and Inclusion

At Databricks, we are committed to fostering a diverse and inclusive culture where everyone can excel. We take great care to ensure that our hiring practices are inclusive and meet equal employment opportunity standards. Individuals looking for employment at Databricks are considered without regard to age, color, disability, ethnicity, family or marital status, gender identity or expression, language, national origin, physical and mental ability, political affiliation, race, religion, sexual orientation, socio-economic status, veteran status, and other protected characteristics.

Compliance

If access to export-controlled technology or source code is required for performance of job duties, it is within Employer's discretion whether to apply for a U.S. government license for such positions, and Employer may decline to proceed with an applicant on this basis alone.