Databricks Engineer

3 weeks ago

bangalore, India Unilode Full time

Databricks Engineer

Position Overview

The job purpose of a Databricks Engineer is to be a key contributor in ensuring that an organization’s data is processed efficiently, securely, and accurately using the Databricks platform. They support data analytics, provide technical expertise, manage platform resources, and optimize data workflows to help the organization derive valuable insights and make informed decisions.

General Responsibilities

Administer and manage Databricks clusters, workspaces, and resources. Monitor platform health, availability, and performance. Develop and maintain data pipelines for ingesting, transforming, and loading data into Databricks. Optimize ETL processes for efficiency and scalability. Collaborate with data scientists and analysts to build and optimize data processing and analytics workflows using Databricks notebooks. Develop and implement data transformation and data analysis solutions. Implement and manage data lakes within Databricks for structured and unstructured data storage. Ensure data lake security and access controls. Identify and address performance bottlenecks in Databricks workloads. Optimize queries, data pipelines, and resource allocation for efficient data processing. Implement security measures to protect data within Databricks, including access controls, encryption, and auditing. Ensure compliance with data privacy regulations and industry standards. Integrate Databricks with various data sources, databases, and external systems to enable seamless data flow. Ensure data source connectivity and data consistency. Manage cluster scaling and resource allocation to meet performance and cost requirements. Optimize cluster configurations for specific workloads. Identify and resolve technical issues, errors, and anomalies in Databricks workflows. Provide support to users encountering problems. Collaborate with data engineers, data scientists, and analysts to understand data requirements and provide technical guidance. Conduct training sessions and knowledge sharing to empower users with Databricks capabilities. Create and maintain documentation for Databricks workflows, configurations, and best practices. Promote and enforce coding and data engineering standards. Monitor and optimize Databricks costs, including cluster utilization and resource allocation. Ensure cost-efficient data processing. Stay current with Databricks updates and enhancements. Continuously improve Databricks workflows, processes, and infrastructure. Establish data governance practices within Databricks, including metadata management and data cataloging. Maintain data lineage and documentation for data assets. Implement data quality checks and validation processes to ensure data accuracy and reliability. Develop and enforce data quality standards. Plan for scalability and resilience in Databricks infrastructure to accommodate growing data volumes and ensure high availability. Additional duties as assigned.

Qualifications

Bachelor’s Degree: A bachelor’s degree in computer science, information technology, data engineering, or a related field is often required. Some employers may consider equivalent work experience or relevant certifications.  A minimum of 5 years of hands-on experience working with Databricks as a data engineer, data scientist, or similar role is typically expected. Strong experience in data engineering, including designing and developing data pipelines, ETL processes, and data transformations. Familiarity with big data technologies and frameworks such as Apache Spark, Hadoop, or similar platforms commonly used in conjunction with Databricks. Proficiency in programming languages commonly used in Databricks, such as Python, Scala, or SQL. Knowledge of Scala is often preferred. Proven ability to identify and address performance bottlenecks in Databricks workloads, optimizing query performance, and resource allocation. A commitment to staying updated with the latest Databricks updates, enhancements, and emerging technologies in the data engineering and analytics field.

  • Engineering Manager

    2 weeks ago

    bangalore, India Databricks Full time

    P-995 At Databricks, we are passionate about enabling data teams to solve the world's toughest problems — from making the next mode of transportation a reality to accelerating the development of medical breakthroughs. We do this by building and running the world's best data and AI infrastructure platform so our customers can use deep data insights to...

  • Sr Engineering Manager

    2 weeks ago

    bangalore, India Databricks Full time

    P-995 At Databricks, we are passionate about enabling data teams to solve the world's toughest problems — from making the next mode of transportation a reality to accelerating the development of medical breakthroughs. We do this by building and running the world's best data and AI infrastructure platform so our customers can use deep data insights to...

  • Technical Solutions Engineer- Platform

    2 weeks ago

    bangalore, India Databricks Full time

    CSQ424R4 Mission As a Platform Technical Solutions Engineer, you will provide technical support for Databricks Platform related issues and resolve any challenges involving the Databricks unified analytics platform. You will assist customers in their Databricks journey and provide them with the guidance, knowledge, and expertise that they need to realize...

  • Sr Recruiter

    2 weeks ago

    bangalore, India Databricks Full time

    GAQ225R99 We are looking for a Senior Tech Recruiter to help grow & expand our Tech verticals (across Engineering / IT / Technical Solutions etc) in India. You will be the driving force behind finding and engaging with some of the finest Tech talent in the industry as we continue to scale our team on all fronts. This is an exciting time to be part of the...

  • Senior Technical Solutions Engineer- Platform

    2 weeks ago

    bangalore, India Databricks Full time

    CSQ424R4 Mission As a Platform Technical Solutions Engineer, you will provide technical support for Databricks Platform related issues and resolve any challenges involving the Databricks unified analytics platform. You will assist customers in their Databricks journey and provide them with the guidance, knowledge, and expertise that they need to realize...

  • Backline Engineer, Platform

    3 weeks ago

    bangalore, India Databricks Full time

    At Databricks, we are passionate about enabling data teams to solve the world’s toughest problems — from making the next mode of transportation a reality to accelerating the development of medical breakthroughs. We do this by building and running the world’s best data and AI infrastructure platform so our customers can use deep data insights to improve...

  • Sr Backline Engineer

    1 week ago

    bangalore, India Databricks Full time

    Mission As a Apache Spark™ Backline Engineer you will help our customers to be successful with the Databricks Data Intelligence platform by resolving important technical customer escalations and the support team. You will be the technical bridge between support and engineering and the first line of defense for engineering. You will ensure that all...

  • Apache Spark™ Backline Engineer

    3 weeks ago

    bangalore, India Databricks Full time

    Mission As a Apache Spark™ Backline Engineer you will help our customers to be successful with the Databricks Data Intelligence platform by resolving important technical customer escalations and the support team. You will be the technical bridge between support and engineering and the first line of defense for engineering. You will ensure that all...

  • Senior Software Engineer

    3 weeks ago

    bangalore, India Databricks Full time

    P-926 At Databricks, we are passionate about enabling data teams to solve the world's toughest problems — from making the next mode of transportation a reality to accelerating the development of medical breakthroughs. We do this by building and running the world's best data and AI infrastructure platform so our customers can use deep data insights to...

  • Staff Software Engineer

    3 weeks ago

    bangalore, India Databricks Full time

    P-926 At Databricks, we are passionate about enabling data teams to solve the world's toughest problems — from making the next mode of transportation a reality to accelerating the development of medical breakthroughs. We do this by building and running the world's best data and AI infrastructure platform so our customers can use deep data insights to...

  • Technical Solutions Engineer- Apache Spark™

    2 weeks ago

    bangalore, India Databricks Full time

    CSQ125R2 Mission As a Technical Solutions Engineer you will help our customers to be successful with the Databricks Data Intelligence platform by resolving important technical customer escalations and the support team. You will be the technical bridge between support and engineering and the first line of defense for engineering. You will ensure that...

  • Sr Technical Solutions Engineer- Apache Spark™

    2 weeks ago

    bangalore, India Databricks Full time

    CSQ125R2 Mission As a Sr Technical Solutions Engineer you will help our customers to be successful with the Databricks Data Intelligence platform by resolving important technical customer escalations and the support team. You will be the technical bridge between support and engineering and the first line of defense for engineering. You will ensure...

  • Cloud Integration Engineer

    3 weeks ago

    bangalore, India Databricks Full time

    GAQ224R78 At Databricks Information Technology, we are a product-led organization transforming the way we work from how easy it is to use our IT services to the applications we develop that help us scale seamlessly in the face of incredible growth. We are hiring a Software Engineer (Cloud Integrations) to be part of a growing IT Data Analytics, Engineering &...

  • Delivery Solutions Architect

    3 weeks ago

    bangalore, India Databricks Full time

    CSQ125R58 At Databricks we are on a mission to empower our customers to solve the world's toughest data problems by utilizing the Lakehouse platform. As the first Delivery Solutions Architect (DSA) for the team in India, you will play an important role during this journey. You will collaborate with our field engineering teams to accelerate the adoption and...

  • Head of Product Security

    3 days ago

    bangalore, India Databricks Full time

    RDQ124R86 The Product Security Team's mission is to Left-shift SDLC (Security Development Lifecycle) processes for ALL code written in Databricks (for Customer Use or Supporting Customer internally) to reduce the likelihood of introducing new vulnerabilities in production and minimize the count and effect of externally identified vulnerabilities on...

  • Sr Manager, Product Security

    2 weeks ago

    bangalore, India Databricks Full time

    RDQ124R86 The Product Security Team's mission is to Left-shift SDLC (Security Development Lifecycle) processes for ALL code written in Databricks (for Customer Use or Supporting Customer internally) to reduce the likelihood of introducing new vulnerabilities in production and minimize the count and effect of externally identified vulnerabilities on...

  • Databricks Engineer

    1 week ago

    bangalore, India Prometheus Consulting Full time

    Role : Databricks Engineer.Experience Required : 6 To 10 years.Location : Experience Required : ML Engineer with STRONG Databricks experience, Spark, Python, SQL About the Role : Are you a passionate Databricks engineer looking to join a top MNC and revolutionize the way data is processed and analyzed? We are searching for a talented individual to...

  • Leader (Specialist), Data & AI

    3 weeks ago

    bangalore, India Databricks Full time

    FEQ225R60 As a Leader  (Strategic Accounts, Data & AI) will lead a team of Specialist Solutions Architects (Data & AI), who are Subject Matter Experts in Big Data / Data Engineering / AI & ML, focusing on large enterprise & strategic customers in India. You will help lead our APJ Region (Asia Pacific and Japan) expansion, building and lead a technical...

  • Azure Databricks Engineer

    4 weeks ago

    bangalore, India Tredence Inc. Full time

    Role: Senior Databricks Engineer / Databricks Technical Lead/ Data ArchitectExperience: 3-12 yearsLocation: Bangalore, Chennai, Delhi, Pune, KolkataAbout Tredence:Tredence focuses on last-mile delivery of powerful insights into profitable actions by uniting its strengths in business analytics, data science and software engineering. The largest companies...

  • Leader , Solutions Architecture (Data & AI)

    3 weeks ago

    bangalore, India Databricks Full time

    FEQ424R284 As a Senior Manager  (Strategic Accounts, Data & AI) will lead a team of Solutions Architects (Data & AI) focusing on large enterprise & strategic customers in India. You will help lead our APJ Region (Asia Pacific and Japan) expansion, you will build and lead a technical pre-sales team. Your experience partnering with the sales organisation...