Databricks Expert

1 week ago


Hyderabad, Telangana, India TechVedika Full time
Job Description

We are seeking a Senior Data Engineer with in-depth knowledge of Databricks and Unity Catalog to serve as the subject matter expert for all things Databricks within our organization.

This role requires deep expertise in Databricks, including CI/CD setup for data bricks, data lineage through Unity Catalog, and strong proficiency in ETL, SQL, and modern data engineering practices.

You will be the go-to person for designing, implementing, and optimizing data solutions with Databricks.

Key Responsibilities:
  • Serve as the point of contact and subject matter expert for all Databricks-related activities, including architecture, development, and operational best practices.
  • Design, develop, and manage ETL/ELT pipelines in Databricks using Python (PySpark), integrating various data sources to support business operations.
  • Leverage Unity Catalog to ensure data lineage, security, and governance are properly managed across the Databricks environment.
  • Implement and maintain CI/CD pipelines for Databricks, ensuring smooth deployments, version control, and automation using Git and other DevOps tools.
  • Build scalable data architectures, including Data Lakes, Lakehouses, and Data Warehouses, ensuring efficient data management and accessibility.
  • Configure and optimize Databricks clusters, jobs, and workflows for both batch and streaming data processing to handle large-scale datasets.
  • Stay up-to-date with the latest Databricks features and advancements, continuously enhancing our data engineering practices.
  • Collaborate with cross-functional teams to implement data governance and ensure compliance with security and industry regulations.
  • Monitor and tune Databricks workloads to ensure high performance and scalability, adapting to business needs as required.
  • Provide guidance and mentorship to junior engineers, ensuring adherence to best practices and fostering a collaborative environment.
Qualifications:
  • 5+ years of experience in data engineering with significant expertise in Databricks and Apache Spark.
  • Proficient in Unity Catalog for managing data lineage, security, and governance within the Databricks ecosystem.
  • Experience building and optimizing ETL pipelines using tools like Azure Data Factory, Informatica, or similar.
  • Strong understanding of CI/CD practices with experience in Git for version control and integration with Databricks.
  • Expertise in SQL development and performance tuning for large-scale datasets.
  • Knowledge of the Azure ecosystem, including data services like Azure Data Lake and Azure Storage.
  • Ability to work with both batch and streaming data processing pipelines.
  • Experience with data modeling and dimensional design (e.g., star schema).
  • Good understanding of data governance, compliance, and security best practices.
  • Excellent communication and problem-solving skills, with the ability to manage multiple priorities.
  • Ability to stay current on Databricks innovations and proactively introduce new features and capabilities to the team.


  • Hyderabad, Telangana, India TechVedika Full time

    About TechVedikaWe are a product engineering company focused on Artificial Intelligence (AI) solutions, providing cutting-edge products and services to various industries.Our mission is to deliver simple, elegant tech solutions that perform complex tasks, enabling organizations to improve operational efficiency and unleash new business potential.Job...


  • Hyderabad, Telangana, India travash Full time

    Job Title: Data Engineer: Spark and Databricks ExpertAbout the Role:We are seeking a highly skilled Data Engineer with expertise in Spark and Databricks to join our team at Travash. As a Data Engineer, you will be responsible for designing, developing, and maintaining data pipelines using Databricks and Spark, as well as other cloud technologies as...


  • Hyderabad, Telangana, India Risk Resources LLP Full time

    About the RoleWe are seeking a highly skilled Data Engineering Expert to join our team at Risk Resources LLP. As a Pyspark Databricks Developer, you will play a critical role in developing and implementing data processing pipelines using Pyspark on the Databricks platform.Key ResponsibilitiesDesign and Develop Data Pipelines: Utilize Pyspark and Databricks...


  • Hyderabad, Telangana, India Tiger Analytics Full time

    Job Title: Databricks Solutions ArchitectWe're seeking an experienced Databricks Expert/Architect to join Tiger Analytics's data services team. This role will play a key part in designing, implementing, and optimizing Databricks solutions across the organization.Key Responsibilities:Analyze business requirements and translate them into technical...


  • Hyderabad, Telangana, India FREELANCER CAREER Full time

    We are seeking a Freelance with a strong background in supporting daily operations for 22.5 hours.Key Requirements:• Relevant experience: 7 years• Heavy focus on Azure Databricks and Terraform• Devops position handling data and infrastructureKey Responsibilities:• Deploying Databricks clusters using Terraform• Converting Unity catalog to Terraform...

  • Data Engineer

    1 week ago


    Hyderabad, Telangana, India travash Full time

    Job SummaryWe are seeking a highly skilled Data Engineer with expertise in Databricks and Spark to join our team at Travash. The successful candidate will design, develop, and maintain data pipelines using Databricks and Spark, ensuring optimal performance, scalability, and reliability.Key ResponsibilitiesDesign and develop data pipelines using Databricks...

  • Data Engineer

    4 days ago


    Hyderabad, Telangana, India FREELANCER CAREER Full time

    We are seeking a Freelance with a strong background in data and infrastructure to support our team.Key requirements include:At least 7 years of relevant experience in Azure Databricks and TerraformProven expertise in deploying Databricks clusters using TerraformAble to convert Unity Catalog to Terraform within a monthStrong understanding of Azure and Azure...

  • Data Engineer

    2 weeks ago


    Hyderabad, Telangana, India travash Full time

    Job SummaryAt Travash, we are seeking a highly skilled Data Engineer to join our team. As a key member of our data engineering team, you will be responsible for designing, developing, and maintaining data pipelines using Databricks and Spark, as well as other cloud technologies as needed.Key Responsibilities• Design and develop data pipelines using...

  • Data Engineer

    7 days ago


    Hyderabad, Telangana, India travash Full time

    Data Engineer Job DescriptionWe are seeking a highly skilled Data Engineer to join our team at Travash. The ideal candidate will have expertise in Databricks and Spark, as well as experience with cloud technologies and big data technologies.Key Responsibilities:Design, develop, and maintain data pipelines using Databricks and Spark.Optimize data pipelines...

  • Data Engineer

    2 weeks ago


    Hyderabad, Telangana, India travash Full time

    Job SummaryWe are seeking a highly skilled Data Engineer to join our team at Travash. As a Data Engineer, you will be responsible for designing, developing, and maintaining data pipelines using Databricks and Spark. You will work closely with data scientists, analysts, and other stakeholders to understand and meet their data needs.Key ResponsibilitiesDesign...

  • Senior Data Engineer

    2 weeks ago


    Hyderabad, Telangana, India FREELANCER CAREER Full time

    We are seeking a highly skilled Freelance to support our team with a daily commitment of 22.5 hours.The ideal candidate will have at least 7 years of relevant experience in Azure Databricks and Terraform.The primary focus of this role will be on deploying Databricks clusters using Terraform and converting Unity Catalog to Terraform within a month.Azure and...


  • Hyderabad, Telangana, India Genpact Full time

    Transform Business with Data-Driven InsightsAt Genpact, we harness the power of data to drive business outcomes. Our team of experts, led by a Senior Databricks Analytics Engineer, is responsible for designing and implementing analytics solutions that deliver actionable insights.Key Responsibilities:Develop and maintain optimal analytics solutions by...


  • Hyderabad, Telangana, India State Street Full time

    Key Responsibilities: As the Databricks Technology Delivery Lead, you will be responsible for providing technical leadership to the Funding & Collateral Optimization Team of Global Markets for the REPO business. You will work closely with business stakeholders and technology teams to deliver all major components of the business. Your key responsibilities...


  • Hyderabad, Telangana, India UnitedHealth Group Full time

    At UnitedHealth Group, we're committed to helping people live healthier lives and making the health system work better for everyone. As a Data Engineering Consultant, you'll play a critical role in delivering care and improving health outcomes by connecting people with the care, pharmacy benefits, data, and resources they need to feel their best.Key...


  • Hyderabad, Telangana, India UnitedHealth Group Full time

    At UnitedHealth Group, we strive to make a positive impact on the health and well-being of our communities. As a Principal Data Engineer, you will play a key role in helping us achieve this mission. This is an exciting opportunity to lead a team of engineers and drive innovation in data engineering, using your expertise in Java, Pyspark, Azure, and...


  • Hyderabad, Telangana, India UnitedHealth Group Full time

    At UnitedHealth Group, our mission is to help people live healthier lives and make the health system work better for everyone. We believe everyone-of every race, gender, sexuality, age, location and income-deserves the opportunity to live their healthiest life. To help us achieve this mission, we are seeking a highly skilled Data Engineering Consultant to...


  • Hyderabad, Telangana, India State Street Full time

    Role Summary & Job Description:The Enterprise Data Strategy Lead is responsible for providing strong technical leadership to the Funding & Collateral Optimization Team of Global Markets for REPO business and closely involved in the delivery of all major components. The Lead should be reviewing the current state data & application landscape and setting the...


  • Hyderabad, Telangana, India 451 Discovery Comm. India Full time

    About the Role:At 451 Discovery Comm. India, we are seeking a highly skilled Senior Software Engineer to join our Data Platform team. As a Senior Software Engineer - Data Validation Expert, you will be responsible for designing, developing, and deploying scalable and efficient data pipelines using Databricks, Apache Spark, and other relevant technologies.Key...

  • Data Engineer

    3 weeks ago


    Hyderabad, Telangana, India AppSierra Solutions Pvt Ltd Full time

    Job Title: Data Engineer - Azure ExpertWe are seeking a highly skilled Data Engineer with expertise in Azure technologies to join our team at AppSierra Solutions Pvt Ltd. As a Data Engineer, you will be responsible for designing and implementing data solutions that meet the needs of our organization.Key Responsibilities:Design, build, and deploy scalable...


  • Hyderabad, Telangana, India Digile Full time

    Data Engineering RoleAs a Senior Data Engineer at Digile, you will utilize your expertise in designing and implementing data pipelines using industry-leading tools like Databricks and Azure Data Factory (ADF).Develop and maintain data ingestion and transformation pipelines, adhering to best practices and ensuring data quality and integrity.Collaborate with...