AWS Databricks

7 days ago

Bengaluru, India Cognizant Full time

Job Summary

We are seeking a AWS Databricks with 6 to 10 years of experience to join our team. The ideal candidate will have expertise in Spark in Scala Delta Sharing and Databricks Unity Catalog . This role involves working with cutting-edge technologies like Databricks CLI Delta Live Pipelines and Structured Streaming. The candidate will play a crucial role in managing risk and ensuring data integrity using tools such as Apache Airflow Amazon S3 and Python. The position is hybrid with no travel

Responsibilities

  • Develop and maintain scalable data pipelines using Spark in Scala to ensure efficient data processing and analysis.
  • Implement Delta Sharing and Databricks Unity Catalog to manage and secure data access across the organization.
  • Utilize Databricks CLI and Delta Live Pipelines to automate data workflows and improve operational efficiency.
  • Design and execute Structured Streaming processes to handle real-time data ingestion and processing.
  • Apply risk management strategies to identify and mitigate potential data-related risks.
  • Integrate Apache Airflow for orchestrating complex data workflows and ensuring seamless data operations.
  • Leverage Amazon S3 for data storage solutions ensuring high availability and durability of data assets.
  • Utilize Python for scripting and automation tasks to enhance productivity and streamline processes.
  • Develop and optimize Databricks SQL queries to extract meaningful insights from large datasets.
  • Implement Databricks Delta Lake to ensure data reliability and consistency across various data sources.
  • Manage Databricks Workflows to automate and schedule data tasks improving overall data management efficiency.
  • Collaborate with cross-functional teams to ensure alignment on data strategies and objectives.
  • Contribute to the continuous improvement of data practices and methodologies to support the companys mission.

Qualifications

  • Possess strong expertise in Spark in Scala and Databricks technologies.
  • Demonstrate proficiency in Delta Sharing and Databricks Unity Catalog .
  • Have experience with Databricks CLI and Delta Live Pipelines.
  • Show capability in Structured Streaming and risk management.
  • Be skilled in Apache Airflow and Amazon S3.
  • Have a strong command of Python for data-related tasks.
  • Be familiar with Databricks SQL and Delta Lake.
  • Understand Databricks Workflows and their applications.
  • Exhibit problem-solving skills and attention to detail.
  • Be able to work in a hybrid model with a focus on day shifts.
  • Have excellent communication and collaboration skills.
  • Be committed to continuous learning and professional development.
  • Be adaptable to changing technologies and business needs.

Certifications Required

Databricks Certified Associate Developer for Apache Spark

  • Databricks & AWS Lead

    2 days ago

    Bengaluru, Chennai, Hyderabad, India Krazy Mantra HR Solutions Pvt. Ltd Full time ₹ 10,00,000 - ₹ 25,00,000 per year

    We are looking for a skilled Databricks & AWS Lead with 7-14 years of experience to join our team in Pune, Mumbai, Chennai, Hyderabad, Bangalore, and Noida. The ideal candidate will have expertise in Databricks and AWS, Unity Catalog, Spark, SQL, and Python.Roles and ResponsibilityDesign and implement data pipelines using Databricks and AWS.Develop and...

  • Databricks + Python + Aws

    2 days ago

    Bengaluru, Karnataka, India ESK Technologies Full time

    **Department**:Software Development**: **We are Looking for Databricks Professionals** with 7-10 years of experience in designing data pipelines, ETL processes, and cloud-based data platforms using **Spark**, **Databricks**, and **AWS**. Proficiency in **Java/Python**, data structures, and modern engineering practices like **CI/CD** and **DevSecOps** is...

  • AWS Databricks Developer

    2 weeks ago

    Bengaluru, Mumbai, Pune, India LTIMindtree Limited Full time

    Job DescriptionMandate Skills:AWS, Databricks, Python, SQLResponsibility:- Designing and implementing scalable data pipelines using Databricks and Apache Spark.- Proficiency in programming languages such as Python and SQL.- Analysing and processing large datasets to uncover actionable insights.- Integrating data flows across AWS services to ensure seamless...

  • Resident Solutions Architect: Big Data

    4 days ago

    Bengaluru, India Databricks Full time

    As a** Resident Solutions Architect** in our Professional Services team you will work with clients on short to medium term customer engagements on their big data challenges using the Databricks platform. You will provide data engineering, data science, and cloud technology projects which require integrating with client systems, training, and other technical...

  • Resident Solutions Architect: Big Data

    1 week ago

    Bengaluru, India Databricks Full time

    As a** Resident Solutions Architect** in our Professional Services team you will work with clients on short to medium term customer engagements on their big data challenges using the Databricks platform. You will provide data engineering, data science, and cloud technology projects which require integrating with client systems, training, and other technical...

  • Solutions Data Architect

    2 days ago

    Bengaluru, India Databricks Full time

    As a **Solutions Architect** (Analytics, AI, Big Data, Public Cloud), you will guide the technical evaluation phase in a hands-on environment throughout the sales process. You will be a **technical advisor **internally to the sales team, and work with the product team as an advocate of your customers in the field. You will help our customers to achieve...

  • Architect : Digital Natives (Data & Ai)

    1 week ago

    Bengaluru, India Databricks Full time

    As a **Solutions Architect** (Analytics, AI, Big Data, Public Cloud), you will guide the technical evaluation phase in a hands-on environment throughout the sales process. You will be a **technical advisor **internally to the sales team, and work with the product team as an advocate of your customers in the field. You will help our customers to achieve...

  • Big Data Architect

    2 weeks ago

    Bengaluru, India Databricks Full time

    As a **Solutions Architect** (Analytics, AI, Big Data, Public Cloud), you will guide the technical evaluation phase in a hands-on environment throughout the sales process. This is a senior role, where will be a **technical advisor **internally to the sales team, and work with the product team as an advocate of your customers in the field. You will help our...

  • Principal Solutions Architect

    6 days ago

    Bengaluru, India Databricks Full time

    As a **Solutions Architect** (Analytics, AI, Big Data, Public Cloud), you will guide the technical evaluation phase in a hands-on environment throughout the sales process. You will be a **technical advisor **internally to the sales team, and work with the product team as an advocate of your customers in the field. You will help our customers to achieve...

  • Scale Solutions Engineer

    7 days ago

    Bengaluru, India Databricks Full time

    CSQ326R201 MissionAt Databricks, we are on a mission to empower our customers to solve the world's toughest data problems by utilising the Data Intelligence platform. As a Scale Solution Engineer, you will play a critical role in advising Customers in their onboarding journey. You will directly work with customers to help them onboard and deploy Databricks...