Databricks Data Engineer

2 weeks ago

Mumbai, India Crisil Full time

Position Overview:

The Databricks Data Engineer role is ideal for a skilled and motivated Databricks Data Engineer to design, build, and optimize data pipelines and analytics solutions on the Databricks Lakehouse Platform. This role is ideal for candidates who have experience with big data technologies and a passion for building robust data systems.

Key Responsibilities:

  • Develop and maintain scalable data pipelines using Databricks and Apache Spark (primarily with PySpark).
  • Build and manage Delta Lake tables and optimize them for analytics and machine learning workloads.
  • Integrate data from diverse sources including APIs, cloud storage, databases, and streaming platforms like Kafka.
  • Transform raw data into clean, structured formats using best practices in ETL/ELT.
  • Implement data quality checks and ensure consistency, completeness, and accuracy of data sets.
  • Collaborate with data scientists, analysts, and business stakeholders to meet data requirements for reporting and analytics.
  • Automate workflows using Databricks Workflows, job clusters, and scheduling tools.
  • Monitor and optimize Spark jobs for performance and cost efficiency.
  • Document and maintain data models, pipeline designs, and technical workflows.

Qualifications:

  • Education: Bachelor's degree in Computer Science, Engineering, or a related field.
  • Experience:
  • 3+ years of experience in data engineering or software development.
  • 1+ years of hands-on experience working with Databricks and Apache Spark in production environments
  • Skills:
  • Proficient in PySpark and SQL.
  • Solid understanding of data warehousing, ETL/ELT design, and big data best practices.
  • Experience with cloud platforms such as Azure, AWS, or GCP.
  • Experience with Delta Lake, Databricks Workflows, and MLflow.
  • Exposure to data governance tools like Unity Catalog or Purview.
  • Experience with real-time data processing (e.g., using Kafka, Structured Streaming).
  • Familiarity with version control systems (e.g., Git) and CI/CD practices.
  • Familiarity with data visualization tools (e.g., Power BI, Tableau) and BI integration
  • Certifications :
  • Databricks Certified Data Engineer Associate or Professional (a plus).

  • Enterprise Architect

    2 weeks ago

    Mumbai, India Databricks Full time

    As a **Solutions Architect** (Analytics, AI, Big Data, Public Cloud), you will guide the technical evaluation phase in a hands-on environment throughout the sales process. This is a senior role, where will be a **technical advisor **internally to the sales team, and work with the product team as an advocate of your customers in the field. You will help our...

  • Principal Solutions Architect

    11 hours ago

    Mumbai, India Databricks Full time

    As a Solutions Architect (Analytics, AI, Big Data, Public Cloud), you will guide the technical evaluation phase in a hands-on environment throughout the sales process. You will be a **technical advisor** internally to the sales team, and work with the product team as an advocate of your customers in the field. You will help our customers to achieve tangible...

  • Principal Solutions Architect

    11 hours ago

    Mumbai, India Databricks Full time

    As a Solutions Architect (Analytics, AI, Big Data, Public Cloud), you will guide the technical evaluation phase in a hands-on environment throughout the sales process. You will be a **technical advisor** internally to the sales team, and work with the product team as an advocate of your customers in the field. You will help our customers to achieve tangible...

  • Solutions Architect

    11 hours ago

    Mumbai, India Databricks Full time

    As a **Solutions Architect** (Analytics, AI, Big Data, Public Cloud), you will guide the technical evaluation phase in a hands-on environment throughout the sales process. You will be a **technical advisor **internally to the sales team, and work with the product team as an advocate of your customers in the field. You will help our customers to achieve...

  • Presales Architect

    2 days ago

    Mumbai, Maharashtra, India Databricks Full time

    As a **Solutions Architect** (Analytics, AI, Big Data, Public Cloud), you will guide the technical evaluation phase in a hands-on environment throughout the sales process. You will be a **technical advisor **internally to the sales team, and work with the product team as an advocate of your customers in the field. You will help our customers to achieve...

  • Delivery Solutions Architect

    11 hours ago

    Mumbai, India Databricks Full time

    At Databricks we are on a mission to empower our customers to solve the world's toughest data problems by utilizing the Lakehouse platform. As a Delivery Solutions Architect (DSA), you will play a critical role during this journey. The DSA works across a small number of our **largest or highest potential key accounts**, collaborating across Databricks teams...

  • Resident Solutions Architect

    2 weeks ago

    Mumbai, India Databricks Full time

    As a** Resident Solutions Architect** in our Professional Services team you will work with clients on short to medium term customer engagements on their big data challenges using the Databricks platform. You will provide data engineering, data science, and cloud technology projects which require integrating with client systems, training, and other technical...

  • Resident Solutions Architect

    1 week ago

    Mumbai, India Databricks Full time

    As a** Resident Solutions Architect** in our Professional Services team you will work with clients on short to medium term customer engagements on their big data challenges using the Databricks platform. You will provide data engineering, data science, and cloud technology projects which require integrating with client systems, training, and other technical...

  • Solutions Architect

    12 hours ago

    Mumbai, Maharashtra, India Databricks Full time

    FEQ126R37 As a **Solutions Architect** (Analytics, AI, Big Data, Public Cloud), you will guide the technical evaluation phase in a hands-on environment throughout the sales process. You will be a **technical advisor **internally to the sales team, and work with the product team as an advocate of your customers in the field. You will help our customers to...

  • Data Engineer(Databricks)

    2 days ago

    Navi Mumbai, Pune, India eClerx Full time ₹ 15,00,000 - ₹ 25,00,000 per year

    We have an opening for Data Engineer(Databricks) - Navi Mumbai/Pune.Interested candidates can share updated resume on PFB JD:Strong proficiency in Python or Scala programming languages.Hands-on experience with Databricks and Apache Spark.Knowledge of SQL and data warehousing concepts.Experience with cloud platforms (AWS, Azure, GCP).Understanding of data...