Azure Databricks Lead/Specialist

2 weeks ago

Pune, India Hoonar Tekwurks Private Limited Full time

Job Title : Azure Databricks Lead

Job Overview :

As an Azure Databricks Lead/Specialist, you will play a critical role in designing, implementing, and optimizing data solutions using Azure Databricks. Your expertise will contribute to building robust data pipelines, ensuring data quality, and enhancing overall performance. You'll collaborate with cross-functional teams to deliver high-quality solutions aligned with business requirements.

Responsibilities :

1. Design and Develop Data Pipelines:

- Create scalable data processing pipelines using Azure Databricks and PySpark.

- Implement ETL (Extract, Transform, Load) processes to ingest, transform, and load data from various sources.

- Collaborate with data engineers and architects to ensure efficient data movement and transformation.

2. Data Quality Implementation :

- Establish data quality checks and validation rules within Azure Databricks.

- Monitor data quality metrics and address anomalies promptly.

- Work closely with data governance teams to maintain data accuracy and consistency.

3. Unity Catalog Integration :

- Leverage Azure Databricks Unity Catalog to manage metadata, tables, and views.

- Integrate Databricks assets seamlessly with other Azure services.

- Ensure proper documentation and organization of data assets.

4. Delta Lake Expertise :

- Understand and utilize Delta Lake, which provides ACID transactions and time travel capabilities on top of data lakes.

- Implement Delta Lake tables for reliable data storage and versioning.

- Optimize performance by leveraging Delta Lake features.

5. Performance Tuning and Query Optimization:

- Profile and analyze query performance.

- Optimize SQL queries, Spark jobs, and transformations for efficiency.

- Tune resource allocation to achieve optimal execution times.

6. Resource Optimization :

- Manage compute resources effectively within Azure Databricks clusters.

- Scale clusters dynamically based on workload requirements.

- Monitor resource utilization and cost efficiency.

7. Source System Integration :

- Integrate Azure Databricks with various source systems (e.g., databases, data lakes, APIs).

- Ensure seamless data ingestion and synchronization.

- Handle schema evolution and changes in source data.

8 Stored Procedure Conversion in Databricks:

- Convert existing stored procedures (e.g., from SQL Server) into Databricks-compatible code.

- Optimize and enhance stored procedures for better performance within Databricks

- SSRS conversion experience

Qualifications and Skills:

- Education: Bachelor's degree in computer science, Information Technology, or a related field.

- Experience:

- Minimum 5 years of experience architecting and building data platforms on Azure.

- Proficiency in Azure Databricks, PySpark, and SQL.

- Familiarity with Delta Lake concepts.

Certifications (preferred) :

- Microsoft Certified: Azure Data Engineer Associate or similar.

- Databricks Certified Associate Developer for Apache Spark.

(ref:hirist.tech)

  • Azure Databricks Technical Lead

    2 days ago

    Pune, Maharashtra, India Hoonar Tekwurks Private Limited Full time

    Job Title: Azure Databricks Technical LeadThe ideal candidate for this role will have a minimum of 5 years of experience architecting and building data platforms on Azure. With expertise in Azure Databricks, PySpark, and SQL, the selected individual will play a critical role in designing, implementing, and optimizing data solutions using Azure Databricks.Our...

  • Lead Data Engineer for Azure Databricks

    2 days ago

    Pune, Maharashtra, India Hoonar Tekwurks Private Limited Full time

    Job Title: Lead Data Engineer for Azure DatabricksWe are seeking an experienced Lead Data Engineer to join our team at Hoonar Tekwurks Private Limited. The ideal candidate will have a strong background in designing and implementing data solutions using Azure Databricks and a minimum of 5 years of experience working with data platforms on Azure.Our estimated...

  • Azure databricks

    6 days ago

    Pune, India LTIMindtree Full time

    LTIMindtree is looking for Databricks-Spark SQL-Scala/Python/Java Total Experience- 12 to 16 Years Job Location- Pan India MANDATORY Azure Databricks Experts with Python and Time series processing experience Design construct install test and maintain highly scalable data management systems using Azure Databricks Develop data processing pipelines...

  • Azure Databricks

    4 weeks ago

    Pune, India LTIMindtree Full time

    LTIMindtree is looking for Databricks-SparkSQL-Scala/Python/JavaTotal Experience- 12 to 16 YearsJob Location- Pan IndiaMANDATORY Azure Databricks Experts with Python and Time series processing experienceDesign construct install test and maintain highly scalable data management systems using Azure DatabricksDevelop data processing pipelines using Databricks...

  • Azure databricks

    3 weeks ago

    Pune, India LTIMindtree Full time

    LTIMindtree is looking for Databricks-Spark SQL-Scala/Python/JavaTotal Experience- 12 to 16 YearsJob Location- Pan IndiaMANDATORY Azure Databricks Experts with Python and Time series processing experienceDesign construct install test and maintain highly scalable data management systems using Azure DatabricksDevelop data processing pipelines using Databricks...

  • Azure Databricks

    4 weeks ago

    Pune, India LTIMindtree Full time

    LTIMindtree is looking for Databricks-SparkSQL-Scala/Python/Java Total Experience- 12 to 16 Years Job Location- Pan India MANDATORY Azure Databricks Experts with Python and Time series processing experience Design construct install test and maintain highly scalable data management systems using Azure Databricks Develop data processing pipelines...

  • Azure Databricks

    4 weeks ago

    Pune, India LTIMindtree Full time

    LTIMindtree is looking for Databricks-SparkSQL-Scala/Python/Java Total Experience- 12 to 16 Years Job Location- Pan India MANDATORY Azure Databricks Experts with Python and Time series processing experience Design construct install test and maintain highly scalable data management systems using Azure Databricks Develop data processing pipelines using...

  • Azure Databricks

    4 weeks ago

    Pune, India LTIMindtree Full time

    LTIMindtree is looking for Databricks-SparkSQL-Scala/Python/JavaTotal Experience- 12 to 16 YearsJob Location- Pan India MANDATORY Azure Databricks Experts with Python and Time series processing experience Design construct install test and maintain highly scalable data management systems using Azure Databricks Develop data processing pipelines using...

  • Azure Databricks

    4 weeks ago

    Pune, India LTIMindtree Full time

    LTIMindtree is looking for Databricks-SparkSQL-Scala/Python/JavaTotal Experience- 12 to 16 YearsJob Location- Pan India MANDATORY Azure Databricks Experts with Python and Time series processing experience Design construct install test and maintain highly scalable data management systems using Azure Databricks Develop data processing pipelines using...

  • Machine Learning Azure Databricks Lead

    3 weeks ago

    Pune, India ACL Digital Full time

    Technical skills and competenciesWe are looking for a Machine Learning Engineer to focus on training and retraining existing ML models using Databricks, hosted entirely on the Azure cloud platform. The role involves creating and managing automated workflows for model retraining and integrating with BigQuery for data extraction and preprocessing. The ideal...

  • Machine learning azure databricks lead

    3 weeks ago

    Pune, India ACL Digital Full time

    Technical skills and competenciesWe are looking for a Machine Learning Engineer to focus on training and retraining existing ML models using Databricks, hosted entirely on the Azure cloud platform. The role involves creating and managing automated workflows for model retraining and integrating with Big Query for data extraction and preprocessing. The ideal...

  • Azure Databricks Engineer

    2 months ago

    Pune, India N-Systems Technologies Pvt Ltd Full time

    Job Title: Azure Databricks EngineerWe are seeking an experienced Data Engineer with Azure Databricks Expertise to join our team at N-Systems Technologies Pvt Ltd. The ideal candidate should have a strong background in designing, developing, and maintaining data pipelines and data streams using Azure Databricks and Python.Key Responsibilities:- Design and...

  • Machine Learning Azure Databricks Lead

    3 weeks ago

    Pune, India ACL Digital Full time

    Technical skills and competenciesWe are looking for a Machine Learning Engineer to focus on training and retraining existing ML models using Databricks, hosted entirely on the Azure cloud platform. The role involves creating and managing automated workflows for model retraining and integrating with BigQuery for data extraction and preprocessing. The ideal...

  • Machine Learning Azure Databricks Lead

    3 weeks ago

    Pune, India ACL Digital Full time

    Technical skills and competenciesWe are looking for a Machine Learning Engineer to focus on training and retraining existing ML models using Databricks, hosted entirely on the Azure cloud platform. The role involves creating and managing automated workflows for model retraining and integrating with BigQuery for data extraction and preprocessing. The ideal...

  • Machine Learning Azure Databricks Lead

    3 weeks ago

    Pune, India ACL Digital Full time

    Technical skills and competencies We are looking for a Machine Learning Engineer to focus on training and retraining existing ML models using Databricks, hosted entirely on the Azure cloud platform. The role involves creating and managing automated workflows for model retraining and integrating with BigQuery for data extraction and preprocessing. The ideal...

  • Azure Databricks

    6 months ago

    Pune, India Nityo Infotech Full time

    17 hours ago **Job Code**: JD-19482 **JOB DESCRIPTION**: Data Engineer(Databricks,NiFi,Microsoft Azure,Python,SQL,Hive,Linux **Experience Required**: 5 - 9 Years **Industry Type**: IT **Employment Type**: Permanent **Location**: India

  • LTIMindtree | Azure Databricks | pune

    4 weeks ago

    pune, India LTIMindtree Full time

    LTIMindtree is looking for Databricks-SparkSQL-Scala/Python/Java Total Experience- 12 to 16 Years Job Location- Pan India MANDATORY Azure Databricks Experts with Python and Time series processing experience Design construct install test and maintain highly scalable data management systems using Azure Databricks Develop data processing pipelines using...

  • LTIMindtree | Azure Databricks | pune

    4 weeks ago

    pune, India LTIMindtree Full time

    LTIMindtree is looking for Databricks-SparkSQL-Scala/Python/JavaTotal Experience- 12 to 16 YearsJob Location- Pan IndiaMANDATORY Azure Databricks Experts with Python and Time series processing experienceDesign construct install test and maintain highly scalable data management systems using Azure DatabricksDevelop data processing pipelines using Databricks...

  • Azure Databricks with Pyspark

    2 weeks ago

    Hinjawadi, Pune, Maharashtra, India BirlaSoft Full time

    Country/Region: IN- Requisition ID: 19901- Work Model: - Position Type: - Salary Range: - Location: INDIA - PUNE - BIRLASOFT OFFICE - HINJAWADI**Title**:Azure Databricks with Pyspark**: - Description: **Area(s) of responsibility**: Birlasoft, a powerhouse where domain expertise, enterprise solutions, and digital technologies converge to redefine business...

  • Azure Databricks with Pyspark

    2 days ago

    Hinjawadi, Pune, Maharashtra, India BirlaSoft Full time

    Country/Region: IN - Requisition ID: 20387 - Work Model: - Position Type: - Salary Range: - Location: INDIA - PUNE - BIRLASOFT OFFICE - HINJAWADI **Title**:Azure Databricks with Pyspark**: - Description: **Area(s) of responsibility**: Birlasoft, a powerhouse where domain expertise, enterprise solutions, and digital technologies converge to redefine...