Lead Azure Databricks Engineer

3 weeks ago

Bangalore, Karnataka, India ResourceTree Global Services Full time

Resource Tree Global Services Pvt Ltd is hiring Senior Databricks Data Engineer for one of its leading client in Content & Data Analytics. Below are the details .

Job Title : Senior Databricks Data Engineer

Skills Required : Azure, Databricks, Datalake, Python/Scala/Java, Data Migration Strategies, Data Warehiusing, Data Modelling, Strong knowledge on Data Engneering, ETL Concepts.

Location : Bangalore

Notice Period : Immediate

Job Summary :

We are seeking an experienced Senior Databricks Data Engineer with a very good understanding of Databricks, Lakehouse concepts, with architecture experience, and a sound knowledge of Azure and its services. The Senior Databricks Data Engineer will play a pivotal role in designing, developing, and maintaining data pipelines and data solutions on the Databricks platform, with a focus on building a Lakehouse architecture. This role requires a deep knowledge of data engineering fundamentals, a strong grasp of Databricks and Lakehouse principles, the ability to architect data solutions for scalability and efficiency while leveraging Azure services, and expertise in data migration strategies. Understanding Hadoop eco-system will add brownie points.

Key Responsibilities :

  • Architect Lakehouse Solutions: Design, implement, and architect Lakehouse solutions on Databricks, leveraging Delta Lake, and utilize Azure services to enhance data storage and processing.
  • In-depth Databricks Expertise: Demonstrate a deep understanding of Databricks platform features, including Spark SQL, Delta Lake, and Databricks notebooks, to optimize data engineering processes.
  • Data Transformation: Implement advanced data transformations and quality checks within the Lakehouse architecture to ensure data accuracy, completeness, and consistency.
  • Performance Optimization: Architect data pipelines for performance and scalability, making efficient use of Databricks clusters and Azure resources.
  • Data Modelling: Create and implement advanced data models and schemas on Databricks, aligning with Lakehouse principles for analytical and reporting needs.
  • Data Integration: Seamlessly integrate data from diverse sources, aligning with Lakehouse principles for data ingestion and storage, leveraging Azure Data Lake Storage and Azure SQL Data Warehouse.
  • Lakehouse Architecture Experience: Apply architecture principles to build and maintain a scalable, efficient, and resilient Lakehouse architecture on Databricks while utilizing Azure for enhanced data processing and storage.
  • Data Security: Implement and maintain comprehensive data security and access controls within the Lakehouse architecture, utilizing Azure security features to safeguard sensitive data.
  • Data Migration Expertise: Possess a deep understanding of data migration strategies and practices, particularly in Azure environments, for seamless and secure data transfers. Experience on Hadoop ecosystem will be given advantage.
  • Collaboration: Collaborate closely with data scientists, data analysts, and cross-functional teams to understand data requirements and deliver Lakehouse-based solutions that meet their needs, incorporating Azure-based solutions and data migration strategies as needed.
  • Documentation: Create detailed documentation for data pipelines, data models, Lakehouse architecture configurations, Azure integrations, and data migration plans.
  • Troubleshooting: Proactively identify and resolve complex data pipeline and architecture issues, ensuring data integrity and availability, with a focus on Azure monitoring and diagnostics.
  • Performance Monitoring: Employ advanced monitoring techniques and Azure-based tools to maintain the performance and health of the Lakehouse architecture, taking proactive measures to address potential bottlenecks or issues. Should also have understanding of cluster monitoring and sizing.
  • Data Governance: Ensure strict adherence to data governance and data management best practices within the Lakehouse architecture, utilizing Azure-based data governance solutions.
  • Continuous Learning: Stay updated with the latest trends and developments in Databricks, Lakehouse concepts, Azure, data migration strategies, and data engineering best practices, and incorporate them into projects.

Qualifications and Skills :

  • Bachelor's or higher degree in computer science, data engineering, or a related field.
  • Proven experience as a data engineer with a strong focus on Databricks, Lakehouse architecture, advanced data engineering concepts, Azure services, and data migration strategies.
  • Exceptional understanding of data engineering fundamentals, including data modeling, ETL, and data warehousing.
  • Proficiency in programming languages such as Python, Scala, or Java.
  • Extensive experience with Databricks platform features, advanced Delta Lake concepts, and a strong working knowledge of Azure services.
  • Strong problem-solving skills and the ability to work in a fast-paced, collaborative environment.
  • Excellent communication and teamwork skills.
  • Relevant certifications in Databricks, Lakehouse, Azure services, and data migration are highly desirable.
(ref:hirist.tech)

  • Azure Databricks Engineer

    3 weeks ago

    Bangalore, Karnataka, India ARR Recruitment Solutions Full time

    Azure Databricks Engineer About the Role :- We are seeking a highly skilled Azure Databricks Engineer to join our team and play a vital role in developing and maintaining our big data processing pipelines.- This role is ideal for a candidate with extensive experience in Azure data services and a strong understanding of Apache Spark for large-scale data...

  • Data Engineer

    3 weeks ago

    Bangalore, Karnataka, India Career Soft Solutions Full time

    Job Description :- 3+ years of relevant experience in Pyspark and Azure Databricks.- Proficiency in integrating, transforming and consolidating data from various structured and -unstructured data sources.- Good experience in SQL or native SQL query languages.- Strong experience in implementing Databricks notebooks using Python.- Good experience in Azure Data...

  • ADC Data Engineering Lead

    2 weeks ago

    Bangalore, Karnataka, India Growel Softech Pvt. Ltd. Full time

    Job Description:We are seeking an experienced ADC (Advanced Data Computing) Data Engineering Lead with 10-13 years of experience to join our team. The ideal candidate will have a strong background in Big Data engineering, with expertise in tools such as Databricks, Azure Data Factory, and SQL. As an ADC Data Engineering Lead, you will be responsible for...

  • Data Scientist

    3 weeks ago

    Bangalore, Karnataka, India VIDPRO CONSULTANCY SERVICES Full time

    About the job:Job Description:We are seeking an experienced data scientist who apart from the required mathematical and statistical expertise also possesses the natural curiosity and creative mind to ask questions, connect the dots, and uncover opportunities that lie hidden with the ultimate goal of realizing the data's full potential.Primary Roles And...

  • Senior Data Engineer

    3 weeks ago

    Bangalore, Karnataka, India Growel Softech Pvt. Ltd. Full time

    Job Title: Azure Data Engineer. Location: Bangalore, India. Experience Level: 4-8 years. Job Description:. We are looking for an experienced Azure Data Engineer to join our team in Bangalore. As an Azure Data Engineer, you will be responsible for designing, developing, and maintaining scalable data pipelines and solutions using Azure Data Factory,...

  • Cloud Engineer

    3 days ago

    Bangalore, Karnataka, India Andor Tech Full time

    Experience Required: 5+ YearsImmediate Joiners Preferred: Max 15 Design, deploy, and maintain cloud-based solutions on the Azure platform, including Azure Databricks, Azure Data Factory, Azure Data Lake Storage (ADLS), and Azure Synapse Analytics.- Develop and optimize software applications and scripts using Python, SQL, PySpark, and Unix scripting to...

  • ETL Data Engineer

    2 weeks ago

    Bangalore, Karnataka, India Codersbrain technology pvt ltd Full time

    Job Description: ETL - Data Design, Develop and maintain ETL pipelines using Pyspark in Azure Databricks using delta tables.- Create build from Github and release pipeline for Ingestion and Databricks using Azure Devops / Harness- Monitor Performance of ETL Jobs, resolve any issue that arose and improve the performance metrics as needed.- Diagnose system...

  • Microsoft Azure Data Engineer

    3 weeks ago

    Bangalore, Karnataka, India Randstad India Full time

    Job Title : Microsoft Azure Data Services. Location : PAN INDIA. Experience : 8 -10 Years. Roles & Responsibilities :- Design, build, and configure applications to meet business process and application requirements using Microsoft Azure Data Services.- Collaborate with cross-functional teams to deliver impactful data-driven solutions.- Develop and maintain...

  • Databricks Engineer

    1 day ago

    Bangalore, Karnataka, India Hunarstreet Technologies Full time

    Databricks Role Overview :As a Data Engineer specializing in Databricks, you will play a pivotal role in collaborating with our clients to leverage the Databricks platform for extracting actionable insights from their data. From designing and implementing scalable data pipelines to optimizing workflows and fostering cross-functional collaboration, you will...

  • CGI - Azure Data Engineer - Data Factory & Databricks

    3 weeks ago

    Bangalore, Karnataka, India CGI Full time

    Job Title : Azure Data EngineerExperience : 4 to 10 yearsCategory : Software Development/ EngineeringLocation : Bangalore onlyNotice Period: Immediate / Max 30 DaysEmployment Type : Full TimeQualifying Criteria :- 4+ years of experience in Azure Data Engineer - ADF, ADB, ADLS- Notice period / Joining duration - Immediately / Less than 30 Days- Willing to...

  • Vsquare Systems

    3 weeks ago

    Bangalore, Karnataka, India Vsquare Systems Pvt. Ltd. Full time

    Databricks AdministratorJob Title : Databricks Administrator. Location : Bangalore / Hybrid. Job Type : Full-time. Job Description : .- We are seeking a skilled and experienced Databricks Administrator to join our team.- As a Databricks Administrator, you will be responsible for the installation, configuration, maintenance, and optimization of our Databricks...

  • Azure Engineer

    2 weeks ago

    Bangalore, Karnataka, India Onward Technologies Limited Full time

    Job Description :- Develop scripts to process both structured and unstructured data, ensuring efficient data processing and management.- Recommend, develop, and implement strategies to improve data reliability, efficiency, and quality, driving continuous improvement in our data processes.- Support the translation of business needs into technical system...

  • Data Modeler

    3 weeks ago

    Bangalore, Karnataka, India Whizz HR Full time

    We are seeking an experienced and skilled Data Modeler to join our team on a remote basis. The ideal candidate will have extensive expertise in data modeling and a strong background in utilizing Azure Data Lake and Databricks. This is an active position with a high priority, offering a competitive rate and the potential for a long-term engagement.Key...

  • Databricks Administrator

    3 weeks ago

    Bangalore, Karnataka, India TalentPulse Consultant Full time

    About the job :Responsibilities :- Install, configure, and maintain Databricks environments to support data engineering and analytics activities.- Collaborate with cross-functional teams to understand data requirements and optimize Databricks clusters accordingly.- Monitor Databricks performance, troubleshoot issues, and implement solutions to ensure system...

  • Azure DataBricks Quality Assurance

    3 weeks ago

    Bangalore, Karnataka, India Visara Full time

    Role: Azure ETL Tester., Databricks tester- Experienced in end-to-end of ETL (Data Pipeline) QA testing using various technologies in Azure cloud involving ingestion from varied data sources. - Should be familiar with data warehousing target and intermediate structures ideally in environments using data lakes. - Well versed in Databricks, SQL Database, Azure...

  • Data Modeler

    3 weeks ago

    Bangalore, Karnataka, India TalentPulse Consultant Full time

    Role : Data ModelerJob Description :We are seeking an experienced and skilled Data Modeler to join our team on a remote basis.The ideal candidate will have extensive expertise in data modeling and a strong background in utilizing Azure Data Lake and Databricks.This is an active position with a high priority, offering a competitive rate and the potential for...

  • Azure Data Engineer

    3 weeks ago

    Bangalore, Karnataka, India Talent Bee Consulting Pvt Ltd Full time

    About the job :Ideal Candidate :- 3+ years in Azure Data Engineering- Very good at ADF (Azure Data Factory)- Hands-on experience in designing and implementing data solutions on AzureOrganization :A global provider of Automation solutions in Assembly, Inspection, and Testing for manufacturing industries. With a mission to enhance the efficiency of the...

  • Azure Data Engineer

    1 week ago

    Bangalore, Karnataka, India Smart Consulting - Creating Future Leaders Full time

    EARLY JOINERS WILL BE PREFERABLE. Company : BIG 4 MNC. Role : Azure Data Engineer. Location : Bangalore. Experience : 3-8 Years. CTC : 22 LPA Max (Depends on current CTC, experience, skills). Mandatory skills : Azure, Azure Cloud, Azure Databricks, Azure Datafactory, Azure Datalake, Python, SQL, Relational databases (Oracle, DB2, SQL Server), Spark, Agile....

  • Data Architect

    3 weeks ago

    Bangalore/Hyderabad, Karnataka, India Job Store Consulting Full time

    Designation : Data ArchitectExp : 12+ yrsLocation : Bangalore / Hyderabad / Indore (Hybrid mode of work)Notice : Immediate-15 daysShift timings : 1 PM-10 PM / 2 PM-11 PM (IST)Key skills : Data Architecture, ADF, Azure Purview, Databricks, Azure Fabric Services, API, MySQL, Knowledge in Python, ETL and Data ModelingWhat you will do:Design and develop data...

  • Azure Data Engineer

    3 weeks ago

    Bangalore, Karnataka, India Rocket Consulting Full time

    Total Experience : 6+ yrsWork Location : BangaloreRole : Lead ConsultantShift: UK ShiftJob Description :- Relevant experience in Azure data engineering: 4 + years.- Relevant experience in SQL : 4 years.- Mandatory Skills : Databricks, PySpark/Scala/SparkSQL ,Azure Synapse, Data Factory, Azure Data Lake Storage, Azure SQL databases,Azure analysis services-...