Databricks PySpark Developer

6 days ago

Davangere, Karnataka, India beBeeDataDeveloper Full time ₹ 8,00,000 - ₹ 15,00,000
Job Overview

We are seeking a highly skilled and experienced ETL developer to join our team. The ideal candidate will have expertise in Databricks PySpark development and a strong understanding of data governance principles.

The primary responsibility of this role is to design, develop, and maintain efficient ETL processes for large-scale data warehousing and analytics. This involves working closely with the data science and business intelligence teams to translate data requirements into efficient workflows and pipelines.

As a key member of our team, you will be responsible for:

  • Designing and developing ETL processes using Databricks PySpark to extract, transform, and load data from heterogeneous sources into our data lake and data warehouse.
  • Optimizing ETL workflows for performance and scalability, leveraging Databricks PySpark and Spark SQL to efficiently process large data volumes.
  • Implementing robust error handling and monitoring mechanisms to proactively detect and resolve issues within ETL processes.
  • Designing and implementing data solutions following the Medallion Architecture principles, organizing data into Bronze, Silver, and Gold layers.
  • Ensuring data quality, integrity, and reliability throughout the ETL lifecycle.
Required Skills and Qualifications
  • Bachelor's degree in Computer Science, Engineering, or a related field.
  • Minimum of 3 years of experience as an ETL developer, with a strong focus on Databricks PySpark development.
  • Proficiency in Python programming, with extensive experience in developing and debugging Databricks PySpark applications.
  • In-depth understanding of Spark architecture and internals, with hands-on experience in Spark RDDs, DataFrames, and Spark SQL.
  • Expertise in writing and optimizing complex SQL queries for data manipulation, aggregation, and analysis.
  • Proven experience in working with large-scale data warehousing and ETL frameworks.
  • Strong problem-solving skills and the ability to troubleshoot and resolve ETL process issues.
Preferred Qualifications
  • Experience with cloud platforms, preferably AWS.
  • Experience with data platform tools such as DataBricks, Snowflake, and Tableau.
  • Demonstrated ability to implement best practices for ETL processes and data management.
  • Strong understanding of data governance and data quality principles.
  • Relevant certifications in Databricks PySpark, Spark SQL, or related technologies.

  • Databricks Data Engineering Leader

    7 days ago

    Davangere, Karnataka, India beBeeData Full time ₹ 1,50,00,000 - ₹ 2,00,00,000

    About this Role:We are seeking a highly skilled Senior Databricks Data Engineer to design and develop scalable data pipelines using Databricks and PySpark.Key responsibilities include:Designing high-performance data pipelines leveraging Databricks and PySpark technologies.Implementing and managing data solutions on AWS, including services like S3, Glue,...

  • Business Intelligence Developer

    13 hours ago

    Davangere, Karnataka, India beBeeDataEngineer Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

    Job OverviewThis is a Data Engineer position that requires expertise in handling big data technologies.Main ResponsibilitiesInteract with customers to provide excellent communication and ensure their needs are met.Analyze and process large datasets using tools like Azure/AWS Databricks, Python/Scala/Spark/PySpark.Demonstrate strong SQL and RDBMS skills for...

  • Cloud Based Data Engineer

    5 days ago

    Davangere, Karnataka, India beBeePythonDeveloper Remote Work Freelance Full time ₹ 15,00,000 - ₹ 25,00,000

    We are seeking a skilled Python Developer to take on a pivotal role in our organization. The ideal candidate will possess exceptional expertise in designing and implementing cloud-based solutions using Azure and Google Cloud platforms.Develop and manage data pipelines using technologies like PySpark and Databricks;Design and implement scalable and reliable...

  • Unlock Data Potential as a Cloud Data Engineer

    2 days ago

    Davangere, Karnataka, India beBeeDataEngineering Full time ₹ 20,00,000 - ₹ 30,00,000

    Data Engineering ExpertWe are seeking a highly skilled data engineer to join our team in designing and implementing scalable data pipelines using Azure Databricks (PySpark/Scala) and developing and deploying Azure Function Apps for event-driven data processing and automation.

  • Senior Data Solutions Engineer

    6 days ago

    Davangere, Karnataka, India beBeeDataEngineering Full time ₹ 13,00,000 - ₹ 20,10,000

    Job Title: Data Architect We are seeking a highly skilled professional with expertise in designing, building, and optimizing scalable data pipelines. The ideal candidate will have hands-on experience working with Databricks, managing large-scale data processing workflows, and implementing data transformation workflows using PySpark and Scala Spark. Key...

  • Freelance Data Engineer – A High-Touch Specialist in Enterprise Infrastructure

    5 days ago

    Davangere, Karnataka, India beBeeDataEngineering Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

    Job TitleData Engineering Expert WantedWe are seeking a highly skilled Freelance Data Engineer to join our team.About the RoleDesign and Develop: ETL/ELT pipelines using PySpark, AWS, and Databricks.Perform Data Profiling: Using Python and SQL.Optimize Workflows: For performance, scalability, and cost efficiency.Build Automation Scripts: Using Linux and Bash...

  • Data Migration Strategist

    6 days ago

    Davangere, Karnataka, India beBeeCloud Full time ₹ 18,00,000 - ₹ 20,00,000

    Job DescriptionAs a senior data engineer, you will play a crucial role in designing and implementing large-scale cloud migration projects. Your expertise in ETL development and migration to the cloud will enable our organization to accelerate the delivery of complex applications and workloads.Key responsibilities include developing and implementing robust...

  • Senior Data Validation Expert

    5 days ago

    Davangere, Karnataka, India beBeeDataValidation Full time ₹ 15,00,000 - ₹ 20,00,000

    Job Opportunity:A highly skilled Data Validation Specialist is required to design and execute test cases for Databricks tables across various layers. The ideal candidate should possess in-depth knowledge of data validation concepts, experience with Databricks testing, and proficiency in validating data transformations, ingestion, and processing logic.Key...

  • Data Architectural Specialist

    7 days ago

    Davangere, Karnataka, India beBeeData Full time ₹ 1,50,00,000 - ₹ 2,00,00,000

    Job OverviewThe Senior Data Engineer is responsible for developing and maintaining large-scale data processing workflows using Azure Databricks, PySpark, and Apache Spark.Main Responsibilities:Design and implement cloud-based data architectures, including data lakes, lakehouses, and warehouses, following a standardized approach.Deploy and optimize data...

  • Big Data Solutions Architect

    2 days ago

    Davangere, Karnataka, India beBeeData Full time ₹ 20,00,000 - ₹ 26,00,000

    Job Title: Databricks EngineerLocation: RemoteJob Type: Full-TimeShift timings: 3 pm - 11 pm ISTAbout the Role:We are seeking an experienced Databricks engineer with a strong background in data engineering to build and optimize scalable, high-performance data solutions.Key Responsibilities:Design, develop, and maintain data pipelines and ETL processes on...