Databricks Engineer

1 week ago

Kozhikode, Kerala, India beBeeData Full time ₹ 15,00,000 - ₹ 30,00,000

Job Title: Databricks Engineer

Job Description:

We are seeking an experienced Databricks Engineer to join our team. As a key member of our data platform implementation team, you will play a crucial role in designing, developing, and maintaining ETL processes that extract, transform, and load data from various sources into our data lake and data warehouse.

You will work closely with data engineers, data scientists, and business intelligence teams to build and optimize data workflows that support the project's analytics and reporting needs.

Main Responsibilities:

  1. ETl Development:

    • Design and develop ETL processes using Databricks PySpark to extract, transform, and load data from heterogeneous sources into our data lake and data warehouse.
    • Optimize ETL workflows for performance and scalability, leveraging Databricks PySpark and Spark SQL to efficiently process large data volumes.
    • Implement robust error handling and monitoring mechanisms to proactively detect and resolve issues within ETL processes.
    • Design and implement data solutions following the Medallion Architecture principles, organizing data into Bronze, Silver, and Gold layers.
    • Ensure data is appropriately cleansed, enriched, and optimized at each stage to support robust analytics and reporting.

  2. Data Pipeline Management:

    • Hands On experience in creating advanced data pipelines using databricks workflows
    • Develop and maintain data pipelines using Databricks PySpark, ensuring data quality, integrity, and reliability throughout the ETL lifecycle.
    • Collaborate with data engineering, data science, and business intelligence teams to translate data requirements into efficient ETL workflows and pipelines.

  3. Data Analysis and Query Optimization:

    • Write and optimize complex SQL queries for data manipulation, aggregation, and analysis within Databricks PySpark applications.

  4. Project Coordination and Continuous Improvement:

    • Participate in project planning and coordination activities to ensure timely delivery of ETL solutions.
    • Stay updated on the latest developments in Databricks PySpark, Spark SQL, and related technologies, recommending and implementing best practices and optimizations.
    • Document ETL processes, data lineage, and metadata to facilitate knowledge sharing and ensure compliance with data governance standards.
Required Skills and Qualifications:
  1. Bachelor's degree in Computer Science, Engineering, or a related field.
  2. Minimum of 3 years of experience as an ETL developer, with a strong focus on Databricks PySpark development.
  3. Proficiency in Python programming, with extensive experience in developing and debugging Databricks PySpark applications.
  4. In-depth understanding of Spark architecture and internals, with hands-on experience in Spark RDDs, DataFrames, and Spark SQL.
  5. Expertise in writing and optimizing complex SQL queries for data manipulation, aggregation, and analysis.
  6. Proven experience in working with large-scale data warehousing and ETL frameworks.
  7. Strong problem-solving skills and the ability to troubleshoot and resolve ETL process issues.
  8. Excellent communication and collaboration skills, with the ability to work effectively in a team environment.

Benefits:

This is a unique opportunity to grow your career as a senior engineer.

Other Opportunities:

The company encourages continuous learning, innovation, and professional growth.

  • Databricks Engineering Manager

    2 weeks ago

    Kozhikode, Kerala, India beBeeDataProcessing Full time ₹ 15,00,000 - ₹ 20,00,000

    Job Title:Databricks Engineering ManagerKey Responsibilities:Databricks Cluster Administration - Plan, implement and manage Databricks clusters to meet business needs.Spark Optimization - Expertise in Spark optimization, data caching, and performance tuning for efficient data processing.Data Processing - Proficiency in Python, Scala, or SQL for data...

  • Data Engineering Lead

    1 day ago

    Kozhikode, Kerala, India beBeeDataEngineer Full time ₹ 15,00,000 - ₹ 20,00,000

    Job Overview:As a seasoned data engineer, you will lead the development and maintenance of large-scale data pipelines using Databricks. Your expertise in designing and deploying scalable data architectures will ensure seamless data processing and storage.Key Responsibilities:Develop and implement complex data engineering projects with 6+ years of experience...

  • Chief Data Engineering Specialist

    1 week ago

    Kozhikode, Kerala, India beBeeDataEngineer Full time ₹ 15,00,000 - ₹ 25,00,000

    Job Description:We are seeking a seasoned data professional with expertise in Databricks and AWS to lead our data engineering initiatives. The ideal candidate will possess hands-on experience in designing, developing, and maintaining scalable data pipelines using Databricks and PySpark.

  • Cloud-Based Big Data Engineer

    3 days ago

    Kozhikode, Kerala, India beBeeDataEngineering Full time US$ 1,08,500 - US$ 1,53,865

    Azure Data Engineer RoleWe are seeking a highly skilled professional to help our clients harness the power of data and drive digital transformation.As an Azure Data Engineer, you will be responsible for designing and implementing scalable data pipelines using Azure Databricks (PySpark/Scala), developing and deploying Azure Function Apps for event-driven data...

  • Senior Data Infrastructure Engineer

    1 week ago

    Kozhikode, Kerala, India beBeeEngineering Full time ₹ 2,00,00,000 - ₹ 2,50,00,000

    Data Engineering LeadA seasoned data engineering professional is sought to lead the development and implementation of scalable data solutions. The ideal candidate will have a deep understanding of big data platforms, cloud-native environments, and distributed systems.About the Role:We are looking for an individual with expertise in designing and delivering...

  • Visionary Data Engineering Leader

    1 day ago

    Kozhikode, Kerala, India beBeeData Full time ₹ 1,80,00,000 - ₹ 2,16,00,000

    Job DescriptionAs a Senior Lead Data Engineer, you will be responsible for designing and implementing scalable data solutions that support various business functions. This includes leading the design and implementation of high-performance data pipelines using Databricks and Azure Data Services.Key aspects of this role include:Leading technical teams and...

  • Highly Skilled Data Engineering Expert

    1 week ago

    Kozhikode, Kerala, India beBeeDataEngineer Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

    We are seeking a seasoned Senior Data Engineer with expertise in Google Cloud (GCP) to take up this challenging role.Key Responsibilities:Design and build scalable workflows using Airflow, Tivoli etc. for efficient data processingWork with Big Data tools: Hadoop, Spark, Kafka for complex data analysisHands-on experience in Databricks (Unity Catalog,...

  • Chief Data Engineering Specialist

    7 days ago

    Kozhikode, Kerala, India beBeeData Full time ₹ 15,00,000 - ₹ 25,00,000

    We're seeking a seasoned professional to lead our data engineering initiatives. This is an excellent opportunity for a highly skilled individual to drive the design, development, and implementation of cutting-edge data solutions.The ideal candidate will have expertise in modern data warehouse technologies, including Snowflake and Azure Data Factory. They...

  • Senior Python Engineer

    7 days ago

    Kozhikode, Kerala, India beBeeEngineering Full time ₹ 1,50,00,000 - ₹ 2,00,00,000

    Job SummaryWe are seeking a seasoned expert to lead our Python engineering efforts.This role involves designing and implementing high-performance data pipelines using Apache Beam, Databricks, and other technologies.The successful candidate will have hands-on experience with performance optimization, concurrency, and observability.About the RoleIn this...

  • Chief Data Architect

    1 week ago

    Kozhikode, Kerala, India beBeeData Full time ₹ 30,00,000 - ₹ 45,00,000

    Job TitleA highly skilled Lead Data Engineer is required to join our team.Design, develop and optimize data pipelines using Azure and Databricks.Hands-on work with Databricks Asset Bundles (DABs) for CI/CD and environment management.Build and manage scalable ELT/ETL workflows with Databricks, Delta Lake, ADF, and other Azure services.Continuously adopt and...