Chief Data Engineer

3 days ago

Mysore, Karnataka, India beBeeDataEngineering Full time ₹ 15,00,000 - ₹ 25,00,000

Job Opportunity

We are seeking a skilled Data Engineer to fill this critical role. In this position, you will design and develop efficient ETL processes using Databricks PySpark to extract, transform, and load data from various sources into our data lake and data warehouse.

Key Responsibilities

  • Data Engineering Development
    • Design and develop ETL processes using Databricks PySpark to extract, transform, and load data from heterogeneous sources into our data lake and data warehouse.
    • Optimize ETL workflows for performance and scalability, leveraging Databricks PySpark and Spark SQL to efficiently process large data volumes.
    • Implement robust error handling and monitoring mechanisms to proactively detect and resolve issues within ETL processes.
    • Design and implement data solutions following the Medallion Architecture principles, organizing data into Bronze, Silver, and Gold layers.
    • Ensure data is appropriately cleansed, enriched, and optimized at each stage to support robust analytics and reporting.
  • Data Pipeline Management
    • Develop and maintain data pipelines using Databricks PySpark, ensuring data quality, integrity, and reliability throughout the ETL lifecycle.
    • Collaborate with data engineering, data science, and business intelligence teams to translate data requirements into efficient ETL workflows and pipelines.
  • Data Analysis and Query Optimization
    • Write and optimize complex SQL queries for data manipulation, aggregation, and analysis within Databricks PySpark applications.
  • Project Coordination and Continuous Improvement
    • Participate in project planning and coordination activities to ensure timely delivery of ETL solutions.
    • Stay updated on the latest developments in Databricks PySpark, Spark SQL, and related technologies, recommending and implementing best practices and optimizations.
    • Document ETL processes, data lineage, and metadata to facilitate knowledge sharing and ensure compliance with data governance standards.

Required Skills and Qualifications

  • Bachelor's degree in Computer Science, Engineering, or a related field.
  • Minimum of 3 years of experience as an ETL developer, with a strong focus on Databricks PySpark development.
  • Proficiency in Python programming, with extensive experience in developing and debugging Databricks PySpark applications.
  • In-depth understanding of Spark architecture and internals, with hands-on experience in Spark RDDs, DataFrames, and Spark SQL.
  • Expertise in writing and optimizing complex SQL queries for data manipulation, aggregation, and analysis.
  • Proven experience in working with large-scale data warehousing and ETL frameworks.
  • Strong problem-solving skills and the ability to troubleshoot and resolve ETL process issues.
  • Excellent communication and collaboration skills, with the ability to work effectively in a team environment.

Preferred Qualifications

  • Experience with cloud platforms with a preference for AWS.
  • Experience with data platform tools such as DataBricks, Snowflake, and Tableau.
  • Demonstrated ability to implement best practices for ETL processes and data management.
  • Strong understanding of data governance and data quality principles.
  • Relevant certifications in Databricks PySpark, Spark SQL, or related technologies.

  • Chief Engineer Leader

    3 days ago

    Mysore, Karnataka, India beBeeExcellence Full time ₹ 20,00,000 - ₹ 25,00,000

    Chief Engineer LeaderWe are seeking a visionary leader to join our team as Chief Engineer Leader. As a member of our senior leadership team, you will be responsible for providing strategic leadership to the School of Engineering.About the RoleLeadership and Strategy: Lead the School of Engineering in achieving its goals and objectives, ensuring alignment...

  • Chief Software Development Officer

    5 days ago

    Mysore, Karnataka, India beBeeSoftware Full time ₹ 2,00,00,000 - ₹ 2,50,00,000

    Job Title: Chief Software Development Officer We are seeking a seasoned software development leader to join our team. As Chief Software Development Officer, you will be responsible for designing and developing back-end SaaS applications, overseeing the software development lifecycle, and ensuring product features meet business requirements. The ideal...

  • Chief Data Engineering Specialist

    5 days ago

    Mysore, Karnataka, India beBeeData Full time ₹ 2,00,00,000 - ₹ 2,50,00,000

    We're seeking a highly skilled data engineer to fill this role.Key Responsibilities:Develop and maintain efficient ETL pipelines using FiveTran, DBT Cloud, and custom frameworksImplement client-specific data transformations for seamless integrationCollaborate with the quality control team for ETL validation and data consistency checksOptimize Snowflake...

  • Lead Bridge Engineering Specialist

    1 day ago

    Mysore, Karnataka, India beBeeBridge Full time ₹ 9,00,000 - ₹ 18,00,000

    Job Title: Chief Bridge Engineer">Main Responsibilities:">A key role within our team is available for a Chief Bridge Engineer. The successful candidate will be responsible for assessing existing bridges and preparing assessment reports with expertise in Steel Composite and Pretension/Posttension structures.">Key Skills and Qualifications:">The ideal...

  • Chief Data Architect

    1 day ago

    Mysore, Karnataka, India beBeeDataEngineering Full time ₹ 15,00,000 - ₹ 20,00,000

    About Data Engineering RoleWe are looking for a skilled Data Engineer to join our team. In this role, you will be responsible for designing and building scalable data platforms and applications that enable fast reporting and analytics.

  • Chief Data Architect

    4 days ago

    Mysore, Karnataka, India beBeeDataEngineering Full time US$ 68,800 - US$ 91,600

    Unlock your expertise as a Data Engineer and contribute to the development of innovative data systems.About this RoleThis is a groundbreaking, multi-year project initiative that demands advanced data engineering skills. You will be working on designing and implementing large-scale data architectures using Palantir Foundry, PySpark, and Typescript.Key...

  • Data Engineer

    3 days ago

    Mysore, Karnataka, India People Prime Worldwide Full time

    About Client:Our Client is a global IT services company headquartered in Southborough, Massachusetts, USA. Founded in 1996, with a revenue of $1.8B, with 35,000+ associates worldwide, specializes in digital engineering, and IT services company helping clients modernize their technology infrastructure, adopt cloud and AI solutions, and accelerate innovation....

  • Chief Data Architect

    3 days ago

    Mysore, Karnataka, India beBeeData Full time ₹ 10,00,000 - ₹ 15,00,000

    Job Overview:We are seeking a skilled and motivated Data Engineer to design, develop, and maintain scalable data pipelines that process and transform large volumes of structured and unstructured data.The ideal candidate will have a strong background in data engineering, cloud technologies, and data architecture, and will collaborate closely with data...

  • Chief Data Architect

    5 days ago

    Mysore, Karnataka, India beBeeDataEngineer Full time ₹ 20,00,000 - ₹ 25,00,000

    Job TitleAs a seasoned data engineer, you will be responsible for designing, building and maintaining high-performance data pipelines that process large volumes of data.Key Responsibilities:Design and implement scalable batch processing systems using Python and big data technologiesOptimize database performance, focusing on slow-running queries and latency...

  • Chief Data Strategist

    5 days ago

    Mysore, Karnataka, India beBeeData Full time ₹ 18,00,000 - ₹ 25,00,000

    Unlock Data ExcellenceWe are seeking an accomplished data specialist to contribute to our cutting-edge data solutions. This role involves designing, developing, and optimizing end-to-end data platforms using Microsoft technologies.Key Responsibilities:Design and implement scalable data solutions using Azure Synapse Analytics, Databricks, and ADF.Develop data...