Data Engineering Specialist

19 hours ago

Mangalore, Karnataka, India beBeeDataEngineering Full time ₹ 1,50,00,000 - ₹ 2,00,00,000

Job Title:

Data Engineering Specialist

A highly skilled Data Engineer is sought to join our team in designing, developing and maintaining scalable ETL processes.

The ideal candidate will possess strong expertise in Databricks PySpark, Spark SQL and Python programming with extensive experience in developing and debugging complex data pipelines.

Responsibilities:

  • Design and Development:
    • Design and develop efficient ETL processes using Databricks PySpark to extract, transform and load data from heterogeneous sources into data lake and data warehouse.
    • Optimize ETL workflows for performance and scalability leveraging Databricks PySpark and Spark SQL to efficiently process large data volumes.
    • Implement robust error handling and monitoring mechanisms to proactively detect and resolve issues within ETL processes.
  • Data Pipeline Management:
    • Develop and maintain advanced data pipelines using Databricks Workflows ensuring data quality, integrity and reliability throughout the ETL lifecycle.
    • Collaborate with cross-functional teams to translate data requirements into efficient ETL workflows and pipelines.
  • Query Optimization:
    • Write and optimize complex SQL queries for data manipulation, aggregation and analysis within Databricks PySpark applications.
  • Project Coordination:
    • Participate in project planning and coordination activities to ensure timely delivery of ETL solutions.

Required Qualifications:

  • Bachelor's degree in Computer Science or related field.
  • Minimum 3 years of experience as an ETL developer with a strong focus on Databricks PySpark development.
  • Proficiency in Python programming with extensive experience in developing and debugging complex Databricks PySpark applications.
  • In-depth understanding of Spark architecture and internals with hands-on experience in Spark RDDs, DataFrames and Spark SQL.
  • Expertise in writing and optimizing complex SQL queries for data manipulation, aggregation and analysis.
  • Proven experience in working with large-scale data warehousing and ETL frameworks.
  • Strong problem-solving skills and ability to troubleshoot and resolve ETL process issues.
  • Excellent communication and collaboration skills.

Preferred Qualifications:

  • Experience with cloud platforms.
  • Experience with data platform tools such as Databricks, Snowflake and Tableau.
  • Demonstrated ability to implement best practices for ETL processes and data management.

  • Cloud Data Engineering Specialist

    2 days ago

    Mangalore, Karnataka, India beBeeData Full time ₹ 60,00,000 - ₹ 1,20,00,000

    Cloud Data Engineering SpecialistWe are seeking a highly skilled data engineer to design and implement large-scale data systems using Azure services such as Azure Data Factory, Databricks, MS Synapse, and SQL.The ideal candidate will have extensive experience in developing real-time and close-to-real-time data integrations using Kafka, MS Event Hubs, etc.Key...

  • Data Architect Specialist

    2 hours ago

    Mangalore, Karnataka, India beBeeDataArchitecture Full time ₹ 1,50,00,000 - ₹ 2,00,00,000

    Job Title: Data Architect SpecialistWe are seeking a highly skilled data architect to lead our team in designing and implementing innovative solutions for data management.The ideal candidate will have experience with ETL processes, data warehousing principles, and strong problem-solving skills.This role is perfect for someone who enjoys working with big data...

  • Cloud Data Engineering Specialist

    14 hours ago

    Mangalore, Karnataka, India beBeeDataEngineer Full time ₹ 15,00,000 - ₹ 25,00,000

    Job Overview:We are seeking a highly skilled Data Engineer to join our team.Key Responsibilities:Collaborate with data scientists to develop and implement complex algorithms for data insightsApply agile software development processes to improve back-end systems iteratively, ensuring efficient data processing and analysisIntegrate front-end and back-end data...

  • Cloud Data

    3 days ago

    Mangalore, Karnataka, India beBeeDataScientist Full time ₹ 20,00,000 - ₹ 25,00,000

    Job Title: Cloud Data & AI SpecialistWe are seeking a highly skilled expert to lead the development of technical training content on Azure's modern data and AI stack. As a Cloud Data & AI Specialist, you will be responsible for defining and reviewing presentations, labs, and real-world scenarios that showcase advanced learning experiences.Key...

  • Data Engineer

    2 days ago

    Mangalore, Karnataka, India beBeeDataEngineer Full time ₹ 80,00,000 - ₹ 1,50,00,000

    Real-Time Data Infrastructure SpecialistThe role involves designing, developing, and maintaining large-scale data processing systems to support real-time data-driven decision making. The candidate will be responsible for building and implementing efficient data pipelines using technologies like Kafka, Flink, and Spark.Key Responsibilities:Design and develop...

  • AI Training Data Specialist

    2 days ago

    Mangalore, Karnataka, India beBeeSTEM Full time ₹ 7,50,000 - ₹ 15,00,000

    Content Development SpecialistWe are seeking an experienced Content Development Specialist to create high-quality training data sets for our AI models.The ideal candidate will have a strong background in STEM fields and excellent writing skills, with the ability to analyze and improve grammar, syntax, and sentence structure of training data.Key...

  • Senior Data Engineering Specialist

    3 days ago

    Mangalore, Karnataka, India beBeeDataEngineer Full time ₹ 9,00,000 - ₹ 12,00,000

    Job Opportunity for Senior Data EngineerWe are seeking a skilled professional to lead our data engineering efforts.5+ years of experienceExpertise in Python, Pyspark and SQLHands-on coding experience with Telemetry, Big Data and backend engineeringThe ideal candidate will have strong understanding of IoT systems, especially in Refrigeration units, HVAC...

  • Data Insights Specialist

    2 days ago

    Mangalore, Karnataka, India beBeeDataInsights Full time ₹ 1,20,00,000 - ₹ 2,00,00,000

    Unlock Your Potential as a Data Insights SpecialistAbout the Role:Data insights specialists play a vital role in driving business growth through data-driven decision making. This exciting opportunity involves working with complex data sets, developing predictive models, and communicating findings to stakeholders.

  • Data Ingestion Specialist

    6 days ago

    Mangalore, Karnataka, India beBeeDataIngestion Full time ₹ 1,00,00,000 - ₹ 2,00,00,000

    Job Title: Data Ingestion SpecialistWe are seeking a skilled data ingestion professional to join our organization. As a data ingestion specialist, you will play a critical role in designing, developing, and optimizing data ingestion pipelines for integrating multiple sources into cloud-based platforms.Key Responsibilities:

  • Data Linguistics Specialist

    4 days ago

    Mangalore, Karnataka, India beBeeLinguistics Full time ₹ 80,00,000 - ₹ 1,20,00,000

    Job OpportunityAre you interested in working on the forefront of innovation? As a Language Specialist, you will be joining a truly global team of language experts across various disciplines and entrusted with a range of responsibilities. We are seeking motivated individuals who can handle the speed required to be on the frontlines of innovation.Key...