Lead Big Data Innovator

2 days ago

Belgaum, Karnataka, India beBeeDataEngineer Full time ₹ 15,00,000 - ₹ 20,00,000

Job Overview:

We are seeking a highly skilled Data Engineer to join our organization. In this role, you will be instrumental in designing and developing efficient ETL processes using Databricks PySpark.

Key Responsibilities:

  • ETL Development:
    • Design and develop ETL processes using Databricks PySpark to extract, transform, and load data from various sources into data lake and data warehouse.
    • Optimize ETL workflows for performance and scalability, leveraging Databricks PySpark and Spark SQL to efficiently process large data volumes.
    • Implement robust error handling and monitoring mechanisms to proactively detect and resolve issues within ETL processes.
    • Ensure data is appropriately cleansed, enriched, and optimized at each stage to support robust analytics and reporting.
  • Data Pipeline Management:
    • Create advanced data pipelines using Databricks workflows.
    • Develop and maintain data pipelines using Databricks PySpark, ensuring data quality, integrity, and reliability throughout the ETL lifecycle.
  • Data Analysis and Query Optimization:
    • Write and optimize complex SQL queries for data manipulation, aggregation, and analysis within Databricks PySpark applications.
  • Project Coordination and Continuous Improvement:
    • Participate in project planning and coordination activities to ensure timely delivery of ETL solutions.
    • Stay updated on the latest developments in Databricks PySpark, Spark SQL, and related technologies, recommending and implementing best practices and optimizations.
    • Document ETL processes, data lineage, and metadata to facilitate knowledge sharing and ensure compliance with data governance standards.

Required Skills:

  • Bachelor's degree in Computer Science, Engineering, or a related field.
  • Minimum of 3 years of experience as an ETL developer, with a strong focus on Databricks PySpark development.
  • Proficiency in Python programming, with extensive experience in developing and debugging Databricks PySpark applications.
  • In-depth understanding of Spark architecture and internals, with hands-on experience in Spark RDDs, DataFrames, and Spark SQL.

Preferred Qualifications:

  • Experience with cloud platforms with a preference for AWS.
  • Relevant certifications in Databricks PySpark, Spark SQL, or related technologies.

  • Data Innovation Lead

    4 days ago

    Belgaum, Karnataka, India beBeeDataEngineer Full time ₹ 2,00,00,000 - ₹ 2,50,00,000

    Senior Data EngineerThis role involves leading a team of data engineers in developing and implementing innovative solutions for data processing and analytics.About the Position:As a Senior Data Engineer, you will be responsible for designing and building large-scale data pipelines to meet business requirements. You will collaborate with cross-functional...

  • Big Data Architect

    1 day ago

    Belgaum, Karnataka, India beBeeData Full time ₹ 80,00,000 - ₹ 1,50,00,000

    Job Title: GCP Data EngineerTransforming raw data into actionable insights is crucial for our business. As a key member of our dynamic team, you will design, develop, and maintain robust data pipelines and analytics platforms that serve as the backbone of our product development.You will collaborate with cross-functional teams including data scientists,...

  • Big Data Engineering Expert

    3 days ago

    Belgaum, Karnataka, India beBeeData Full time ₹ 20,00,000 - ₹ 25,00,000

    Job OverviewA Senior Data Engineer leads a team of big data engineers, working closely with stakeholders and team members to understand their requirements and develop architectures that meet their needs.This role involves developing data pipelines for new sources, data transformations within the data lake, implementing GraphQL, working with NoSQL databases,...

  • Data Innovation Lead

    2 days ago

    Belgaum, Karnataka, India beBeeData Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

    Data Scientist RoleThis role offers a challenging and dynamic opportunity for an experienced data scientist to leverage their expertise in machine learning, deep learning, and AI to drive business growth. Key Responsibilities: Design and develop advanced machine learning models to solve complex business problems. Implement and optimize Large Language Models...

  • Big Data Engineer

    21 hours ago

    Belgaum, Karnataka, India beBeeData Full time ₹ 1,80,00,000 - ₹ 2,50,00,000

    We are seeking a skilled Big Data Engineer to manage our large-scale data processing systems. As a key member of the IT infrastructure team, you will be responsible for designing and implementing high-performance clusters to support big data workloads.The ideal candidate will have experience in Cloudera administration and data processing systems, with strong...

  • Innovative Data Strategist

    2 days ago

    Belgaum, Karnataka, India beBeeDataScientist Full time ₹ 2,00,00,000 - ₹ 2,50,00,000

    Principal Data Scientist/AI ArchitectWe are seeking a visionary leader to spearhead our data science initiatives. This role will be instrumental in defining our data strategy, driving innovation in AI/ML, and building scalable data solutions that directly impact our business.Data Strategy: Define and champion the company's data science vision and strategy,...

  • Senior Big Data Engineer

    4 days ago

    Belgaum, Karnataka, India beBeeDataEngineer Full time ₹ 1,80,00,000 - ₹ 2,50,00,000

    Senior Big Data EngineerSkillful and experienced big data engineer sought after to design, build, deploy, and maintain large-scale data processing systems using cloud-based technologies. Key responsibilities include development of high-performance data pipelines, optimization of existing codebases, and implementation of data warehousing concepts.Required...

  • Innovative Lead Data Scientist Position

    7 days ago

    Belgaum, Karnataka, India beBeeDataScientist Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

    Lead Data Scientist – Gen AIWe are seeking a highly skilled Data Scientist with expertise in leveraging data, analytics, and artificial intelligence (AI) technologies to drive business outcomes.The successful candidate will be responsible for leading end-to-end initiatives, including business understanding, data preparation, modeling, evaluation, and...

  • Chief Big Data Solutions Architect

    4 days ago

    Belgaum, Karnataka, India beBeeData Full time US$ 1,00,000 - US$ 1,50,000

    Big Data Engineer">Transform online content into tangible business results by processing massive amounts of data to provide actionable insights to clients.Design, build and maintain high-performance data pipelines that process terabytes of data. Work on optimizing existing systems, identifying and resolving performance bottlenecks, and implementing solutions...

  • Senior Big Data Solutions Architect

    4 days ago

    Belgaum, Karnataka, India beBeeData Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

    Job DescriptionWe are seeking a skilled Data Engineer to design, develop, and maintain data pipelines and ETL processes on Databricks. The ideal candidate will lead the architecture and implementation of scalable, secure, and high-performance data solutions.The Data Engineer will collaborate with business stakeholders, data scientists, and analysts to gather...