Senior Data Engineer

16 hours ago

Mumbai, Maharashtra, India beBeeData Full time US$ 1,04,000 - US$ 1,30,878
Job Title: Azure Databricks Lead

As a senior data engineer, you will play a crucial role in designing and developing scalable data pipelines using Azure Databricks and PySpark/Scala.

Our ideal candidate has experience architecting and building data platforms on Azure and is proficient in Azure Databricks, PySpark, and SQL. You will work closely with data engineers and architects to ensure efficient data movement and transformation.

Key Responsibilities:

  • Design and Develop Data Pipelines: Create scalable data processing pipelines using Azure Databricks and PySpark/Scala, implement ETL (Extract, Transform, Load) processes to ingest, transform, and load data from various sources.
  • Data Quality Implementation: Establish data quality checks and validation rules within Azure Databricks, monitor data quality metrics, and address anomalies promptly.
  • Unity Catalog Integration: Leverage Azure Databricks Unity Catalog to manage metadata, tables, and views, integrate Databricks assets seamlessly with other Azure services.
  • Delta Lake Expertise: Understand and utilize Delta Lake, which provides ACID transactions and time travel capabilities on top of data lakes, implement Delta Lake tables for reliable data storage and versioning.
  • Performance Tuning and Query Optimization: Profile and analyze query performance, optimize SQL queries, Spark jobs, and transformations for efficiency.
  • Resource Optimization: Manage compute resources effectively within Azure Databricks clusters, scale clusters dynamically based on workload requirements.
  • Source System Integration: Integrate Azure Databricks with various source systems, ensure seamless data ingestion and synchronization.

Qualifications and Skills:

  • Education: Bachelor's degree in Computer Science, Information Technology, or a related field.
  • Experience: Minimum 5-6 years of relevant experience architecting and building data platforms on Azure.
  • Certifications: Microsoft Certified: Azure Data Engineer Associate or similar.
  • Soft Skills: Strong problem-solving abilities, excellent communication and collaboration skills, ability to work in a fast-paced, agile environment.

  • Senior Data Engineer

    4 weeks ago

    Mumbai, Maharashtra, India Periscope-Tech Full time

    We're seeking a Senior Data Engineer to join our data engineering team. In this role, you'll be responsible for maintaining and extending existing data pipelines and partnering with our implementation team to design and deliver custom ETL workflows for new customers. This hands-on role requires strong coding skills, experience with modern data tools, and...

  • Senior Data Engineer

    2 weeks ago

    Mumbai, Maharashtra, India Straive Full time

    Dear Candidate,Straive is hiring for Senior Data Engineer, Interested candidate can apply to below link. Description:-Design- Analyze relevant internally and externally sourced data (raw data) to generate BI and Advanced Analytics- datasets based on your stakeholders' requirements- Design data pipelines to curate sourced data into the inhouse data warehouse-...

  • Senior Data Engineer

    4 weeks ago

    Mumbai, Maharashtra, India Straive Full time

    Dear Candidate, Straive is hiring for Senior Data Engineer, Interested candidate can apply to below link. Job Description:- Design Analyze relevant internally and externally sourced data (raw data) to generate BI and Advanced Analytics datasets based on your stakeholders' requirements Design data pipelines to curate sourced data into the inhouse data...

  • Senior Data Engineering Lead

    17 minutes ago

    Mumbai, Maharashtra, India beBeeDataEngineering Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

    Are you a seasoned leader in data engineering seeking a new challenge? Do you have the expertise to drive innovation and deliver complex projects on time? We are looking for an exceptional Data Engineering Director to join our team.Key Responsibilities:Lead and manage a high-performing team of data engineers, providing mentorship and technical guidance to...

  • Senior Data Engineer

    1 day ago

    Mumbai, Maharashtra, India HealthyMe Ai Full time

    About the RoleWe are seeking a Senior Data Engineer with deep expertise in building data pipelines and integrating healthcare data. You will report to the Head of AI and Engineering and be responsible for designing, building, and operating the backbone of our AI-powered applications. This is a critical role where you will create scalable pipelines that...

  • Senior Data Engineer

    3 weeks ago

    Navi Mumbai, Maharashtra, India Celebal Technologies Full time

    Dear Applicants,Please find following Job Description for Senior Data Engineer role at Celebal Technologies, Please let us know your level of comfort along with most recent copy of your resume. You can directly apply or share your details at shailendra.sharma@celebaltech.comDesignation – Senior Data EngineerExperience – 4 -9 YearsLocation Navi Mumbai...

  • Senior Data Engineer

    3 weeks ago

    Navi Mumbai, Maharashtra, India Celebal Technologies Full time

    Dear Applicants, Please find following Job Description for Senior Data Engineer role at Celebal Technologies, Please let us know your level of comfort along with most recent copy of your resume. You can directly apply or share your details at Designation – Senior Data Engineer Experience – 4 -9 Years Location Navi Mumbai (onsite) Job Summary: We...

  • Data Engineering Director

    3 weeks ago

    Mumbai, Maharashtra, India eClerx Full time

    Job Title: Senior Data Engineering Lead (Databricks)Company Overview:At eClerx, we are a leading IT firm specializing in innovative technologies and solutions that drive business transformation. Leveraging expertise in business process management, advanced analytics, and smart automation, we empower our clients to achieve operational excellence and...

  • Senior Data Engineering Expert

    1 week ago

    Mumbai, Maharashtra, India beBeeDataEngineering Full time US$ 90,000 - US$ 1,20,000

    Job Title: Senior Data Engineering ExpertThe successful candidate will be responsible for designing, developing, and maintaining large-scale data systems to support business growth.Required Skills and QualificationsTechnical Skills:Proficiency in Data Bricks Lakehouse Platform, Delta Lake, Genie, ML Flow.Expertise in SQL and NoSQL databases including MySQL,...

  • Senior Data Migration Engineer

    3 days ago

    Mumbai, Maharashtra, India Oracle Full time US$ 1,20,000 - US$ 1,50,000 per year

    Senior Data Migration Engineer About Oracle FSGIU - Finergy: The Finergy division within Oracle FSGIU is dedicated to the Banking, Financial Services, and Insurance (BFSI) sector. We offer deep industry knowledge and expertise to address the complex financial needs of our clients. With proven methodologies that accelerate deployment and...