Lead Data Engineer

4 days ago

Bengaluru, Karnataka, India The Scalers Full time ₹ 20,00,000 - ₹ 25,00,000 per year
Job Description

PEI Group is a subscriber-focused business intelligence company that focuses on private investment markets in real estate, infrastructure, private equity, private debt, and specialist sector-specific activities within private asset classes. We provide industry-leading journalism, data, and market insight to subscribing clients via a wide portfolio of specialist brands supported by our robust and scalable digital publishing, analytics, and database platform.

Since its inception in 2001, we have grown into a subscriber-focused business intelligence company with our multi-talented global team of over 400 people spread across EMEA, USA & Asia, and our purpose is to inform and connect investment professionals across global, specialised markets.

As a Lead Data Engineer, you will lead, design, implement, and maintain data processing pipelines and workflows using Databricks on the Azure platform. Your expertise in PySpark, SQL, Databricks, test-driven development, and Docker will be essential to the success of our data engineering initiatives.

Roles and Responsibilities:
  • Collaborate with cross-functional teams to understand data requirements and design scalable and efficient data processing solutions.
  • Develop and maintain data pipelines using PySpark and SQL on the Databricks platform.
  • Optimise and tune data processing jobs for performance and reliability.
  • Implement automated testing and monitoring processes to ensure data quality and reliability.
  • Work closely with data scientists, data analysts, and other stakeholders to understand their data needs and provide effective solutions.
  • Troubleshoot and resolve data-related issues, including performance bottlenecks and data quality problems.
  • Stay up to date with industry trends and best practices in data engineering and Databricks.
Key Requirements:
  • 8 years of experience as a Data Engineer, with a focus on Databricks and cloud-based data platforms, with a minimum of 4 years of experience in writing unit/end-to-end tests for data pipelines and ETL processes on Databricks.
  • Hands-on experience in PySpark programming for data manipulation, transformation, and analysis.
  • Strong experience in SQL and writing complex queries for data retrieval and manipulation.
  • Experience in Docker for containerising and deploying data engineering applications is good to have.
  • Strong knowledge of the Databricks platform and its components, including Databricks notebooks, clusters, and jobs.
  • Experience in designing and implementing data models to support analytical and reporting needs will be an added advantage.

Locations - Bangalore, Karnataka, India

  • Data Engineer

    2 weeks ago

    Bengaluru, Karnataka, India Data Analytics Full time ₹ 15,00,000 - ₹ 25,00,000 per year

    Get to Know the Team Grab has been expanding into multiple lines of businesses. The Grab Data Warehouse team builds business intelligence systems using data from all business lines to provide insights to key stakeholders across the company. Get to Know the Role We are looking for a Lead Data Engineer to lead the design and enhancement of our data...

  • Data Engineer

    4 days ago

    Bengaluru, Karnataka, India NTT DATA Full time ₹ 15,00,000 - ₹ 25,00,000 per year

    Design and implement tailored data solutions to meet customer needs and use cases, spanning from streaming to data lakes, analytics, and beyond within a dynamically evolving technical stack. Provide thought leadership by recommending the most appropriate technologies and solutions for a given use case, covering the entire spectrum from the application layer...

  • Lead Data Scientist

    4 weeks ago

    Bengaluru, Karnataka, India NTT DATA Full time

    NTT DATA strives to hire exceptional innovative and passionate individuals who want to grow with us If you want to be part of an inclusive adaptable and forward-thinking organization apply now We are currently seeking a Lead Data Scientist - Computer Vision Generative AI to join our team in Bangalore Karn taka IN-KA India IN We are seeking a...

  • Lead Data Scientist

    1 week ago

    Bengaluru, Karnataka, India NTT DATA Full time ₹ 12,00,000 - ₹ 36,00,000 per year

    NTT DATA strives to hire exceptional, innovative and passionate individuals who want to grow with us. If you want to be part of an inclusive, adaptable, and forward-thinking organization, apply now. We are currently seeking a Lead Data Scientist - Computer Vision & Generative AI to join our team in Bangalore, Karnātaka (IN-KA), India (IN). We are seeking...

  • Lead Data Scientist

    2 weeks ago

    Bengaluru, Karnataka, India Aptus Data Labs Full time US$ 1,25,000 - US$ 1,75,000 per year

    Job Description: Senior/lead Data ScientistLocation: Bangalore/WFHEmployment Type: Full-timeAbout the RoleWe are seeking a highly skilled Senior /lead Data Scientist with strong expertise in Artificial Intelligence (AI), Machine Learning (ML), Deep Learning, and Natural Language Processing (NLP). The ideal candidate will also have hands-on exposure to...

  • Lead Data Scientist

    4 days ago

    Bengaluru, Karnataka, India NTT DATA, Inc. Full time ₹ 12,00,000 - ₹ 36,00,000 per year

    We are currently seeking a Lead Data Scientist - Computer Vision & Generative AI to join our team in Bangalore, Karntaka (IN-KA), India (IN).We are seeking a highly skilled and experienced Lead Data Scientist with 7+ years of expertise in Machine Learning, Deep Learning (especially Computer Vision), and Generative AI. The ideal candidate will lead the design...

  • Lead Data Scientist

    2 weeks ago

    Bengaluru, Karnataka, India Enable Data Incorporated Full time ₹ 20,00,000 - ₹ 25,00,000 per year

    Enable Data Incorporated is looking for a talented and experienced Lead Data Scientist to join our innovative team. In this role, you will be responsible for driving data science initiatives, developing predictive models, and providing insights that guide key business decisions. The ideal candidate has a strong background in statistical analysis, machine...

  • Digital Engineering Lead Engineer

    4 weeks ago

    Bengaluru, Karnataka, India NTT DATA Full time

    Req ID 299336NTT DATA strives to hire exceptional innovative and passionate individuals who want to grow with us If you want to be part of an inclusive adaptable and forward-thinking organization apply now We are currently seeking a Digital Engineering Lead Engineer to join our team in Bangalore Karn xc4 x81taka IN-KA India IN How We Will Help...

  • Data Engineer

    4 days ago

    Bengaluru, Karnataka, India NTT DATA, Inc. Full time ₹ 15,00,000 - ₹ 25,00,000 per year

    Key Responsibilities:Design and implement tailored data solutions to meet customer needs and use cases, spanning from streaming to data lakes, analytics, and beyond within a dynamically evolving technical stack.Provide thought leadership by recommending the most appropriate technologies and solutions for a given use case, covering the entire spectrum from...

  • Data Engineer

    4 days ago

    Bengaluru, Karnataka, India NTT DATA, Inc. Full time ₹ 1,04,000 - ₹ 1,30,878 per year

    Req ID:321800We are currently seeking a Data Engineer (Talend &Pyspark) to join our team in Bangalore, Karntaka (IN-KA), India (IN)."Job Duties: Key Responsibilities: Design and implement tailored data solutions to meet customer needs and use cases, spanning from streaming to data lakes, analytics, and beyond within a dynamically evolving technical...