Technical Lead: Open Source Data Engineering Platform

6 days ago

Pune, India BAJAJ FINSERVE Full time

Role & responsibilities

We are looking for an accomplished Technical Lead - Data Engineering to design and develop of a cloud-agnostic, open-source data ingestion and processing solution that will be embedded within a larger Low-Code platform to strengthen the overall low code platform capabilities.

The ideal candidate will bring deep expertise in large-scale Data Engineering platforms design, development and execution experience, with hands-on experience in ETL, Real Time Streaming, Distributed Data Processing, Python, Spark, and will have the ability to guide a product development team through the entire lifecycle - from concept to deployment.

This role involves architecting and delivering a robust, scalable, and future-ready data ingestion and processing framework, leveraging strong technical proficiency in Open Source Softwares along with modern data engineering best practices.

Preferred candidate profile

Experience Required:

  • Overall 8+ Years of experience in Data Engineering domain
  • Ability to lead the team to develop a cloud-agnostic, open-source data engineering platform
  • Hands on Architecture, Design and Programming skills to develop an Open Source based Data Engineering Application and Framework within a larger Enterprise Application product suite.
  • Strong Hands On experience in Python, Spark, API and preferably with other open source product like Apache Nifi etc.
  • Hands on experience in Java is preferrable
  • Strong experience in ETL/ELT pipelines, data streaming, CDC etc.
  • Design and implement real-time data streaming solutions as a part of product feature
  • Design and develop robust data quality assurance, failure logging features
  • Design and develop Monitoring, diagnostic and logging features to maintain production and non-production systems proactively
  • Develop automation scripts/tools (Python, Perl, Shell) to eliminate manual tasks and accelerate issue resolution
  • Engage with stakeholders to continuously enhance the product
  • Capture user feedback and translate it into actionable architecture or support initiatives.
  • Maintain structured documentation, including performance metrics and enhancement logs.

  • Technical Lead: Open Source Data Engineering Platform

    1 week ago

    Pune, Maharashtra, India BAJAJ FINSERVE Full time ₹ 5,00,000 - ₹ 8,00,000 per year

    Role & responsibilitiesWe are looking for an accomplished Technical Lead - Data Engineering to design and develop of a cloud-agnostic, open-source data ingestion and processing solution that will be embedded within a larger Low-Code platform to strengthen the overall low code platform capabilities.The ideal candidate will bring deep expertise in large-scale...

  • Data Platform Engineer

    2 weeks ago

    Pune, Maharashtra, India beBeeDataEngineer Full time ₹ 5,00,000 - ₹ 8,00,000

    About the RoleThis is an opportunity to work with a leading company in the field of data engineering. We are seeking a talented professional to support, develop and maintain our data and analytics platform.Key ResponsibilitiesThe successful candidate will be responsible for implementing and automating the deployment of our distributed system for ingesting...

  • Open Source Advocate

    2 weeks ago

    Pune, Maharashtra, India beBeeDeveloper Full time US$ 90,000 - US$ 1,20,000

    As a pioneering figure in the realm of Developer Relations, you will play a pivotal role in bridging the gap between our organization and the open-source community.We are seeking an accomplished Developer Relations Engineer to spearhead our efforts in engaging directly with open-source communities and developer-centric organizations. This individual will be...

  • Data Platform Lead

    3 weeks ago

    Pune, Maharashtra, India Kalyani Technologies Full time

    About UsWe are building a state-of-the-art, on-premise data platform from the ground up to revolutionize how our organization leverages data. Inspired by a unified and modern approach, our mission is to create a single, cohesive ecosystem that eliminates silos between data engineering, business analytics, and machine learning. We are looking for a key...

  • Vice President

    1 week ago

    Pune, India Valcon USA Full time

    An Ecommerce Startup is looking for a VP IT who can start everything from Scratch .ROLES & RESPONSIBILITIES :- Responsible for delivering products with high quality, on time while maintaining high efficiency of resources utilized.- Identify opportunities and risks for delivering company's services- Evaluate and identify appropriate technology platforms-...

  • Technical Lead

    6 days ago

    Pune, India NCS Group Full time

    1 Designing and implementing data ingestion pipelines from multiple sources using Azure Databricks 2 Ensure data pipelines run smoothly and efficiently 3 Developing scalable and re-usable frameworks for ingesting of data sets 4 Integrating the end-to-end data pipeline - to take data from source systems to target data repositories ensuring the quality and...

  • Data Engineer

    3 weeks ago

    Pune, Maharashtra, India aQb Solutions Pvt Ltd Full time

    We are looking for a highly skilled Data Engineer with strong expertise in PostgreSQL and hands-on experience in building data warehouses and data lakes. In this role, you will design and implement a scalable PostgreSQL-based data warehouse and manage a data lake on AWS infrastructure, leveraging primarily open-source technologies.The ideal candidate has a...

  • Technical Lead – Data Engineering

    1 week ago

    Pune, India Zywave Full time

    Job Description – Technical Lead (Data Engineering, TurboRater RQR: CPQ Rating)Location: Pune, IndiaWork Mode: 5 Days Work from OfficeAbout the RoleZywave is seeking a Technical Lead – Data Engineering (TurboRater RQR: CPQ Rating) with expertise in Snowflake, SQL Server, and modern data architecture principles including Medallion Architecture and...

  • Technical Lead – Data Engineering

    3 weeks ago

    Pune, Maharashtra, India Zywave Full time

    Job Description – Technical Lead (Data Engineering, TurboRater RQR: CPQ Rating)Location: Pune, IndiaWork Mode: 5 Days Work from OfficeAbout the RoleZywave is seeking a Technical Lead – Data Engineering (TurboRater RQR: CPQ Rating) with expertise in Snowflake, SQL Server, and modern data architecture principles including Medallion Architecture and Data...

  • Technical Lead

    1 week ago

    Pune, India Zywave Full time

    Job Description – Technical Lead (Data Engineering, TurboRater RQR: CPQ Rating) Location: Pune, India Work Mode: 5 Days Work from Office Zywave is seeking a Technical Lead – Data Engineering (TurboRater RQR: CPQ Rating) with expertise in Snowflake, SQL Server, and modern data architecture principles including Medallion Architecture and Data...