Lead Data Platform Engineer

3 weeks ago


Bangalore, Karnataka, India Digitopcs consulting Full time

Job Description :

Primary responsibilities :

- Lead the design and engineering of a scalable, secure, and cost-optimized Streaming Data Platform.

- Lead role in evolving from (current) on-prem to hybrid cloud infrastructure with Kafka as the core technology.

- Lead platform upgrades and installs.

- Perform the procedures in Development.

- Collaborate with Operations team in production.

- Develop automation of platform components (infrastructure as code).

- Integrate platform metrics into the Syniverse monitoring stack.

- Execute DB performance analysis.

- Collaborate with Development teams to provide results and recommendations.

- This includes queries, DB schema design, and aggregation layers.

- Perform Capacity analysis to be applied directly in planning infrastructure design/purchases.

- Communicate results/recommendations to leadership team.

Qualifications :

- Bachelor's Degree in Computer Science or equivalent.

- 10+ years relevant experience managing Database, Analytics, and Streaming Platforms.

- 7+ years of programming with Java, python, or Go.

- 5+ years of Linux experience in both scripting and administrative role.

- 4+ years Kafka with a preference for Confluent platform:.

- Deploying and managing Brokers, Zookeeper, Schema Registry, ksqlDB, and Connectors.

- Setup mirroring of Kafka topics data across clusters, DCs, and Cloud.

Cloud skills :

- Production experience with Public Cloud.

- AWS strongly preferred.

- Production experience deploying Kafka in cloud and on-prem.

- Production experience deploying cloud native DB services in the cloud.

- Experience performing assessment of Cloud and on-prem data system.

- DBA experience with preference for PostgreSQL or similar relational database.

- Production experience tuning queries and optimizing performance on large scale Analytic DBs.

- Experience developing automation scripts for infrastructure and platform with Ansible strongly preferred.

- Demonstrates understanding of Data Model design and tradeoffs.

- Demonstrates ability to work across teams including Operations, Development, DBA, and Infrastructure admins.

- Demonstrates clear communication to peers and leadership.

Desired :

- Production experience with Impala (Cloudera) or Snowflake DB including data ingestion, schema design, and tuning.

- Experience setting up Grafana and Prometheus for metrics capture.

- Development of Stream processing applications using Confluent (ksqlDB and Kafka Streams).

- Experience in DBA role on TimescaleDB.

- Data pipeline experience using python, Spark Streaming, Streamsets, Kafka Connect, and ksqlDB.

- Experience deploying on Kubernetes using Containers.

- OLTP DB data model and workload tuning experience.

- Previous experience managing Cloudera CDH clusters.

- Logging using Logstash, Fluentd or equivalent

(ref:hirist.tech)
  • Solution Engineer

    3 weeks ago


    Bangalore, Karnataka, India HEVO DATA Full time

    Overview :. We're seeking a talented Solutions Engineer with strong background in Data Domain to join our growing Sales team. You'll work one-on-one with the sales team to help discover our customers' business and technical needs and demonstrate how Hevo's platform can solve their real-time data replication needs. In this role, you will have...


  • Bangalore, Karnataka, India MakeMyTrip Full time

    Our main focus is on Scala , Spark , Spark Streaming .MakeMyTrip - Lead Data EngineerAbout the Role :MakeMyTrip, India's leading online travel company, is searching for a seasoned Lead Data Engineer to join our Data Platform Team. You will play a pivotal role in architecting, developing, and managing our data infrastructure and pipelines, empowering...

  • Lead Data Engineer

    3 weeks ago


    Bangalore, Karnataka, India Rakuten Full time

    Senior Lead Data Engineer, will play a pivotal role in designing, developing, and maintaining data infrastructure. You will collaborate with cross-functional teams, and ensure the seamless integration of data solutions across multiple platforms. Your expertise in Python or Java, combined with hands-on experience in various data platforms, cloud DevOps, and...


  • Bangalore, Karnataka, India WIZSTAFFING PRIVATE LIMITED Full time

    Responsibilities :- Own the data platform at Fermat that powers many customer-facing and internal dashboards.- Build OLAP stack and data pipelines to support our reporting products.- Be responsible for data QA, testing, and debugging - answering the question "Why is this number different? "- Influence architecture, technology selections, and trends of the...

  • Senior Data Engineer

    4 months ago


    Bangalore, Karnataka, India Talent Corner HR Services Full time

    Job Description : Are you someone looking for a challenge to work in a startup environment and innovate in a global environment to help build a SaaS Service to scale? Are you willing to roll up your sleeves and do what it takes to deliver value to Customers?As we expand our Engineering operations in India, we are looking for a Sr. Data Engineer to join us...

  • Technical Lead

    3 weeks ago


    Bangalore, Karnataka, India FxConsulting Full time

    Job Title : Lead Data EngineerWe are seeking a highly skilled Lead Data Engineer with expertise in Big Data technologies to join our dynamic team. As a Lead Data Engineer, you will play a pivotal role in designing, building, and optimizing large-scale data pipelines and systems to drive business insights and innovation.Key Responsibilities :1. Design,...

  • Codex Tech

    2 days ago


    Bangalore, Karnataka, India CodeX Tech-IT LLC Full time

    About the Role : We are looking for a highly skilled Senior Data Platform Engineer to join our growing team. You will play a critical role in building and scaling our data platform to support complex data processing and analytics needs. As part of this role, you will lead the design and development of scalable systems that handle large volumes of batch and...


  • Bangalore, Karnataka, India CP360 INDIA PRIVATE LIMITED Full time

    Role : AI Lead EngineerJob Location : BengaluruShift Timing : EST Timing (IST - 06:30 PM - 03:30 AM)The candidate we are looking for :We are looking for an experienced AI Lead Engineer with a minimum of 6+ years of experience.Primary Responsibilities :- Partner with product managers and product owners to understand business needs and develop architecture...

  • Development Manager

    3 weeks ago


    Bangalore, Karnataka, India Talpro Full time

    Position : Development ManagerReports to : Director of EngineeringAbout Us : We are a dynamic tech company in Bangalore, passionate about building cutting-edge data analytics platforms. We're looking for a skilled Development Manager to lead our talented team of Platform and Big Data engineers.Your Role : As a Development Manager, you will spearhead the...

  • Aptus Data Labs

    3 weeks ago


    Bangalore, Karnataka, India Aptus Data Labs Full time

    Job Role : Lead Data ScientistExperience : 5-7 yrs.Location : Bangalore (Currently remote)As the Lead Data Scientist, you will play a crucial role in driving data-driven decision making within the organization. Your expertise in data analysis, machine learning, and statistical modelling will enable you to uncover insights, develop predictive models, and...

  • Lead Data Engineer

    3 weeks ago


    Bangalore, Karnataka, India Sentienz Full time

    Job Description : As a Lead Data Engineer, you will play a pivotal role in driving our data-driven initiatives. You will be responsible for designing, developing, and maintaining robust data engineering solutions that support our analytics and decision-making processes.Responsibilities :- Lead and mentor a team of data engineers, providing technical guidance...

  • Principal Architect

    2 weeks ago


    Bangalore, Karnataka, India yitrobc Full time

    Description :As a Platform Architect, you will be responsible for designing and implementing a robust, scalable, secure and cutting edge technology based healthcare data platform. Your role will involve working closely with cross-functional teams to ensure the platform meets industry standards and supports the efficient integration and management of...

  • Data Engineer

    3 weeks ago


    Bangalore, Karnataka, India TekPillar Full time

    Job Title : GCP Data EngineerLocation : BangaloreExperience : 3 to 6 yearsKey Skills :- Google Cloud Platform (GCP)- Apache Airflow- ETL (Extract, Transform, Load)- PythonJob Description :We are looking for a skilled Data Engineer with experience in GCP, Airflow, ETL processes, and Python to join our team in Bangalore. The ideal candidate will have 3 to 6...


  • Bangalore, Karnataka, India Talpro Full time

    Job Description :Key Responsibilities :Design and Development :- Architect, design, and develop scalable and high-performance data platforms.- Implement data ingestion, transformation, and processing pipelines using technologies such as Apache Kafka, Hadoop, Spark, and Flink.- Develop real-time data processing and stream processing applications.Data...


  • Bangalore, Karnataka, India Squareroot Consulting Pvt Ltd. Full time

    Overview :As the Sr Engineering Manager for our E-commerce Platform, you will lead a team of skilled engineers in designing, developing, and maintaining scalable and innovative solutions that power our e-commerce platform. Your role will be pivotal in driving technical excellence, fostering innovation, and ensuring the successful delivery of high-quality...

  • Data Engineer

    5 days ago


    Bangalore, Karnataka, India Aritha Consulting Services Pvt. Ltd. Full time

    Job Description :- Develops and maintains scalable data pipelines to support continuing increases in data volume and complexity.- Collaborates with analytics and business teams to improve data models that feed business intelligence tools, increasing data accessibility and fostering data-driven decision making across the organization.- Implements processes...

  • Lead Engineer

    3 weeks ago


    Bangalore, Karnataka, India Selsoftinc Full time

    Job Description :As a Cloud GCP Lead Engineer, you will lead the design, implementation, and management of Google Cloud Platform (GCP) infrastructure. You will collaborate with cross-functional teams to ensure that the cloud environment is optimized, secure, and scalable for the company's needs. The ideal candidate has a strong background in cloud...

  • Principal Engineer

    4 months ago


    Bangalore, Karnataka, India Astuto Technologies Private Limited Full time

    About AstutoWe are a Cloud FinOps startup focused on helping businesses save Cloud costs, avoid waste, improve efficiency, and optimize their RoI on Cloud Spend. Our vision is to build OneLens as one of the industry-leading Cloud FinOps platforms. We have the backing of a global institutional investor and raised pre-seed funding. Role Overview : We are...


  • Bangalore, Karnataka, India Coders Brain Technology Private Limited Full time

    Role : Lead Platform Engineer (Azure Kubernetes Services)Seeking a seasoned Lead Platform Engineer to drive the maintenance and expansion of our Azure Kubernetes Services (AKS) environment. This critical role is central to our IT services and cloud platform strategy, overseeing the architecture, design, development, implementation, and on-call production...


  • Bangalore, Karnataka, India HEVO DATA Full time

    About Hevo : Ever wondered, what's common between Zepto, Napster, Curefit, Groww, Jumio, Cars24 ? They all super-charge their business growth with Hevo Data Pipelines. How? 2 of every 3 modern day companies are unable to take data driven decision making due to fragmented data across 100s business software. Hevo is a simple, intuitive, and powerful...