Lead Data Platform Engineer

2 months ago

Chennai, India Digitopcs consulting Full time

Job Description :


- Lead the design and engineering of a scalable, secure, and cost-optimized Streaming Data Platform

- Lead role in evolving from (current) on-prem to hybrid cloud infrastructure with Kafka as the core technology

- Lead platform upgrades and installs. Perform the procedures in Development. Collaborate with Operations team in production.

- Develop automation of platform components (infrastructure as code)

- Integrate platform metrics into the Syniverse monitoring stack

- Execute DB performance analysis. Collaborate with Development teams to provide results and recommendations. This includes queries, DB schema design, and aggregation layers.

- Perform Capacity analysis to be applied directly in planning infrastructure design/purchases

- Communicate results/recommendations to leadership team

Qualifications :

- Bachelor's Degree in Computer Science or equivalent

- 10+ years relevant experience managing Database, Analytics, and Streaming Platforms.

- 7+ years of programming with Java, python, or Go

- 5+ years of Linux experience in both scripting and administrative role

- 4+ years Kafka with a preference for Confluent platform :

1. Deploying and managing Brokers, Zookeeper, Schema Registry, ksqlDB, and Connectors

2. Setup mirroring of Kafka topics data across clusters, DCs, and Cloud

- Cloud skills:

1. Production experience with Public Cloud. AWS strongly preferred.

2. Production experience deploying Kafka in cloud and on-prem

3. Production experience deploying cloud native DB services in the cloud

4. Experience performing assessment of Cloud and on-prem data system

- DBA experience with preference for PostgreSQL or similar relational database

- Production experience tuning queries and optimizing performance on large scale Analytic DBs

- Experience developing automation scripts for infrastructure and platform with Ansible strongly preferred

- Demonstrates understanding of Data Model design and tradeoffs

- Demonstrates ability to work across teams including Operations, Development, DBA, and Infrastructure admins

- Demonstrates clear communication to peers and leadership

Desired :

- Production experience with Impala (Cloudera) or Snowflake DB including data ingestion, schema design, and tuning

- Experience setting up Grafana and Prometheus for metrics capture

- Development of Stream processing applications using Confluent (ksqlDB and Kafka Streams)

- Experience in DBA role on TimescaleDB

- Data pipeline experience using python, Spark Streaming, Streamsets, Kafka Connect, and ksqlDB

- Experience deploying on Kubernetes using Containers

- OLTP DB data model and workload tuning experience

- Previous experience managing Cloudera CDH clusters

- Logging using Logstash, Fluentd or equivalent

(ref:hirist.tech)

  • Lead Consultant

    5 months ago

    Chennai, India AstraZeneca Full time

    Job Title: Lead Consultant - Data Platforms Career Level - E Introduction to role Are you ready to disrupt an industry and change lives? AstraZeneca is seeking a Lead Consultant - Data Platforms to join our dynamic IT team. This role is part of the Data, Analytics, and AI (DAAI) team within the Solution and Service Delivery (SSD) arm of Commercial...

  • Data Platform Engineering Director

    1 month ago

    Chennai, India ValGenesis India P Ltd Full time

    Job Title: Data Platform Engineering DirectorWe are seeking an experienced Data Platform Engineering Director to lead our cloud data platform engineering team. The ideal candidate will have a strong background in designing and implementing scalable and highly performant enterprise cloud products.Key Responsibilities:Lead the design and architecture of our...

  • Data Platform Engineering Director

    3 weeks ago

    Chennai, Tamil Nadu, India ValGenesis Full time

    Data Platform Engineering DirectorWe are seeking an experienced Data Platform Engineering Director to lead our cloud data platform engineering team. The successful candidate will have a proven track record in designing and implementing scalable and highly performant enterprise cloud data platforms.Key Responsibilities:Lead the design and architecture of our...

  • Data Platform Engineering Director

    4 weeks ago

    Chennai, India ValGenesis India P Ltd Full time

    Job Title: Data Platform Engineering DirectorValGenesis India P Ltd is seeking a highly experienced Data Platform Engineering Director to lead our cloud data platform engineering team. As a key member of our technology leadership team, you will be responsible for designing, implementing, and maintaining our data platform infrastructure.Key...

  • Data Platform Engineering Leader

    3 weeks ago

    Chennai, Tamil Nadu, India ValGenesis Full time

    Lead Our Data Platform Engineering EffortsWe are seeking a seasoned Cloud Data Platform Director to drive the strategic direction of our data platform infrastructure. As a key member of our engineering team, you will oversee the design, implementation, and deployment of data storage, processing, and integration components. Your expertise in high-trust cloud...

  • Senior Software Engineering Lead

    1 month ago

    Chennai, Tamil Nadu, India Crayon Data Full time

    Role: Senior Software Engineering LeadCrayon Data, a leading provider of AI-led revenue acceleration solutions, is seeking a seasoned Senior Software Engineering Lead to drive the development of our flagship platform, maya.ai. As a key member of our engineering team, you will be responsible for leading the design and implementation of scalable, flexible, and...

  • Data Engineer

    3 weeks ago

    Chennai, Tamil Nadu, India NTT DATA Full time

    Job SummaryWe are seeking a highly skilled and experienced Data Engineer to join our team in Chennai, Tamil Nādu, India. The ideal candidate will have expertise in designing, implementing, and supporting Cloud Data Management and Advanced Analytics platforms.Main ResponsibilitiesDesign and implement Cloud Data Management and Advanced Analytics...

  • Data Platform Engineer

    4 weeks ago

    Chennai, Tamil Nadu, India 2coms Full time

    Job Title: Data Platform EngineerJob Description:We are seeking a highly skilled Data Platform Engineer to join our team at 2COMS Group. As a Data Platform Engineer, you will be responsible for assisting with the data platform blueprint and design, collaborating with Integration Architects and Data Architects to ensure cohesive integration between systems...

  • Data Engineer

    3 weeks ago

    Chennai, India NTT DATA Full time

    Job Description Req ID: 298625 NTT DATA strives to hire exceptional, innovative and passionate individuals who want to grow with us. If you want to be part of an inclusive, adaptable, and forward-thinking organization, apply now.We are currently seeking a Data Engineer to join our team in Chennai, Tamil Nādu (IN-TN), India (IN).Data EngineerJob SummaryWe...

  • Big Data Platform Technical Lead

    2 weeks ago

    Chennai, Tamil Nadu, India 1HResource Solutions Full time

    Job Title: Senior Data Engineer for Large-Scale Data IntegrationJob Description: We are seeking an experienced Senior Data Engineer to lead our big data platform initiatives. The successful candidate will be responsible for designing and implementing large-scale data integration projects using big data technologies such as Hadoop, Spark, and Kafka.Key...

  • Data Engineer

    4 weeks ago

    Chennai, Tamil Nadu, India Poshmark Full time

    Data Engineering RoleThe Big Data team plays a vital role in Poshmark’s organization. Our mission is to create a world-class big data platform that brings value to data for us and our customers. We aim to democratize data, support business growth, provide self-service reporting and analytics tools, and fuel business-critical initiatives.The Data...

  • Technical Lead

    3 weeks ago

    Chennai, Tamil Nadu, India Crayon Data Full time

    Crayon Data Engineering Manager Role OverviewCrayon Data is a leading provider of AI-led revenue acceleration solutions, headquartered in Singapore with a local presence in India and the UAE. We're seeking an experienced Engineering Manager to lead our software development team in delivering high-quality, scalable, and secure platforms.Key...

  • Cloud Data Platform Engineering Leader

    1 month ago

    Chennai, Tamil Nadu, India ValGenesis Full time

    Job Title: Cloud Data Platform Engineering LeaderWe are seeking an experienced Cloud Data Platform Engineering Leader to join our team at ValGenesis. As a key member of our cloud product engineering team, you will be responsible for leading the design and implementation of our data platform infrastructure.Key Responsibilities:Design and implement scalable...

  • Director, Data Platform Engineering

    2 months ago

    Chennai, India ValGenesis Full time

    We are looking for experienced cloud data platform engineering/development leader who could join our cloud product engineering team to lead the data platform of ValGenesis’s next generation applications. If you are a technology leader and have passion to develop highly scalable and highly performant enterprise cloud products, we are eager to discuss with...

  • Data Platform Engineer

    4 weeks ago

    Chennai, Tamil Nadu, India 2coms Full time

    Job Opportunity for Data ProfessionalsAt 2COMS Group, we are seeking a skilled Data Platform Engineer to assist with the development of our data platform blueprint and design. This role involves collaborating with Integration Architects and Data Architects to ensure seamless integration between systems and data models.Key Responsibilities:1. Assist with the...

  • Data Platform Architect

    1 month ago

    Chennai, Tamil Nadu, India Gainwell Technologies Full time

    SummaryWe are seeking a highly skilled Data Platform Architect to lead our product development team in designing and implementing a cloud-based data platform architecture that will power BI and advanced analytics at Gainwell Technologies. The ideal candidate will have industry-leading programming skills and established knowledge of implementing, designing,...

  • Data Engineer

    4 weeks ago

    Chennai, India NTT DATA Services Full time

    Req ID: 298625 NTT DATA strives to hire exceptional, innovative and passionate individuals who want to grow with us. If you want to be part of an inclusive, adaptable, and forward-thinking organization, apply now.We are currently seeking a Data Engineer to join our team in Chennai, Tamil Nādu (IN-TN), India (IN).Data EngineerJob SummaryWe are looking for...

  • Data Engineer Lead

    3 weeks ago

    Chennai, Tamil Nadu, India Bounteous Full time

    About the RoleWe are seeking a highly skilled Data Engineer Lead to join our team at Bounteous x Accolite. As a Data Engineer Lead, you will be responsible for designing, developing, and maintaining large-scale data systems and architectures.Key ResponsibilitiesCollaborate with project managers and senior architects to evaluate data systems and project...

  • Eucloid - Lead Data Engineer - Python/Azure Databricks

    6 months ago

    Chennai, India Eucloid Data Solutions Pvt Ltd Full time

    Role : Lead Data Engineer (Databricks) About Eucloid : Eucloid Data Solutions is a fast-growing AI-consulting firm that enables business growth for its clients through advanced Data Analytics, Machine Learning and Data engineering. Eucloid offers bespoke, full stackand datadriven approach to solve highly complex and large-scale business problems for some of...

  • Data Engineering Lead

    3 weeks ago

    Chennai, Tamil Nadu, India Databricks Full time

    We are seeking an experienced Data Engineering Lead to join our Professional Services team at Databricks. As a key member of our team, you will work closely with customers to deliver short to medium-term customer engagements on their big data challenges using the Databricks platform.Your responsibilities will include providing data engineering, data science,...