Team Lead-Advanced Data Systems Delivery

Team Lead – Advanced Data Systems Delivery

Location:
Pune

Employment Type:
Full-Time

Department:
Delivery

Position Overview
:

The Team Lead for Advanced Data Systems Delivery will lead a team responsible for

delivering high-performance, scalable data solutions, with a focus on designing and

optimizing data architectures for mission-critical Tier 1 systems. This role requires deep

expertise in deploying real-time and batch processing systems, utilizing the ArcOne AI

platform to architect scalable topologies, tune critical components, and ensure

compliance with NFRs and SLAs, while maintaining a strong foundation in data

architecture principles.

Key Responsibilities:

Project Leadership & Delivery:

• Lead the end-to-end delivery of advanced data systems for mission-critical Tier 1

applications, focusing on revenue management and other high-impact use cases.

• Ensure solutions meet NFRs (e.g., scalability, performance, availability, security)

and SLAs, delivering on time, within scope, and budget.

• Utilize the ArcOne AI platform to design, scale, and tune system topologies to

support complex real-time and batch workloads.

Data Architecture Leadership:

• Architect and oversee the implementation of robust, scalable, and secure data

architectures to support mission-critical applications.

• Design data models, schemas, and storage solutions (e.g., data lakes, data

warehouses, NoSQL databases) optimized for performance, scalability, and

accessibility.

• Implement and optimize ETL/ELT pipelines, ensuring efficient data ingestion,

transformation, and integration across distributed systems.

• Ensure data governance, integrity, and security standards are maintained across

all data architectures.

Team Management:

• Manage, mentor, and inspire a team of data engineers, architects, and

performance engineers, fostering a culture of technical excellence and

collaboration.

• Provide technical guidance on data architecture, advanced data systems, and

system optimization, ensuring alignment with industry best practices.

• Conduct performance reviews, set team goals, and support career development

for team members.

Technical Expertise:

• Oversee the deployment of high-scale, mission-critical systems handling real-time

(e.g., streaming, event-driven) and batch processing workloads.

• Leverage the ArcOne AI platform to design and scale system topologies, tuning

components (e.g., compute, storage, network) to meet performance and

scalability requirements.

• Optimize data processing frameworks (e.g., Apache Kafka, Spark, Hadoop, Flink)

and databases (e.g., SQL, NoSQL) for low latency, high throughput, and reliability.

• Lead investigations into performance regressions, scalability issues, or system

failures, ensuring rapid resolution and root cause analysis.

Stakeholder Collaboration:

• Act as the primary point of contact for stakeholders, translating business

requirements into technical solutions with a focus on robust data architectures.

• Collaborate with infrastructure, platform, AI, and product teams to align on

system design, performance, and data architecture goals.

• Communicate project progress, risks, and performance metrics to senior

leadership and clients.

System Design & Optimization:

• Design and implement data architectures that support scalability, fault tolerance,

and high availability for Tier 1 applications.

• Optimize real-time (e.g., Apache Kafka, Flink) and batch processing (e.g., Spark,

Hadoop) workloads for high-scale environments.

• Ensure data pipelines are optimized for performance, leveraging the ArcOne AI

platform to automate and enhance topology scaling and component tuning.

• Advocate for simplicity and rigor in system and data architecture design to

address complex performance and scalability challenges.

Process & Standards:

• Define and enforce best practices for advanced data systems delivery, data

architecture, CI/CD pipelines, and performance testing strategies.

• Develop and monitor SLAs/SLOs for latency, throughput, data availability, and

system reliability, ensuring compliance with mission-critical standards.

• Stay updated on industry trends in advanced data systems, data architecture, and

AI technologies to drive innovation within the team.

Qualifications:

Education: Bachelor's or Master's degree in Computer Science, or a related field.

Experience:

• 7+ years of experience in delivering advanced data systems or similar solutions,

with at least 3 years in a leadership or team lead role.

• Proven track record of deploying high-scale, mission-critical Tier 1 systems with

real-time and batch workloads.

• Extensive experience in designing and implementing data architectures for large-scale, distributed systems, including data modeling, ETL/ELT pipelines, and

storage solutions (e.g., data lakes, data warehouses).

• Hands-on expertise with the ArcOne AI platform or similar AI-driven tools to

design, scale, and tune system topologies.

• Deep knowledge of performance tuning for data processing frameworks (e.g.,

Apache Kafka, Spark, Hadoop, Flink) and databases (e.g., SQL, NoSQL, data lakes).

• Strong understanding of NFRs (e.g., scalability, availability, performance, security)

and SLAs for Tier 1 systems.

• Experience in revenue management or similar domains (e.g., pricing, forecasting,

optimization) is a plus.

Technical Skills:

• Proficiency in programming languages such as Python, Java, or Scala.

• Expertise in data processing technologies (e.g., Apache Kafka, Spark, Hadoop,

Flink) and cloud platforms (e.g., AWS, Azure, GCP).

• Strong experience with data architecture tools and platforms, including data

modeling (e.g., ERD, dimensional modeling), data lakes (e.g., Delta Lake, Iceberg),

and data warehouses (e.g., Databricks, Dremio, DuckDB, Trino).

• Proficiency with performance profiling tools (e.g., perf, eBPF) and observability

systems (e.g., Prometheus, Grafana, OpenTelemetry).

• Deep understanding of OS internals, networking, storage, and compute

optimization (e.g., GPU utilization, memory management).

• Experience with containerization (e.g., Docker, Kubernetes) and data

orchestration tools (e.g., Airflow, Dagster).

• Knowledge of AI/ML frameworks (e.g., TensorFlow, PyTorch) for optimizing data-driven solutions is a plus.

Soft Skills
:

• Exceptional leadership, communication, and stakeholder management skills.

• Strong problem-solving abilities with a focus on simplicity, rigor, and

collaboration.

• Ability to navigate ambiguity and drive alignment across cross-functional teams.

Preferred:

• Experience with the ArcOne AI platform for system design and optimization.

• Familiarity with agile methodologies and tools like Jira or Trello.

• Certification in cloud platforms (e.g., AWS Certified Solutions Architect), data

engineering, or data architecture (e.g., TOGAF, DAMA-DMBOK).