Senior Observability Engineer

Job Description

Senior Observability Engineer u00A0

Location:

Eli Lilly, Hyderabad

Position Type:

Full-Time

Level:

P2u2013P3

Note: Roles are posted at the lowest level of a band however, employees should search across all levels of the band to identify all opportunities. Employees hired on banded positions (e.g., P1u2013P3, R1u2013R2, B1u2013B3, etc.) transfer at their current level, despite the level indicated on the job posting.

Observability Engineer u2013 Tech@Lilly

We are seeking a skilled and motivated SeniorObservability Engineer to join our growing team. This role is critical to enhancing operational visibility, enabling proactive issue detection, and improving system reliability across Lillyu2019s digital ecosystem. You will work with cutting-edge observability tools such as Splunk Cloud, Splunk IT Service Intelligence (ITSI), and OpenTelemetry, collaborating with cross-functional teams to build robust, scalable monitoring solutions.

Key Responsibilities

Data Ingestion & Integration

- Design, implement, and manage data ingestion pipelines from diverse sources into Splunk Cloud.
- Ensure data quality, efficiency, and compliance across ingestion workflows.
- Implement and manage Open Telemetry for standardized collection of traces, metrics, and logs.

Splunk ITSI Configuration

- Develop and maintain ITSI components including:

- Service Tree Design: Map IT infrastructure and applications into logical service trees reflecting business criticality.

- KPI Creation: Define and refine KPIs to measure service health and performance.

- Glass Table Development: Build intuitive dashboards for real-time operational insights.

- Anomaly Detection: Configure machine learning-based anomaly detection and intelligent thresholds.

Performance Optimization

- Tune Splunk Cloud and ITSI for optimal performance, including search execution and dashboard responsiveness.
- Implement Logs as a Code (LaaC) practices for managing logging configurations and parsing rules in a version-controlled, automated manner.

Troubleshooting & Support

- Provide expert-level support for Splunk and ITSI-related issues, including data discrepancies and performance bottlenecks.
- Collaborate with SREs, developers, and operations teams to translate observability needs into actionable solutions.

Collaboration & Documentation

- Work closely with application and infrastructure teams to align observability strategies with business goals.
- Maintain comprehensive documentation and promote observability best practices across the organization.

Qualifications

- Bacheloru2019s degree in Computer Science, Information Technology, or a related field, or equivalent practical experience.
- 3u20136 years of hands-on experience with Splunk Cloud and/or Splunk ITSI in enterprise environments.
- Strong proficiency in Splunk SPL, including advanced queries, macros, and data model acceleration.
- Experience with Open Telemetry instrumentation and observability pipelines.
- Familiarity with scripting languages (e.g., Python, Bash) for automation and API integrations.
- Understanding of cloud platforms (AWS, Azure, GCP) and container technologies (Docker, Kubernetes) is a plus.

Preferred Certifications

- Splunk Core Certified User or Administrator
- Splunk ITSI Certified Admin
- Splunk Enterprise Certified Architect (bonus)

Lilly is dedicated to helping individuals with disabilities to actively engage in the workforce, ensuring equal opportunities when vying for positions. If you require accommodation to submit a resume for a position at Lilly, please complete the accommodation request form () for further assistance. Please note this is for individuals to request an accommodation as part of the application process and any other correspondence will not receive a response.

Lillyu00A0does not discriminate on the basis of age, race, color, religion, gender, sexual orientation, gender identity, gender expression, national origin, protected veteran status, disability or any other legally protected status.

#WeAreLilly