Observability Engineer

This is a job for an Observability Engineer who will be responsible for building and fine-tuning the platform components for the Observability product. The candidate will work closely with the Lead engineer, performance team, data ingestion, platform DevOps and data visualization teams under Observability product.

The Observability Engineer will support and maintain the applications onboarded to Grafana Observability, Ingestion and visualization written in PromQL, Log queries, etc., and monitoring technologies. This position will preferably be based out of India GCC, Bangalore.

Responsibilities:

• Experience in Observability and Monitoring initiatives as platform Engineer.
• Troubleshoot platform issues and restore service by resolving customer-facing incidents
• Development and implementation of build release pipelines with accountability for managing deployment schedules, issues, risks, and impediments.
• Agile development experience with team member accountability for commitment and delivery each sprint.
• Troubleshoot and implement corrections to problems associated with connectivity between the supported applications and the clients they serve
• Provide technical guidance, in the diagnosis of issues as they arise in support of critical applications
• Drive collaboration sessions among IT and business groups to facilitate optimal support and operation of the relevant applications
• Provide Site Reliability Engineering techniques such as observability, alerting and performance tuning
• Contribute to the design, implementation, and enhancement of critical applications
• Perform proactive analysis and troubleshooting to predict and prevent production incidents
• Define and contribute to monitoring capabilities for critical applications
• Collaborate with key vendors on functional, performance and capacity improvements
• Design and build tools to automate support and monitoring functions
• Ensure that all implementations of observability meet the requirements prescribed by IT Services through the effective implementation or use of approved processes, methodologies, and deliverables.
• Provide expertise and build solutions for observability applications as well as system integration with internal systems and external vendors.
• Able to provide coding and technical direction to less experienced staff or develops highly complex original code.
• Track infrastructure delivery and dependencies to implementation.

Requirements:

• Experience with gathering and organizing large volume of data to use for instrumentation into an Enterprise Observability solution.
• Experience with recommending baseline monitoring thresholds, and performance monitoring KPIs and SLAs.
• Experience with installing agents, forwarders, APIs, performance monitoring alerts, dashboards, and data trend analysis.
• Good Knowledge and understanding of Azure foundation components e.g. App GW, APIM, Virtual Network, NSG, Load Balancer, Azure VM etc. is required.
• Experience with Databases Azure SQL, PostgreSQL, MySQL, MongoDB, TSDB or similar databases.
• Knowledge of monitoring tools such as Log Analytics, App Dynamics, Grafana, Prometheus, Splunk, and Sitescope
• Azure / GCP hands-on with details around pulling observability data from managed services
• Golang / Python coding or from solutioning background with experience on SRE development and Open telemetry implementation
• Deploying/managing and optimizing enterprise level observability platform for Grafana OSS products like Mimir,Loki,Tempo, Fluentbit/Vector
• Design and develop standard Grafana dashboards for critical metrics for various Azure/GCP services using the observability data
• Experience must include at least one of the following languages: Java (required), Desired--Python, GoLang, node.js
• Experience in working with ServiceNow or similar Service Management tools
• Familiarity with Cloud technologies in Azure, AWS, and Google Cloud
• Experience on PCF, Docker, Kubernetes platform is required.
• Experience with DevOps and CI/CD tools and processes is required.
• Experience in high-performance and high-frequency data streaming (using Kafka etc.) and handling large volume of batch data is strongly preferred but not required.
• Experience with Agile / Scrum methodologies is required.

Benefits:

We are looking for someone who has the skills and qualifications described below. If you are interested in this role, please apply.

Qualifications:

• 4-year degree (Computer Science, Information Systems, or relational functional field) and/or equivalent combination of education or work experience.
• 1-3+ years of experience on integration engineering related to Observability / Monitoring framework with open source technologies such as Grafana, Mimir, Loki, Tempo, Fluentbit, Vector etc.,
• Hands-on experience with Tools and Technology is preferred.
• 1+ years of experience as a System Reliability Engineer is required.
• Experience working with Open-source platforms and Open Telemetry libraries e.g. Grafana is preferred.