Enterprise Observability Platform Developer

This individual contributor will play a key role in developing and fine-tuning platform components for our Observability product.

The ideal candidate will work closely with our engineering team, performance specialists, data ingestion experts, DevOps professionals, and data visualization teams under the Observability product.

• Experience in Observability and Monitoring initiatives as a platform Engineer.
• Troubleshoot platform issues and restore service by resolving customer-facing incidents.
• Development and implementation of build release pipelines with accountability for managing deployment schedules, issues, risks, and impediments.
• Agile development experience with team member accountability for commitment and delivery each sprint.
• Troubleshoot and implement corrections to problems associated with connectivity between supported applications and clients they serve.
• Provide technical guidance in diagnosing issues arising in support of critical applications.
• Drive collaboration sessions among IT and business groups to facilitate optimal support and operation of relevant applications.
• Provide Site Reliability Engineering techniques such as observability, alerting, and performance tuning.
• Contribute to the design, implementation, and enhancement of critical applications.
• Perform proactive analysis and troubleshooting to predict and prevent production incidents.
• Define and contribute to monitoring capabilities for critical applications.
• Collaborate with key vendors on functional, performance, and capacity improvements.
• Design and build tools to automate support and monitoring functions.
• Ensure that all implementations of observability meet requirements prescribed by our IT Services through effective implementation or use of approved processes, methodologies, and deliverables.
• Provide expertise and build solutions for observability applications as well as system integration with internal systems and external vendors.
• Able to provide coding and technical direction to less experienced staff or develops highly complex original code.
• Track infrastructure delivery and dependencies to implementation.

• Experience with gathering and organizing large volumes of data to use for instrumentation into an Enterprise Observability solution.
• Experience with recommending baseline monitoring thresholds, and performance monitoring KPIs and SLAs.
• Experience with installing agents, forwarders, APIs, performance monitoring alerts, dashboards, and data trend analysis.
• Good knowledge and understanding of Azure foundation components e.g. App GW, APIM, Virtual Network, NSG, Load Balancer, Azure VM etc.
• Experience with Databases Azure SQL, PostgreSQL, MySQL, MongoDB, TSDB or similar databases.
• Knowledge of monitoring tools such as Log Analytics, App Dynamics, Grafana, Prometheus, Splunk, and Sitescope.
• Azure / GCP hands-on with details around pulling observability data from managed services.
• Golang / Python coding or from solutioning background with experience on SRE development and Open telemetry implementation.
• Deploying/managing and optimizing enterprise level observability platform for Grafana OSS products like Mimir, Loki, Tempo, Fluentbit/Vector.
• Design and develop standard Grafana dashboards for critical metrics for various Azure/GCP services using the observability data.
• Experience must include at least one of the following languages: Java (required), Desired--Python, GoLang, node.js.
• Experience in working with ServiceNow or similar Service Management tools.
• Familiarity with Cloud technologies in Azure, AWS, and Google Cloud.
• Experience on PCF, Docker, Kubernetes platform is required.
• Experience with DevOps and CI/CD tools and processes is required.
• Experience in high-performance and high-frequency data streaming (using Kafka etc.) and handling large volume of batch data is strongly preferred but not required.
• Experience with Agile / Scrum methodologies is required.