Site Reliability Engineer

Senior Member of Technical Staff

Join us as we work to create a thriving ecosystem that delivers accessible, high-quality, and sustainable healthcare for all.

We are looking for a Senior Site Reliability Engineer to join our Cloud Infrastructure Engineering & Operations division. Ultimately your work will focus on improving the performance and efficiency of our teams by joining our quest to continue building a world-class observability platform and contribute to the success of our business.

The Team:

The Logging, Metrics and Monitoring team is responsible for building and providing observability services and tools for engineering teams within the Cloud Engineering & Operations and Research & Development zones. Our services are highly visible and used every day to develop, monitor, troubleshoot and scale our web services. The team is responsible for collecting and hosting large volumes of metrics and log data; we do this by running large scale distributed, fault tolerant systems to collect and host all this data.

Our team has a big impact on productivity of hundreds of developers across athenaNation.

In a typical week, our engineers work on problems ranging from tuning performance and scaling services to debugging hard problems. We're responsible for delivering new features and partnering with development teams to solve their pressing monitoring and logging issues. We work in an agile, sprint-based schedule running daily standups and work in both the private and public cloud

Job Responsibilities

· Automate the deployment of logging, metrics, and monitoring services through configuration management utilizing Puppet.

· Address and resolve production incidents by applying Linux administration and engineering expertise.

· Lead projects from inception to completion, including designing technical solutions, managing timelines, and executing deliverables.

· Design and implement metrics dashboards and alert criteria to effectively monitor and scale services.

· Participate in a week-long on-call rotation in collaboration with team members.

· Assist development teams in enhancing their logging and metrics collection processes.

· Demonstrate the ability to manage on-call rotations every few weeks.

Typical Qualifications

· Possess 5 to 8 years of prior experience in a production environment, exhibit strong system administration and DevOps skills for managing services within a Linux environment.

· Observability: Strong understanding of observability concepts, including monitoring, logging, tracing, and alerting across distributed systems.

· OpenTelemetry: Hands-on experience in implementing distributed tracing and metrics using OpenTelemetry SDKs and collectors.

· OpenSearch or Elasticsearch: Proficiency in managing, querying, and optimizing search and analytics engines like OpenSearch or Elasticsearch.

· Jaeger: Practical experience in configuring and using Jaeger for distributed tracing in microservices environments.

· Proficient in programming with experience writing and maintaining scripts in the following languages: Bash, Ruby, Python, Perl, C , Java, and Golang.

· Experience developing Infrastructure as Code utilizing Terraform and CloudFormation.

· Display adaptability and flexibility in response to changing environmental and business demands.

Additional Qualifications

· Demonstrated experience in managing production server fleets at a scale of thousands.

· Subject matter expertise in relevant technologies, including FluentD, Kafka, Elasticsearch, Graphite, Clickhouse, Prometheus, Grafana, Graylog, Terraform, CloudFormation, Docker, Jenkins, and Git.

· Exposure to Amazon Web Services (AWS) for deploying, managing, and scaling applications, with a foundational understanding of AWS services, architecture, and best practices.

· Proficient in using protocol analyzers such as tcpdump and Wireshark.

About athenahealth

Our vision: In an industry that becomes more complex by the day, we stand for simplicity. We offer IT solutions and expert services that eliminate the daily hurdles preventing healthcare providers from focusing entirely on their patients — powered by our vision to create a thriving ecosystem that delivers accessible, high-quality, and sustainable healthcare for all.

Our company culture: Our talented employees — or athenistas, as we call ourselves — spark the innovation and passion needed to accomplish our vision. We are a diverse group of dreamers and do-ers with unique knowledge, expertise, backgrounds, and perspectives. We unite as mission-driven problem-solvers with a deep desire to achieve our vision and make our time here count. Our award-winning culture is built around shared values of inclusiveness, accountability, and support.

Our DEI commitment: Our vision of accessible, high-quality, and sustainable healthcare for all requires addressing the inequities that stand in the way. That's one reason we prioritize diversity, equity, and inclusion in every aspect of our business, from attracting and sustaining a diverse workforce to maintaining an inclusive environment for athenistas, our partners, customers and the communities where we work and serve.

What we can do for you:

Along with health and financial benefits, athenistas enjoy perks specific to each location, including commuter support, employee assistance programs, tuition assistance, employee resource groups, and collaborative workspaces — some offices even welcome dogs.

We also encourage a better work-life balance for athenistas with our flexibility. While we know in-office collaboration is critical to our vision, we recognize that not all work needs to be done within an office environment, full-time. With consistent communication and digital collaboration tools, athenahealth enables employees to find a balance that feels fulfilling and productive for each individual situation.

In addition to our traditional benefits and perks, we sponsor events throughout the year, including book clubs, external speakers, and hackathons. We provide athenistas with a company culture based on learning, the support of an engaged team, and an inclusive environment where all employees are valued.

Learn more about our culture and benefits here:

Experience LevelSenior Level

---