What Is Site Reliability Engineering? A Practical Guide
Summary
Site reliability engineering (SRE) is the practice of running production systems with software engineering instead of manual operations. A team sets a numeric reliability target (an SLO), measures it with SLIs, and uses the leftover error budget to decide when to ship and when to stabilize. It also caps manual toil, commonly at 50% of an engineer's time, and runs blameless postmortems so the same incident does not repeat. Start with one service and one SLO.
So, what is site reliability engineering? It is the practice of running production systems with software engineering instead of manual operations. An SRE team sets a numeric reliability target, measures against it, and automates away the repetitive work that keeps services up. If your pager goes off at 3 a.m. for the same disk alert every month, that is the problem SRE exists to remove.
Google coined the term around 2003. Today the idea is used well beyond Google, from 20-person startups to banks. What changes between them is scale, not the core mechanics.
What does a site reliability engineer actually do?
Ben Treynor Sloss, who started Google's SRE team, put it simply in the opening chapter of the Google SRE book: SRE is what happens when you ask a software engineer to design an operations team. The job is operations work done with code.
In a normal week that means four things. You define what "working" means for a service. You build the monitoring that tells you when it is not. You respond when it breaks, and you write the fix so the next person is not paged for the same thing.
Two examples make it concrete. A team that deploys by hand every Thursday builds a pipeline with automatic rollback. A team that restarts a leaking service nightly finds the leak. Both turn a recurring chore into a one-time project, which is the whole point.
The fourth part is the one teams skip. An SRE who only responds to incidents is an ops engineer with a new title. The engineering half is what makes the role different.
How is SRE different from DevOps?
DevOps is a culture: developers and operators share ownership of what ships and how it runs. It tells you the wall between teams is a problem. It does not tell you how much downtime is acceptable or when to stop shipping.
SRE supplies the numbers. It gives you service level objectives, error budgets, and a cap on manual work. Google's own framing is that SRE is one concrete way to implement DevOps ideas.
Skip the debate about which one is "better". If you run a team of 8, you will borrow from both. The useful question is whether your team can answer, in one sentence, how reliable a given service has to be.
What are SLIs, SLOs, and SLAs, and why do they matter?
Three acronyms carry most of the weight in SRE, and people mix them up constantly.
SLI (service level indicator): a measurement. For example, the share of HTTP requests that return a non-5xx response in under 300 ms.
SLO (service level objective): the target for that measurement. For example, 99.9% of requests over a rolling 28 days.
SLA (service level agreement): the contract with consequences, usually refunds, if you miss. Your SLA should be looser than your SLO.
The reason to bother is that "the site should be fast and reliable" cannot be tested. A target such as "99.9% of checkout requests succeed in under 500 ms" can be, and it gives you a way to settle arguments.
Start with two or three SLIs per service. More than that and nobody remembers them.
Pick SLIs users can feel. CPU usage is a cause, not a symptom. Failed checkouts, slow page loads, and stale data are what a customer notices.
How do error budgets work?
An error budget is the unreliability your SLO allows. It is simply 1 minus the SLO. A 99.9% availability target leaves 0.1%, which is roughly 43 minutes of downtime in a 30-day month.
Here is a small version in code, because the arithmetic is the whole idea:
slo = 0.999
window_minutes = 30 * 24 * 60 # 43,200
budget = (1 - slo) * window_minutes # 43.2 minutes
spent = 31 # minutes of bad service so far
print(f"{budget - spent:.1f} minutes left") # 12.2 minutes leftWhile budget remains, the team ships fast. When it runs out, feature releases pause and the work shifts to reliability. The error budget policy example in the SRE Workbook shows how a team writes that rule down.
The point is not punishment. It replaces the old fight, where developers want to ship and operators want stability, with a shared number both sides can read. It only works if leadership agrees in advance to honor the pause. A budget policy that gets overridden every quarter is decoration.
Why does the 50% toil rule exist?
Toil is manual, repetitive work that grows with your service and leaves nothing behind. Restarting a stuck worker by hand is toil. Writing the health check that restarts it automatically is engineering.
Google's SRE organization aims to keep toil below 50% of each engineer's time, as described in the chapter on eliminating toil. The rest goes to projects that reduce future toil or add features. The reasoning is blunt: toil expands to fill 100% of everyone's time if nobody stops it.
Treat 50% as a ceiling, not a target. A healthy team on a mature service sits well under it. If you are above it for a quarter, you are not doing SRE, you are running an ops queue.
A simple way to start: for two weeks, tag every ticket and page as toil or project. Then rank the toil by hours. The top three items are your automation backlog.
What do blameless postmortems change?
When something breaks, the postmortem asks how the system allowed a person to make that mistake, not who made it. Blame teaches people to hide information. A blameless review teaches them to write it down.
A useful postmortem is short and has four parts: a timeline, the contributing causes, what went well, and action items with owners and dates. Action items without owners are wishes.
Most of the value comes from the follow-through. Review open action items monthly. If the same cause shows up twice in six months, the first postmortem failed, whatever the document said.
Which tools does an SRE team need first?
Not many. Start with the signals, then add on-call and automation.
Metrics and dashboards. Enough to compute your SLIs. Grafana and Prometheus are a common open source pairing; Datadog and New Relic are the hosted route.
Alerting tied to SLOs. Alert when the budget is burning fast, not when a single host spikes. Fewer pages, better ones.
Feature flags and gradual rollout. Roughly speaking, most outages follow a change, so the cheapest reliability win is making changes reversible. LaunchDarkly and Statsig cover this.
A written runbook for each page. If an alert has no runbook, either write one or delete the alert.
Buy or build in that order. Teams that start with a tool purchase and no SLO end up with beautiful dashboards for nothing in particular.
Where does reading unfamiliar code fit into SRE?
This is the part nobody puts in the job description. During an incident, an SRE often has to understand a service they did not write, in a repository they have not opened in months. The question is rarely abstract. It is "where does this retry loop live, and what does it do on a timeout?"
You have done this before: grep, blame, then message whoever last touched the file. On a repo of 80K lines it works. On twelve services owned by six teams it eats the first 20 minutes of an incident, and those minutes count against the budget.
That is where a code search tool that answers questions about your own repo earns its place, for example asking where a service sets its timeouts and which callers depend on them. It will not fix the outage. It shortens the time spent reading before you can act, and that time is measurable.
Is SRE overkill for your team, and how do you start?
Skip a dedicated SRE team if you have one service, a handful of engineers, and users who tolerate an occasional bad hour. You do not need a team. You need an SLO, a decent alert, and someone who owns the pager.
Consider it once one of these is true: outages cost real revenue, on-call is burning people out, or developers are slowed by ad hoc operations requests. Many companies begin by making one or two engineers part-time SREs and see how it goes.
A common mistake is renaming the ops team "SRE" without changing how it works. If the team still owns tickets, manual deploys, and no error budget, the title changed and nothing else did.
How to start next week
You do not need a reorganization. Pick the one user-facing service that hurts most and do five things.
Write one SLI and one SLO for it. Keep them boring.
Compute the error budget and put it on a dashboard the whole team sees.
Track toil for two weeks and pick the top item to automate.
Hold a blameless postmortem for your next incident, with owned action items.
Agree in writing what happens when the budget hits zero.
If a month later the team argues less about whether to ship and more about what the numbers say, it is working. If the budget never moves because the SLO is too loose, tighten it.