What Actually Causes a Server to Run Out of Memory

When a website goes down at 2am and comes back on its own by 2:20, or a server becomes unresponsive and needs a reboot, the cause is memory exhaustion far more often than anything else. It is also the outage that is most often misdiagnosed as “a traffic spike” or “a hack”, because from the outside it looks like the server just stopped.

Here is what is actually going on, without the jargon, and what to do about it.

Memory is not shared out fairly

A Linux server has a fixed amount of RAM. Every process on it — the web server, each PHP worker, the database, the cache, the backup job, the monitoring agent — asks for memory as it needs it, and the kernel hands it out until there is none left.

Nothing tells a process “you have used your share, stop”. The database will happily take 4 GB if it is configured to. Each PHP worker takes whatever the page it is rendering needs. If the sum of everyone’s needs exceeds the RAM, the kernel has two options.

Option one: swap, which makes everything slow

If swap space is configured, the kernel starts moving the least-recently-used memory to disk. Disk is thousands of times slower than RAM, so a server that is swapping heavily crawls. Requests take longer, so more PHP workers are busy at once, so more memory is needed, so more swapping happens. This feedback loop is why “slow” turns into “down” over a few minutes rather than instantly.

Option two: the OOM killer, which makes something disappear

When there is no swap left (or none configured), the kernel invokes the out-of-memory killer. It picks the process it judges least important and most memory-hungry, and terminates it. On a typical web server that is very often MySQL, because MySQL is the largest single process.

The website then fails with “Error establishing a database connection” until MySQL restarts — automatically, if the service is configured to, or when someone notices. That is your 20-minute outage.

You can see this happen after the fact: dmesg or /var/log/kern.log will contain lines like Out of memory: Killed process 1234 (mysqld). If you have ever had a mystery outage, it is worth looking.

The usual suspects

In practice, memory exhaustion on a web server comes from a short list of causes:

PHP-FPM allowed to spawn more workers than the RAM can hold. This is the big one. PHP-FPM’s pm.max_children sets how many requests can be processed at once. Each worker uses somewhere between 30 and 150 MB depending on the site. If max_children is 50 and each worker needs 80 MB, a busy moment can demand 4 GB from PHP alone — on a server that might have 4 GB total, with MySQL already using half of it. The default value is frequently wrong for the server it is on, because it was set once and never revisited.

MySQL configured for a bigger server than it is on. Copying a my.cnf from a tutorial written for a 32 GB server onto a 4 GB VPS. The innodb_buffer_pool_size alone can be set to more RAM than exists.

A traffic spike that is not customers. Bots hitting uncached pages (login, search, random query strings) keep every PHP worker busy at once, which is the exact condition the first cause needs to become an outage. The spike did not cause the problem; it revealed a configuration that could never survive one.

A backup or cron job running at the wrong time. A database dump or a tar of the uploads folder briefly needs a lot of memory and disk I/O. If it runs at peak traffic instead of 3am, it can tip a server that is fine the rest of the day.

A memory leak. Rarer than people think, but real: a plugin or an application bug that allocates memory on every request and never frees it, so each PHP worker grows until it is recycled. PHP-FPM’s pm.max_requests exists to limit the damage by restarting workers periodically.

The three settings that prevent most of it

1. Size PHP-FPM to the server. Measure the average memory of a PHP worker on the actual site (ps will tell you), subtract what MySQL and the OS need from total RAM, and divide. That number, not the default, is pm.max_children. It is usually far lower than people expect, and a lower number with a queue is much better than a higher number with an OOM kill.

2. Size MySQL to the server. innodb_buffer_pool_size should be a fraction of RAM that leaves room for everything else — on a small server that shares with PHP, often only 25–40%.

3. Alert before it runs out, not after. A monitoring check on free memory and swap usage, with a threshold that fires while there is still time to do something. If you find out from the outage, the monitoring is not doing its job.

And one habit: after any OOM event, look in dmesg to see which process was killed and what was using the memory at the time. The kernel logs a table of every process and its memory. It is the closest thing to a crash report a Linux server produces, and it usually names the cause outright.

If you’d rather not think about this

Correctly sizing PHP-FPM and MySQL for a given server and workload is one of the things a managed hosting or cloud service exists to do — along with the monitoring that tells someone before the OOM killer does. If you want to know whether your current server is sized sensibly, item 12 and 13 on the Website & Infrastructure Health Checklist walk through exactly what to look at.