Blamed on DDoS, the Nginx Logs Disagreed

The monitoring dashboard flashed red precisely at the top of the hour. Eighty percent of requests were returning 504 Gateway Timeout errors, holding steady for ten to twenty minutes before recovering on its own, only to repeat the cycle an hour later. The report said one thing: DDoS. I took a breath and decided to look at the actual data before spinning up any expensive firewalls.

My first anomaly check revealed the logs were trapped inside the Docker container, not in the host’s /var/log/nginx/ directory where the system logrotate could reach them. The access.log had ballooned to 1.0 GB with roughly 4.1 million lines, while the error.log sat at 42 MB. It was the classic container problem I wrote about before, when a bare Docker entrypoint picked the wrong interpreter: the host and the container disagree about where things live. Pulling a 24-hour sample, I counted 106,752 requests. Exactly 85,329 of them (80%) were 504s, 15,749 succeeded, and the rest were harmless 404s. Every single timeout wave matched the same error line: "upstream timed out (110: Connection timed out) ... upstream: fastcgi://[php-fpm ip]:9000".

Reading the Pattern Inside the Logs

Digging deeper into the access logs, I noticed a brief burst from a feroxbuster/2.13.1 scanner. It fired 655 requests in three seconds, probing for .env, .git/config, and phpinfo.php. Every single one returned a 404. While noisy, this was just background internet radiation, not the attacker causing the hourly meltdowns. The real culprit was hiding in the timing.

The periodic nature of the 504 errors pointed to a queue, not a flood. Periodic traffic spikes were exhausting the PHP-FPM worker pool. To make matters worse, request_terminate_timeout was actively killing half-processed requests, forcing them to restart and clog the pipeline even further. The server was essentially choking on its own backlog every hour.

I checked the host crontab for any scheduled jobs that might trigger this hourly spike, but it was completely empty. What exactly was causing this rhythmic traffic surge remains an open question, perhaps an external crawler or a poorly timed third-party webhook. Regardless, the diagnosis was clear: this was a resource exhaustion queue, not a malicious volumetric DDoS.

A rate limit that never fired

Curiously, the server already had a rate limiting module configured to stop abusive traffic. The setup looked solid on paper, designed to throttle aggressive clients before they could overwhelm the backend.

limit_req_zone $binary_remote_addr zone=LIMITATTACK rate=2r/s;
limit_req zone=LIMITATTACK burst=5 nodelay;

However, a search through the 42 MB error.log revealed exactly zero "limiting requests" entries. The rate limit was completely dormant. This happened because the limit_req_zone directive uses $binary_remote_addr as its key [1]. Since the site sat behind Cloudflare, nginx only saw the CDN’s edge IP addresses, not the actual visitors.

Cloudflare forwards the real client IP in the CF-Connecting-IP header [2], but the default nginx combined log format and rate limit modules do not read this header automatically [4]. Consequently, thousands of legitimate, distinct visitors were collapsing into a single rate-limit bucket tied to Cloudflare’s IP. The limit never fired because the bucket was shared by everyone, masking the real traffic pattern. Same lesson as the ISP DNS poisoning incident that broke my automated research: when an intermediary sits in the middle, conventional diagnosis points at the wrong suspect.

So the response plan focused on the real bottleneck, in three steps. First, I enabled Cloudflare page caching with a Cache Everything rule and a long Edge TTL, explicitly bypassing wp-admin and logged-in user cookies. Second, I scheduled a production review of the PHP-FPM pool, specifically adjusting pm.max_children and request_terminate_timeout against available server RAM, since the latter kills workers exceeding the per-request timeout [3], alongside tuning fastcgi_read_timeout. Finally, I configured logrotate with copytruncate for the container logs to prevent them from bloating indefinitely.

The next time someone claims the site is under a DDoS attack, skip the panic. Pull a local log sample, count the status codes per hour, and ask whether the pattern is random or periodic. This time, the rhythm was perfectly periodic, and recognizing that alone saved me a sleepless night of unnecessary firewall tuning.

Sources:

• Nginx Documentation, Module ngx_http_limit_req_module [1]
• Cloudflare Fundamentals docs, Cloudflare HTTP headers [2]
• PHP Documentation, FastCGI Process Manager (FPM) configuration [3]
• Nginx Documentation, Module ngx_http_log_module [4]

[1] https://nginx.org/en/docs/http/ngx_http_limit_req_module.html [2] https://developers.cloudflare.com/fundamentals/reference/http-headers/ [3] https://www.php.net/manual/en/install.fpm.configuration.php [4] https://nginx.org/en/docs/http/ngx_http_log_module.html