From Full Buffering to Streaming Uploads in Go

50 MiB per Click on a Single Endpoint

A rental form on a public service portal accepted five document slots, up to 10 MiB each. The old handler read everything with io.ReadAll. In the worst case one request buffered 50 MiB in RAM, and the multipart parser held a second copy. A request that failed halfway did not return that memory any faster; the copies only left through garbage collection later.

My first guess: lowering the file size cap would be enough. Simple arithmetic disagreed. A hundred authenticated users submitting at once, even with valid files, would still force the server to hold roughly 5 GiB of buffers. A per-request cap controls the size of each wave, not the number of waves. Two different problems, memory per request and request frequency, need two different mechanisms.

The Frequency Layer: a Per-User Limiter

The first defense is a rate limit of 30 requests per minute. The key is built from the logged-in user's ID; anonymous traffic falls back to a per-IP bucket. When the limit is crossed, the server answers 429 RATE_LIMITED with a Retry-After header, so a legitimate client knows when to retry. Verification is concrete: send 31 requests within one minute with the same token, and request 31 must be rejected before touching any business logic.

The Memory Layer: Streaming, Not Buffering

The second defense changes how files are read. The request body is wrapped with http.MaxBytesReader before parsing. The official documentation states this function is intended for limiting the size of incoming request bodies and prevents clients from wasting server resources[3], returning a MaxBytesError once reads pass the limit[3].

r.Body = http.MaxBytesReader(w, r.Body, 55<<20) // total headroom, not per file

The parser is then invoked with a small memory window. The ParseMultipartForm contract guarantees that at most maxMemory bytes of file parts stay in memory, with the remainder written to temporary files on disk[3].

err := r.ParseMultipartForm(1 << 20) // 1 MiB in RAM, the rest disk-backed
if err != nil {
    http.Error(w, "payload too large", http.StatusRequestEntityTooLarge)
    return
}

Each part then streams into storage with io.Copy instead of being read whole. Before writing, the first 512 bytes are read for magic-byte sniffing, a cheap identity check so the file extension is never trusted blindly. During the copy, the exact 10 MiB per-slot cap is enforced. On any failure path, the partial file is deleted. The verification is tangible: send a large file and cut the connection halfway through, and no leftover file remains in the storage directory.

The end state: the endpoint holds roughly 1 MiB per request regardless of file size, with a limiter answering the question buffering cannot, namely how often. Together, streaming uploads and rate limiting close two vectors at once: memory amplification and request floods.

Sources

1. http package - net/http - Go Packages (accessed 2026-10-12).