Am I missing something or is none of this information documented in net/http itself? This is a surprising implementation of "Close" and should be documented somewhere.
I can attest to the fact this is a Frequently Asked Question among people who are trying to Go as an HTTP client, so even though this feels somewhat icky, I can understand where it is coming from. However, I'm also missing, if I want to slam this connection closed, which is a reasonable thing to want, how do I do that without uselessly draining a socket I don't intend to reuse? Fake the "Connection: close" header into the Response object? Hope I can dig out an underlying socket of some sort under the concrete type of the Body.Response that I can bypass this code with? Seems like there should be a way to do that too.
I think it is reasonable for it to not be documented directly on Close. Close is arguably too high level for this: for example, on HTTP/2 this doesn't do what you would intend at all, since the request lifecycle is completely untethered from any underlying socket. If you really wanted to slam the connection closed ASAP it does actually seem more logical to grab the underlying connection and directly force it closed, even though this feels uglier since it requires knowing the underlying layers. To me this "Close" is logically from the perspective of the HTTP request itself and if I'm not getting myself confused and misunderstanding it doesn't really close the connection anyways (e.g. if there is no body and the connection is keep-alive'd, this wouldn't close the underlying connection, right?)
Now if you use json’s Decoder you can read a body and still might end up without reused connection - if a response contains a newline.
Decoder only reads X bytes (512? I don’t remember) and if the structure fits into these bytes exactly (and the multiplications of it) the newline leaves hanging there.
I always check for Decoder.More (this drains whitespace) after decoding.
> Using HTTP/1 (GODEBUG=http2client=0), calling this function 100 times against index.golang.org takes ~710 milliseconds on Go 1.26 on my machine. On Go 1.27, the same program takes ~150 milliseconds.
Does this mean I need to disable http2 for this to work or is that just a further proof that http2 isn't actually being used?
As explained in the post, you don’t need any of this in HTTP/2, since you get the benefits of connection sharing without having to read the drain connection body by protocol design
Right. Although the article demonstrates the changes by making requests of a public web server, I would imagine this change has been primarily motivated by a desire to reduce overhead and latency on internal communication between microservices, which very often uses unencrypted HTTP/1 traffic.
It sucks that this changes the behaviour under the hood that you previously coukd rely on the fact that closing the response body lets you skip downloading the HTTP body entirely, but I do like the tradeoff here, since it also limits the download time to just 50ms you'd not get overwhelmed if your network can't keep up with all the cancelled HTTP bodies, you'll just wait a little bit of traffic instead.
I built that package exactly to stop copying code from project to project.
Not because of the copy itself, since a little copy is better than a little dependency, as they say, but because it was a pain to update everything when I inevitably found a bug in what I copied.
At the very end it mentions it won't drain bodies for Connection: close
Does this work with reverse proxies?
I've had issues with nginx still getting IO errors when my server doesn't consume bodies and bubbling them up as 5xx codes to the client.
I believe I've seen this with Firefox as well, where XHR status codes would be 0.
Early closing, without consuming body, seems to work as expected with Chrome and curl though.
Am I missing something or is none of this information documented in net/http itself? This is a surprising implementation of "Close" and should be documented somewhere.
I can attest to the fact this is a Frequently Asked Question among people who are trying to Go as an HTTP client, so even though this feels somewhat icky, I can understand where it is coming from. However, I'm also missing, if I want to slam this connection closed, which is a reasonable thing to want, how do I do that without uselessly draining a socket I don't intend to reuse? Fake the "Connection: close" header into the Response object? Hope I can dig out an underlying socket of some sort under the concrete type of the Body.Response that I can bypass this code with? Seems like there should be a way to do that too.
I think it is reasonable for it to not be documented directly on Close. Close is arguably too high level for this: for example, on HTTP/2 this doesn't do what you would intend at all, since the request lifecycle is completely untethered from any underlying socket. If you really wanted to slam the connection closed ASAP it does actually seem more logical to grab the underlying connection and directly force it closed, even though this feels uglier since it requires knowing the underlying layers. To me this "Close" is logically from the perspective of the HTTP request itself and if I'm not getting myself confused and misunderstanding it doesn't really close the connection anyways (e.g. if there is no body and the connection is keep-alive'd, this wouldn't close the underlying connection, right?)
https://pkg.go.dev/net/http#Client.Do
> Note, however, that Transport will automatically try to read a Response Body to EOF asynchronously up to a conservative limit when a Body is closed.
Similar wording appears in a few other places; search for “asynchronously”.
Oh, good. Not exactly where I expected it but it's there at least.
Now if you use json’s Decoder you can read a body and still might end up without reused connection - if a response contains a newline.
Decoder only reads X bytes (512? I don’t remember) and if the structure fits into these bytes exactly (and the multiplications of it) the newline leaves hanging there.
I always check for Decoder.More (this drains whitespace) after decoding.
> Using HTTP/1 (GODEBUG=http2client=0), calling this function 100 times against index.golang.org takes ~710 milliseconds on Go 1.26 on my machine. On Go 1.27, the same program takes ~150 milliseconds.
Does this mean I need to disable http2 for this to work or is that just a further proof that http2 isn't actually being used?
As explained in the post, you don’t need any of this in HTTP/2, since you get the benefits of connection sharing without having to read the drain connection body by protocol design
Right. Although the article demonstrates the changes by making requests of a public web server, I would imagine this change has been primarily motivated by a desire to reduce overhead and latency on internal communication between microservices, which very often uses unencrypted HTTP/1 traffic.
It sucks that this changes the behaviour under the hood that you previously coukd rely on the fact that closing the response body lets you skip downloading the HTTP body entirely, but I do like the tradeoff here, since it also limits the download time to just 50ms you'd not get overwhelmed if your network can't keep up with all the cancelled HTTP bodies, you'll just wait a little bit of traffic instead.
I can finally retire xhttp.DrainResponseBody[1], which I just learned, wasn’t doing that good of a job anyway.
[1]: https://github.com/cipherdothost/x-go/blob/caa562ded82b8c417...
I can finally remove a similar helper that I copied from project to project and from company to company since r60
I built that package exactly to stop copying code from project to project.
Not because of the copy itself, since a little copy is better than a little dependency, as they say, but because it was a pain to update everything when I inevitably found a bug in what I copied.
At the very end it mentions it won't drain bodies for Connection: close Does this work with reverse proxies? I've had issues with nginx still getting IO errors when my server doesn't consume bodies and bubbling them up as 5xx codes to the client. I believe I've seen this with Firefox as well, where XHR status codes would be 0. Early closing, without consuming body, seems to work as expected with Chrome and curl though.