request body and client_body_buffer_size

[Jeroen Ooms]

Is it correct that when $content_length > client_body_buffer_size,
then $request_body == "" ? If so this would be worth documenting at
request_body.

I am using:

proxy_cache_methods POST;
proxy_cache_key "$request_method$request_uri$request_body";

Which works for small requests, but for large requests clients got
very strange results due to $request_body being empty and hence
getting false cache hits for completely different form posts.

Is there something available like $body_hash that can be used as a
caching key even for large request bodies? Or alternatively, how
would I configure nginx to not cache requests when content_length
is larger than client_body_buffer_size?

[Maxim Dounin]

Hello!

Post by Jeroen Ooms
Is it correct that when $content_length > client_body_buffer_size,
then $request_body == "" ? If so this would be worth documenting at
request_body.

Yes, it's intended behaviour. If a request body is larger than
client_body_buffer_size, it's written to disk and not available in
memory, hence no $request_body. The limitation is more or less
obvious, and it's also explicitly documented here in
the $r->request_body() method documentation:

http://nginx.org/en/docs/http/ngx_http_perl_module.html#methods

It might worth adding some short reference into $request_body
variable description though.

Post by Jeroen Ooms
proxy_cache_methods POST;
proxy_cache_key "$request_method$request_uri$request_body";
Which works for small requests, but for large requests clients got
very strange results due to $request_body being empty and hence
getting false cache hits for completely different form posts.
Is there something available like $body_hash that can be used as a
caching key even for large request bodies? Or alternatively, how
would I configure nginx to not cache requests when content_length
is larger than client_body_buffer_size?

The

proxy_no_cache $request_body_file;

should do the trick, see http://nginx.org/r/proxy_no_cache.

[Jeroen Ooms]

@ Maxim Dounin

Thanks! This is very helpful. I have also set:

client_body_buffer_size 1m;

Could this setting have any side effects? I am not expecting too many
large POST request. From what I read, client_body_buffer_size is
actually the maximum amount of memory allocated. Does this mean that
for small requests (e.g. without a body) there is no additional
overhead introduced by raising this value?

[Maxim Dounin]

Hello!

Post by Jeroen Ooms
@ Maxim Dounin
client_body_buffer_size 1m;
Could this setting have any side effects? I am not expecting too many
large POST request. From what I read, client_body_buffer_size is
actually the maximum amount of memory allocated. Does this mean that
for small requests (e.g. without a body) there is no additional
overhead introduced by raising this value?

Yes, it's not allocated if there is no request body, and only
needed buffer is allocated if a request body is known to be
smaller. On the other hand, it can be used as a DoS vector if an
attacker is allowed to open many connections but you can't afford
them all to allocate client_body_buffer_size buffer.

Additionally, using such a big $request_body in proxy_cache_key
implies various overheads. In particular, proxy_buffer_size should
be set big enough to be able to contain cache header with a key.
Not even talking about reading/writing cache files with such keys.

[Jeroen Ooms]

Post by Maxim Dounin
The
proxy_no_cache $request_body_file;
should do the trick, see http://nginx.org/r/proxy_no_cache.

I tried this and get a warning:
nginx: [warn] "proxy_no_cache" functionality has been changed in
0.8.46, now it should be used together with "proxy_cache_bypass"

Do I just need to add an additional line:
proxy_cache_bypass $request_body_file;

It is not clear to me how proxy_cache_bypass is different from proxy_no_cache.

Post by Maxim Dounin
Is it correct that when $content_length > client_body_buffer_size,
then $request_body == "" ? If so this would be worth documenting at
request_body.
proxy_cache_methods POST;
proxy_cache_key "$request_method$request_uri$request_body";
Which works for small requests, but for large requests clients got
very strange results due to $request_body being empty and hence
getting false cache hits for completely different form posts.
Is there something available like $body_hash that can be used as a
caching key even for large request bodies? Or alternatively, how
would I configure nginx to not cache requests when content_length
is larger than client_body_buffer_size?

[Maxim Dounin]

Hello!

Post by Jeroen Ooms

Post by Maxim Dounin
The
proxy_no_cache $request_body_file;
should do the trick, see http://nginx.org/r/proxy_no_cache.

nginx: [warn] "proxy_no_cache" functionality has been changed in
0.8.46, now it should be used together with "proxy_cache_bypass"
proxy_cache_bypass $request_body_file;
It is not clear to me how proxy_cache_bypass is different from proxy_no_cache.

The "proxy_no_cache" directive prevents caching of a response got
from an upstream (i.e., do not save to a cache). In contrast, the
"proxy_cache_bypass" directive prevents returing a response from a
cache if it's there.

The warning in question is an artifact from 0.8.x time when
proxy_no_cache used to mean both "no cache" and "bypass". It
probably should be removed.