Tuning workers and connections

[Avleen Vig]

Hi folks, I have some questions about tuning nginx for best
performance, on a site which handle ~1000 requests per second.

In our config, we have:

worker_processes 16;
worker_rlimit_nofile 32768;
events {
worker_connections 8192;
}

This is on a server with 8 CPU cores and 8Gb RAM.
My understanding is that this should allow nginx to establish up to
32k network connections, because that is the limit of
worker_rlimit_nofile. We serve very few files off the disk, maybe a
dozen CSS and JS files. Everything else is handed to backends using
proxy_pass.
So with one fd for the browser, and one fd to the upstream, we should
be able to handle 16k concurrent connections.

However what we're seeing, is that around 5k connections we get a
performance hit. Connections slow down, take longer to establish, etc.
The load on the box is almost zero, nginx is clearly working very fast
and efficiently, but I'm not sure why it slows down.
Any thoughts?


Also, I'm thinking about enabling the multi_accept option, but I
couldn't find much documentation on how this works.
It sounds like a high-performance tweak which we should be using if we
get this many requests per second.
Could it be that not using multi_accept is the problem?
Nginx otherwise has to handle the incoming connection requests in
serial and this is a bottleneck?


Thanks :)

[Gabriel Ramuglia]

I'm no expert, but if your load level is low, it does sound possible
that you are having issues accepting requests in serial for each
worker connection. I would try increasing the number of worker
processes to 32 and see if that helps. Also, if you're seeing 5k
connections, you should try increasing the number of worker
connections to double what it is now, just in case.

Be happy to hear any dissenting opinions.
-Gabe

Post by Avleen Vig
Hi folks, I have some questions about tuning nginx for best
performance, on a site which handle ~1000 requests per second.
   worker_processes  16;
   worker_rlimit_nofile 32768;
   events {
       worker_connections  8192;
   }
This is on a server with 8 CPU cores and 8Gb RAM.
My understanding is that this should allow nginx to establish up to
32k network connections, because that is the limit of
worker_rlimit_nofile. We serve very few files off the disk, maybe a
dozen CSS and JS files. Everything else is handed to backends using
proxy_pass.
So with one fd for the browser, and one fd to the upstream, we should
be able to handle 16k concurrent connections.
However what we're seeing, is that around 5k connections we get a
performance hit. Connections slow down, take longer to establish, etc.
The load on the box is almost zero, nginx is clearly working very fast
and efficiently, but I'm not sure why it slows down.
Any thoughts?
Also, I'm thinking about enabling the multi_accept option, but I
couldn't find much documentation on how this works.
It sounds like a high-performance tweak which we should be using if we
get this many requests per second.
Could it be that not using multi_accept is the problem?
Nginx otherwise has to handle the incoming connection requests in
serial and this is a bottleneck?
Thanks :)

[Dave Cheney]

Try adding

events {
...
use epoll;
}

http://wiki.nginx.org/NginxHttpEventsModule#use

Cheers

Dave

Post by Avleen Vig
Hi folks, I have some questions about tuning nginx for best
performance, on a site which handle ~1000 requests per second.
worker_processes 16;
worker_rlimit_nofile 32768;
events {
worker_connections 8192;
}
This is on a server with 8 CPU cores and 8Gb RAM.
My understanding is that this should allow nginx to establish up to
32k network connections, because that is the limit of
worker_rlimit_nofile. We serve very few files off the disk, maybe a
dozen CSS and JS files. Everything else is handed to backends using
proxy_pass.
So with one fd for the browser, and one fd to the upstream, we should
be able to handle 16k concurrent connections.
However what we're seeing, is that around 5k connections we get a
performance hit. Connections slow down, take longer to establish, etc.
The load on the box is almost zero, nginx is clearly working very fast
and efficiently, but I'm not sure why it slows down.
Any thoughts?
Also, I'm thinking about enabling the multi_accept option, but I
couldn't find much documentation on how this works.
It sounds like a high-performance tweak which we should be using if we
get this many requests per second.
Could it be that not using multi_accept is the problem?
Nginx otherwise has to handle the incoming connection requests in
serial and this is a bottleneck?
Thanks :)

[Avleen Vig]

Post by Dave Cheney
Try adding
events {
 ...
 use epoll;
}
http://wiki.nginx.org/NginxHttpEventsModule#use

Thanks Dave!

Turns out this was the default (we're using a locally recompiled
version of the Ubuntu package) :-)

[Maxim Dounin]

Hello!

Post by Avleen Vig
Hi folks, I have some questions about tuning nginx for best
performance, on a site which handle ~1000 requests per second.
worker_processes 16;
worker_rlimit_nofile 32768;
events {
worker_connections 8192;
}
This is on a server with 8 CPU cores and 8Gb RAM.
My understanding is that this should allow nginx to establish up to
32k network connections, because that is the limit of
worker_rlimit_nofile.

No. With this settings you will be able to handle up to 8192
connections in each worker, about 128k in total for 16 workers.
If your system can cope with it.

Directive worker_rlimit_nofile just raises open files limit for
nginx processes if system allows this, nothing more. It's usually
needed only if you started nginx with low limit, then raised the
system one and want nginx to raise it's limit without restarting.

Post by Avleen Vig
We serve very few files off the disk, maybe a
dozen CSS and JS files. Everything else is handed to backends using
proxy_pass.
So with one fd for the browser, and one fd to the upstream, we should
be able to handle 16k concurrent connections.
However what we're seeing, is that around 5k connections we get a
performance hit. Connections slow down, take longer to establish, etc.
The load on the box is almost zero, nginx is clearly working very fast
and efficiently, but I'm not sure why it slows down.
Any thoughts?

There are several things to check:

1. Which event method used? Make sure you use kqueue for *BSD,
epoll for Linux.

2. What's in error_log? Your system logs/memory stats/etc? You
may run out of some resources (file descriptors, network buffers,
connection states in your firewall, ...).

3. In which states nginx workers are? If they are disk bound it's
probably a good idea to check why and e.g. try to tune proxy
buffers / output buffers / sendfile_max_chunk.

Post by Avleen Vig
Also, I'm thinking about enabling the multi_accept option, but I
couldn't find much documentation on how this works.
It sounds like a high-performance tweak which we should be using if we
get this many requests per second.
Could it be that not using multi_accept is the problem?
Nginx otherwise has to handle the incoming connection requests in
serial and this is a bottleneck?

With multi_accept nginx tries to accept all connections from
listen queue at once. Without it - only one connection will be
accepted on each return from event function. The bad thing with
multi_accept that if you have constant stream of incomming
connections at high rate - it may overflow your worker_connections
without any chance to process already accepted connections.

Note that this setting doesn't matter for kqueue (there is number
of unaccepted connection at return from kernel, and nginx accepts
all them). And it's always on for rtsig (not sure, but likely due
to the fact that rtsig is fragile).

Maxim Dounin

[Avleen Vig]

Post by Maxim Dounin
Directive worker_rlimit_nofile just raises open files limit for
nginx processes if system allows this, nothing more.  It's usually
needed only if you started nginx with low limit, then raised the
system one and want nginx to raise it's limit without restarting.

Ah thanks :)

Post by Maxim Dounin

Post by Avleen Vig
We serve very few files off the disk, maybe a
dozen CSS and JS files. Everything else is handed to backends using
proxy_pass.
So with one fd for the browser, and one fd to the upstream, we should
be able to handle 16k concurrent connections.
However what we're seeing, is that around 5k connections we get a
performance hit. Connections slow down, take longer to establish, etc.
The load on the box is almost zero, nginx is clearly working very fast
and efficiently, but I'm not sure why it slows down.
Any thoughts?

1. Which event method used?  Make sure you use kqueue for *BSD,
epoll for Linux.

epoll.

Post by Maxim Dounin
2. What's in error_log?  Your system logs/memory stats/etc?  You
may run out of some resources (file descriptors, network buffers,
connection states in your firewall, ...).

I saw nothing about resource starvation in the error log.
Also I should mention this was on 0.6.35.
We've also upgraded to 0.7.59.

Post by Maxim Dounin
3. In which states nginx workers are?  If they are disk bound it's
probably a good idea to check why and e.g. try to tune proxy
buffers / output buffers / sendfile_max_chunk.

They don't appear to be disk bound. I will check again tonight at our
peak hours.

Post by Maxim Dounin

Post by Avleen Vig
Also, I'm thinking about enabling the multi_accept option, but I
couldn't find much documentation on how this works.
It sounds like a high-performance tweak which we should be using if we
get this many requests per second.
Could it be that not using multi_accept is the problem?
Nginx otherwise has to handle the incoming connection requests in
serial and this is a bottleneck?

With multi_accept nginx tries to accept all connections from
listen queue at once.  Without it - only one connection will be
accepted on each return from event function.  The bad thing with
multi_accept that if you have constant stream of incomming
connections at high rate - it may overflow your worker_connections
without any chance to process already accepted connections.

Ok :-)
I've massively increased the number of worker_connections now, and the
box is otherwise quite powerful. It should be able to cope, but
there's only one way to find out!

Thanks guys!

[Michael Shadle]

Post by Maxim Dounin
1. Which event method used?  Make sure you use kqueue for *BSD,
epoll for Linux.

Is epoll enabled by default?

Does it make sense to enable this for all servers, or only those with
massive amounts of load?

[Maxim Dounin]

Hello!

Post by Michael Shadle

Post by Maxim Dounin
1. Which event method used?  Make sure you use kqueue for *BSD,
epoll for Linux.

Is epoll enabled by default?

By default nginx uses best event method available for particular
system. If it finds epoll during ./configure - it uses epoll.

Post by Michael Shadle
Does it make sense to enable this for all servers, or only those with
massive amounts of load?

There is no reason to use slow/limited event methods if better
ones are available.

Maxim Dounin

[Michael Shadle]

Post by Maxim Dounin
Hello!

Post by Michael Shadle

Post by Maxim Dounin
1. Which event method used?  Make sure you use kqueue for *BSD,
epoll for Linux.

Is epoll enabled by default?

By default nginx uses best event method available for particular
system.  If it finds epoll during ./configure - it uses epoll.

That's what I thought.

Post by Maxim Dounin

Post by Michael Shadle
Does it make sense to enable this for all servers, or only those with
massive amounts of load?

There is no reason to use slow/limited event methods if better
ones are available.

How can I tell which method I'm currently using?

[Maxim Dounin]

Hello!

On Thu, Jul 02, 2009 at 11:34:43AM -0700, Michael Shadle wrote:

[...]

Post by Michael Shadle

Post by Maxim Dounin

Post by Michael Shadle
Does it make sense to enable this for all servers, or only those with
massive amounts of load?

There is no reason to use slow/limited event methods if better
ones are available.

How can I tell which method I'm currently using?

At start nginx logs it at notice level, e.g.

2009/07/02 23:04:02 [notice] 83443#0: using the "kqueue" event method

Maxim Dounin

[Momchil Ivanov]

Post by Avleen Vig
Hi folks, I have some questions about tuning nginx for best
performance, on a site which handle ~1000 requests per second.
worker_processes 16;
worker_rlimit_nofile 32768;
events {
worker_connections 8192;
}
This is on a server with 8 CPU cores and 8Gb RAM.
My understanding is that this should allow nginx to establish up to
32k network connections, because that is the limit of
worker_rlimit_nofile. We serve very few files off the disk, maybe a
dozen CSS and JS files. Everything else is handed to backends using
proxy_pass.
So with one fd for the browser, and one fd to the upstream, we should
be able to handle 16k concurrent connections.
However what we're seeing, is that around 5k connections we get a
performance hit. Connections slow down, take longer to establish, etc.
The load on the box is almost zero, nginx is clearly working very fast
and efficiently, but I'm not sure why it slows down.
Any thoughts?
Also, I'm thinking about enabling the multi_accept option, but I
couldn't find much documentation on how this works.
It sounds like a high-performance tweak which we should be using if we
get this many requests per second.
Could it be that not using multi_accept is the problem?
Nginx otherwise has to handle the incoming connection requests in
serial and this is a bottleneck?

The first thing to do would be playing with sysctls, especially increasing
backlog and somaxconn, decreasing tcp_fin_timeout. If this doesn't help a lot
you may consider using keepalive connections (only if you are hitting some
issue with handling lots of accepts).

If you move to keepalive connections, you should make sure that you befefit at
all, therefore monitor requests/handled accepts.

Momchil