Large number of invalid packets detected

[Rt Ibmer]

Hi all - I'm using the latest version of nginx 6 and recently put iptables in place.

I am seeing a significant number of matches for the following iptables rule:

iptables -A INPUT -m state --state INVALID -j LOG --log-prefix INPUT-BAD-PACKET:
iptables -A INPUT -m state --state INVALID -j DROP

Before it is suggested that perhaps the server is under attack, I am fairly well certain it is not. A great summary of the problem I am seeing is here: http://www.webmasterworld.com/forum40/1642.htm .

My situation is very much the same as described in that article, including the original posters view that there is no way that was happening as the result of an attack. Also like the original posted, my rate is very high relatively speaking. Seems like about nearly 1 out of every 100 visitors triggers this bad state issue.

If you scroll to near the bottom of that page you will find some conclusions that the invalid state of the packet was likely coming from bad checksums.

Occasionally (but a lot less than the INPUT rule) I see the same type of issue with OUTPUT, specifically from this rule:

iptables -A OUTPUT -m state --state INVALID -j LOG --log-prefix OUTPUT-BAD-PACKET:
iptables -A OUTPUT -m state --state INVALID -j DROP


Also like the original posted, the problem seems to occur a few minutes AFTER successfully serving them a web page. For instance if I take the IP from a failed bad state log on the INPUT rule and look through the nginx log files, I will see a successful 200 status code in the logs where the page was served a couple minutes BEFORE the bad state occurred. In case it matters I am using nginx with proxy_pass to serve content.

A few questions please:

1) Is it possible something with nginx is contributing to this?

2) Am I correct to assume that I might as well DROP the packets anyway? For now I changed my config to just log the error but not drop the packet, just out of fear that perhaps this rule is dropping packets that there is nothing wrong with and causing more harm then good. But it seems that if the packets are indeed bad I might as well drop them... ?

3) If the answer to #2 is that I might as well drop them, should I use REJECT instead of DROP? In case it matters this is only affecting http traffic on port 80.

4) How worried should I be about this issue? Is it just a matter of the Internet that there are going to be errored packets and that "out of sight, out of mind" is better (before installing iptables of course I had no way to know of this problem, yet everything still seems to be working ok).

5) Any tips on how I can go about tracking this down further?

Thanks!!

[Aleksandar Lazic]

Post by Rt Ibmer
Hi all - I'm using the latest version of nginx 6 and recently put iptables in place.

[snipp]

Post by Rt Ibmer
5) Any tips on how I can go about tracking this down further?

Is it possible to deactive the iptables, due the fact that a lot of the
high performance setups out there have seen that the connection tracking
with iptables have really bad performance impacts?

Hth

Aleks

[Rt Ibmer]

Post by Aleksandar Lazic
Is it possible to deactive the iptables, due the fact that a lot of the
high performance setups out there have seen that the connection tracking
with iptables have really bad performance impacts?

Thanks for your reply. Can you elaborate on what the "really bad performance impacts" are on this?

At this point, however, I am more concerned about whether indeed there is some sort of problem with my hardware and/or software at some other place (outside of iptables) that iptables has brought to light due to its logging of this issue (which otherwise is invisible - in other words it does not seem to impact people using the site).

Thanks.

[Floren Munteanu]

Any idea why domain.com/index.html is not detected, when I visit:
http://www.domain.com/ ?

listen 80;
server_name www.domain.com;

location / {
root /var/www/html;
index index.html index.htm;

fastcgi_index index.php;
fastcgi_pass 127.0.0.1:9000;
fastcgi_param SCRIPT_FILENAME
/var/www/html$fastcgi_script_name;
include fastcgi.conf;

}

Thanks.

[Igor Sysoev]

Post by Floren Munteanu
http://www.domain.com/ ?
listen 80;
server_name www.domain.com;
location / {
root /var/www/html;
index index.html index.htm;
fastcgi_index index.php;
fastcgi_pass 127.0.0.1:9000;
fastcgi_param SCRIPT_FILENAME
/var/www/html$fastcgi_script_name;
include fastcgi.conf;
}

Because http://www.domain.com/ in this configuraiton is handled by
/var/www/html/index.php

Is index.html simple static file ?

[Aleksandar Lazic]

Post by Rt Ibmer

Post by Aleksandar Lazic
Is it possible to deactive the iptables, due the fact that a lot of
the high performance setups out there have seen that the connection
tracking with iptables have really bad performance impacts?

Thanks for your reply. Can you elaborate on what the "really bad
performance impacts" are on this?

http://people.netfilter.org/kadlec/nftest.pdf

--- on page 6
.
.
Figure 7, 8 displays the results on conntrack: the maximal performance
halved compared to the plain routing case and the maximal new connection
rate is around 25,000 new connections/s, while the packet rate is about
330-340,000 pps. It is clear that connection tracking is an expensive
operation, which requires a lot of resources from the system.
.
.
---

BR

Aleks

[Cliff Wells]

Post by Rt Ibmer

Post by Aleksandar Lazic
Is it possible to deactive the iptables, due the fact that a lot of the
high performance setups out there have seen that the connection tracking
with iptables have really bad performance impacts?

It scales pretty well until getting well past 10k connections. I don't
think this is a real concern in this case.

Post by Rt Ibmer
At this point, however, I am more concerned about whether indeed there
is some sort of problem with my hardware and/or software at some other
place (outside of iptables) that iptables has brought to light due to
its logging of this issue (which otherwise is invisible - in other
words it does not seem to impact people using the site).

I'd suspect hardware/firmware issues, either your server's NIC (perhaps
not 100% Linux-supported?), your hosting company's switch/router, or
even a bad cable. The forum you linked to is quite descriptive and
you'll find that one poster (IPTables Friend 10:59 am on Nov. 10, 2007)
specifically mentions resolving this issue by reconfiguring an upstream
router that was doing packet-mangling.

Regards,
Cliff

[Brice Figureau]

Post by Rt Ibmer
Hi all - I'm using the latest version of nginx 6 and recently put iptables
in place.
I am seeing a significant number of matches for the following iptables
iptables -A INPUT -m state --state INVALID -j LOG --log-prefix
iptables -A INPUT -m state --state INVALID -j DROP

You didn't show us what was important: the log that was produced.

If the rejected packet was a ACK/FIN, then it is harmless.
On busy servers conntrack removes the connection from its table as soon as
it gets the FIN packet. When the ACK/FIN then comes, the connection is not
known and this is not a SYN. It logs it as invalid.

This could also have been a scan (nmap can trigger invalid packets too).

What kernel version are you running?
I remember that there was some issues around 2.6.23.4 about tcp reopening
in nf_conntrack, see this commit for more information:
http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=b2155e7f70b3f058efe94c0c459db023b05057bd
It was definitely fixed around 2.6.24.3. Maybe you encounter this problem.

HTH,

[Rt Ibmer]

Post by Brice Figureau
You didn't show us what was important: the log that was produced.

I included various samples from the different types of invalid state errors at the end of this message.

Post by Brice Figureau
If the rejected packet was a ACK/FIN, then it is harmless.

About 25% of the invalid packets are of this type. Should I drop them using DROP or using RESET? How about for the other types? I take it there is no harm in dropping or rejecting these - as they are obviously bad packets anyway?

I should note that in EVERY case it appears that nginx served the content successfully and that browser got it just fine (no obvious errors), and these bad packet log messages are showing up anywhere from 2-5 minutes after the user hits the page and not part of a new request. So it seems there is some packet activity that occurs minutes after the user has left the site.

Post by Brice Figureau
On busy servers conntrack removes the connection from its table as soon as
it gets the FIN packet. When the ACK/FIN then comes, the connection is not
known and this is not a SYN. It logs it as invalid.

My server is not very busy (yet). It is handling about 500-1500 requests per hour and no more than a 2 or 3 concurrently at times.

Post by Brice Figureau
This could also have been a scan (nmap can trigger invalid packets too).

I really don't think this could be a scan. I took the site live for the first time and immediately started seeing these invalid packets.

Post by Brice Figureau
What kernel version are you running?

uname -a returns the following:
2.6.21.7-2.fc8xen #1 SMP Fri Feb 15 12:39:36 EST 2008 i686 athlon i386 GNU/Linux

Here are some log samples. Thank you and I look forward to hearing your thoughts on this. I really appreciate your help!!
==========
Out of 436 log entries here is the break down (these are not exact counts but will give you an idea of what % are for what type of bad packet):

218 are type "ACK RST" on the INPUT chain. None of this type are for the OUTPUT chain. Examples:
--------
May 18 15:21:44 domU kernel: PCK-INP-BADSTATE-DROP:IN=eth0 OUT= MAC=12:....:ff:08:00 SRC=x.y.z.a DST=10.x.x.x LEN=40 TOS=0x00 PREC=0x00 TTL=238 ID=13770 DF PROTO=TCP SPT=1391 DPT=80 WINDOW=0 RES=0x00 ACK RST URGP=0

May 18 15:17:25 domU kernel: PCK-INP-BADSTATE-DROP:IN=eth0 OUT= MAC=12:....:ff:08:00 SRC=x.y.z.a DST=10.x.x.x LEN=40 TOS=0x00 PREC=0x00 TTL=240 ID=17970 DF PROTO=TCP SPT=49671 DPT=80 WINDOW=0 RES=0x00 ACK RST URGP=0
--------

110 are of type "ACK FIN". Out of this group 82 are on the INPUT chain and 28 are on the OUTPUT chain. Examples:
-------- (OUTPUT)
May 18 15:14:13 domU kernel: PCK-OUT-BADSTATE-DROP:IN= OUT=eth0 SRC=10.x.x.x DST=x.y.z.a LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=31199 DF PROTO=TCP SPT=80 DPT=3055 WINDOW=6432 RES=0x00 ACK FIN URGP=0

May 18 13:39:17 domU kernel: PCK-OUT-BADSTATE-DROP:IN= OUT=eth0 SRC=10.x.x.x DST=x.y.z.a LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=299 DF PROTO=TCP SPT=80 DPT=1157 WINDOW=6432 RES=0x00 ACK FIN URGP=0
-------- (INPUT)
May 18 15:24:08 domU kernel: PCK-INP-BADSTATE-DROP:IN=eth0 OUT= MAC=12:....:ff:08:00 SRC==x.y.z.a DST=10.x.x.x LEN=40 TOS=0x00 PREC=0x00 TTL=49 ID=25196 DF PROTO=TCP SPT=57214 DPT=80 WINDOW=65535 RES=0x00 ACK FIN URGP=0

May 18 12:58:10 domU kernel: PCK-INP-BADSTATE-DROP:IN=eth0 OUT= MAC=12:....:ff:08:00 SRC=x.y.z.a DST=10.x.x.x LEN=52 TOS=0x00 PREC=0x00 TTL=47 ID=52221 DF PROTO=TCP SPT=55817 DPT=80 WINDOW=65535 RES=0x00 ACK FIN URGP=0

99 are for type "RST", out of those 22 are for the INPUT chain and 77 are for the OUTPUT chain. Examples:
-------- (OUTPUT)
May 18 10:49:15 domU kernel: PCK-OUT-BADSTATE-DROP:IN= OUT=eth0 SRC=10.x.x.x DST=x.y.z.a LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=80 DPT=19144 WINDOW=0 RES=0x00 RST URGP=0

May 18 15:35:40 domU kernel: PCK-OUT-BADSTATE-DROP:IN= OUT=eth0 SRC=10.x.x.x DST=x.y.z.a LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=80 DPT=62353 WINDOW=0 RES=0x00 RST URGP=0
-------- (INPUT)
May 18 13:54:58 domU kernel: PCK-INP-BADSTATE-DROP:IN=eth0 OUT= MAC=12:....:ff:08:00 SRC==x.y.z.a DST=10.x.x.x LEN=40 TOS=0x00 PREC=0x00 TTL=238 ID=9943 PROTO=TCP SPT=4361 DPT=80 WINDOW=0 RES=0x00 RST URGP=0


May 18 12:31:37 domU kernel: PCK-INP-BADSTATE-DROP:IN=eth0 OUT= MAC=12:....:ff:08:00 SRC=x.y.z.a DST=10.x.x.x LEN=40 TOS=0x00 PREC=0x00 TTL=237 ID=65259 PROTO=TCP SPT=12568 DPT=80 WINDOW=6432 RES=0x00 RST URGP=0


9 are for type "ACK PSH FIN". Example:
--------
May 18 02:00:03 domU kernel: PCK-OUT-BADSTATE-DROP:IN= OUT=eth0 SRC=10.x.x.x DST=x.y.z.a LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=29585 DF PROTO=TCP SPT=80 DPT=5178 WINDOW=54 RES=0x00 ACK PSH FIN URGP=0

-- end of message --

[Rt Ibmer]

Futhermore here is some output from tcpdump which shows a very high rate of checksum errors in the packets:

tcpdump -s1500 -vvv port 80|grep -i incorrect
(I only ran this for a few seconds and here is what it output - at the
time I was getting requests at the rate of 1 request every 2 seconds)

16:37:22.719802 IP (tos 0x0, ttl 64, id 23246, offset 0, flags [DF],
proto TCP (6), length 693) domU-xxx > xxx.xxx.net.netclip: P, cksum
0x5fc4 (incorrect (-> 0xb52e), 26461:27114(653) ack 447 win 6432
16:37:29.492240 IP (tos 0x0, ttl 64, id 33170, offset 0, flags [DF],
proto TCP (6), length 1290) domU-xxx > xxxx.xxxx.com.57132: P, cksum
0x8131 (incorrect (-> 0x24da), 7241:8479(1238) ack 583 win 55
<nop,nop,timestamp 478987053 46606982>
16:37:30.047027 IP (tos 0x0, ttl 64, id 2132, offset 0, flags [DF],
proto TCP (6), length 89) domU-xxx > xxx.xxx.net.52956: P, cksum
0x0da0 (incorrect (-> 0x3f1d), 7241:7278(37) ack 396 win 54
<nop,nop,timestamp 478987191 384369485>
16:37:31.966523 IP (tos 0x0, ttl 64, id 14827, offset 0, flags [DF],
proto TCP (6), length 1294) domU-xxx > xxxx.xxxx.dialog-port: P,
cksum 0x1153 (incorrect (-> 0xc555), 26281:27535(1254) ack 415 win
6432
16:37:32.155399 IP (tos 0x0, ttl 64, id 39841, offset 0, flags [DF],
proto TCP (6), length 871) domU-xxx > x.y.z.a.51963: P, cksum 0xd32a
(incorrect (-> 0x9716), 26281:27112(831) ack 412 win 6432
16:37:32.636392 IP (tos 0x0, ttl 64, id 58696, offset 0, flags [DF],
proto TCP (6), length 1450) domU-xxx >
mobilexxxx.xx.x.x.llsurfup-http: ., cksum 0xe939 (incorrect (->
0x3c94), 23971:25381(1410) ack 293 win 6432
16:37:33.258956 IP (tos 0x0, ttl 64, id 58697, offset 0, flags [DF],
proto TCP (6), length 1450) domU-xxx >
mobilexxx.x.x.x.llsurfup-http: ., cksum 0xe939 (incorrect (->
0x743f), 25381:26791(1410) ack 293 win 6432
16:37:33.258963 IP (tos 0x0, ttl 64, id 58698, offset 0, flags [DF],
proto TCP (6), length 364) domU-xxx > mobilexxx.x.x.x.llsurfup-http:
P, cksum 0xe4fb (incorrect (-> 0x89ee), 26791:27115(324) ack 293 win
6432
16:37:38.080571 IP (tos 0x0, ttl 64, id 64581, offset 0, flags [DF],
proto TCP (6), length 1290) domU-xxx > x.x.x.x.x.6096: P, cksum
0x6517 (incorrect (-> 0x5f33), 7301:8551(1250) ack 465 win 6432

[Jure Pečar]

On Sun, 18 May 2008 13:49:32 -0700 (PDT)

Post by Rt Ibmer
Futhermore here is some output from tcpdump which shows a very high rate

If your NIC does checksumming in hardware, then tcpdump will likely report
invalid cksums in packet headers.

[Brice Figureau]

Hi,

Post by Rt Ibmer

Post by Brice Figureau
You didn't show us what was important: the log that was produced.

I included various samples from the different types of invalid state
errors at the end of this message.

Post by Brice Figureau
If the rejected packet was a ACK/FIN, then it is harmless.

About 25% of the invalid packets are of this type. Should I drop them
using DROP or using RESET?

I think you can drop them.

Post by Rt Ibmer
How about for the other types? I take it
there is no harm in dropping or rejecting these - as they are obviously
bad packets anyway?

I think you really might better ask those questions on the netfilter list:
http://www.netfilter.org/mailinglists.html
There is certainly lot of people way more knowledgeable than on the nginx
list about iptables/netfilter :-)

Post by Rt Ibmer
I should note that in EVERY case it appears that nginx served the content
successfully and that browser got it just fine (no obvious errors), and
these bad packet log messages are showing up anywhere from 2-5 minutes
after the user hits the page and not part of a new request. So it seems
there is some packet activity that occurs minutes after the user has left
the site.

Maybe you could tcpdump the whole traffic to analyze exactly what's
happening?

Post by Rt Ibmer

Post by Brice Figureau
On busy servers conntrack removes the connection from its table as soon
as
it gets the FIN packet. When the ACK/FIN then comes, the connection is
not
known and this is not a SYN. It logs it as invalid.

My server is not very busy (yet). It is handling about 500-1500 requests
per hour and no more than a 2 or 3 concurrently at times.

Yes, definitely not busy. Anyway, I think what I wrote still applies,
although I'm really not a netfilter expert.

Post by Rt Ibmer

Post by Brice Figureau
This could also have been a scan (nmap can trigger invalid packets too).

I really don't think this could be a scan. I took the site live for the
first time and immediately started seeing these invalid packets.

You bet? I get so much automatic scans per seconds, that I don't even look
to the logs anymore :-(

Post by Rt Ibmer

Post by Brice Figureau
What kernel version are you running?

2.6.21.7-2.fc8xen #1 SMP Fri Feb 15 12:39:36 EST 2008 i686 athlon i386
GNU/Linux
Here are some log samples. Thank you and I look forward to hearing your
thoughts on this. I really appreciate your help!!
==========
Out of 436 log entries here is the break down (these are not exact counts
218 are type "ACK RST" on the INPUT chain. None of this type are for the
--------
May 18 15:21:44 domU kernel: PCK-INP-BADSTATE-DROP:IN=eth0 OUT=
MAC=12:....:ff:08:00 SRC=x.y.z.a DST=10.x.x.x LEN=40 TOS=0x00 PREC=0x00
TTL=238 ID=13770 DF PROTO=TCP SPT=1391 DPT=80 WINDOW=0 RES=0x00 ACK RST
URGP=0
May 18 15:17:25 domU kernel: PCK-INP-BADSTATE-DROP:IN=eth0 OUT=
MAC=12:....:ff:08:00 SRC=x.y.z.a DST=10.x.x.x LEN=40 TOS=0x00 PREC=0x00
TTL=240 ID=17970 DF PROTO=TCP SPT=49671 DPT=80 WINDOW=0 RES=0x00 ACK RST
URGP=0

It is indeed the same issue than ACK/FIN
http://lists.netfilter.org/pipermail/netfilter/2004-June/053782.html

Post by Rt Ibmer
110 are of type "ACK FIN". Out of this group 82 are on the INPUT chain and
-------- (OUTPUT)
May 18 15:14:13 domU kernel: PCK-OUT-BADSTATE-DROP:IN= OUT=eth0
SRC=10.x.x.x DST=x.y.z.a LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=31199 DF
PROTO=TCP SPT=80 DPT=3055 WINDOW=6432 RES=0x00 ACK FIN URGP=0
May 18 13:39:17 domU kernel: PCK-OUT-BADSTATE-DROP:IN= OUT=eth0
SRC=10.x.x.x DST=x.y.z.a LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=299 DF
PROTO=TCP SPT=80 DPT=1157 WINDOW=6432 RES=0x00 ACK FIN URGP=0
-------- (INPUT)
May 18 15:24:08 domU kernel: PCK-INP-BADSTATE-DROP:IN=eth0 OUT=
MAC=12:....:ff:08:00 SRC==x.y.z.a DST=10.x.x.x LEN=40 TOS=0x00 PREC=0x00
TTL=49 ID=25196 DF PROTO=TCP SPT=57214 DPT=80 WINDOW=65535 RES=0x00 ACK
FIN URGP=0
May 18 12:58:10 domU kernel: PCK-INP-BADSTATE-DROP:IN=eth0 OUT=
MAC=12:....:ff:08:00 SRC=x.y.z.a DST=10.x.x.x LEN=52 TOS=0x00 PREC=0x00
TTL=47 ID=52221 DF PROTO=TCP SPT=55817 DPT=80 WINDOW=65535 RES=0x00 ACK
FIN URGP=0

We already talked about this one above. This could also be a Maimon scan
which uses ACK+FIN.

Post by Rt Ibmer
99 are for type "RST", out of those 22 are for the INPUT chain and 77 are
-------- (OUTPUT)
May 18 10:49:15 domU kernel: PCK-OUT-BADSTATE-DROP:IN= OUT=eth0
SRC=10.x.x.x DST=x.y.z.a LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF
PROTO=TCP SPT=80 DPT=19144 WINDOW=0 RES=0x00 RST URGP=0
May 18 15:35:40 domU kernel: PCK-OUT-BADSTATE-DROP:IN= OUT=eth0
SRC=10.x.x.x DST=x.y.z.a LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF
PROTO=TCP SPT=80 DPT=62353 WINDOW=0 RES=0x00 RST URGP=0
-------- (INPUT)
May 18 13:54:58 domU kernel: PCK-INP-BADSTATE-DROP:IN=eth0 OUT=
MAC=12:....:ff:08:00 SRC==x.y.z.a DST=10.x.x.x LEN=40 TOS=0x00 PREC=0x00
TTL=238 ID=9943 PROTO=TCP SPT=4361 DPT=80 WINDOW=0 RES=0x00 RST URGP=0
May 18 12:31:37 domU kernel: PCK-INP-BADSTATE-DROP:IN=eth0 OUT=
MAC=12:....:ff:08:00 SRC=x.y.z.a DST=10.x.x.x LEN=40 TOS=0x00 PREC=0x00
TTL=237 ID=65259 PROTO=TCP SPT=12568 DPT=80 WINDOW=6432 RES=0x00 RST
URGP=0

Hum, that one seems strange. I don't see any easy explanation.
Maybe with a full capture of the traffic you will be able to understand
what's that.

Post by Rt Ibmer
--------
May 18 02:00:03 domU kernel: PCK-OUT-BADSTATE-DROP:IN= OUT=eth0
SRC=10.x.x.x DST=x.y.z.a LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=29585 DF
PROTO=TCP SPT=80 DPT=5178 WINDOW=54 RES=0x00 ACK PSH FIN URGP=0

http://lists.netfilter.org/pipermail/netfilter/2005-August/062059.html

As a side note, I'm also running conntrack on my servers, and I do see
some of the ACK+FIN and ACK+RST invalid packets. I drop them and doesn't
really care: nobody ever complained :-)

Good luck :-)

[Rt Ibmer]

Quick update: I change my nginx config from:
keepalive_timeout 30;
to
keepalive_timeout 0;

and the log entries have stopped almost completely. Prior to making this change I was getting somewhere between 1 - 10 entries per minute in the log for the bad packets (in the cases of multiples like 5-10 range often it was the same IP reporting over and over). And now that I have made the keepalive change I've gotten just 2 log entries in the past hour.

Does this give us any more insight as to the root cause of the issue?

With this in mind should I keep the keepalive at 0, or go back to the 30 setting and just don't worry about the log entries?

Thanks again!

[Igor Sysoev]

Post by Rt Ibmer
keepalive_timeout 30;
to
keepalive_timeout 0;
and the log entries have stopped almost completely. Prior to making this change I was getting somewhere between 1 - 10 entries per minute in the log for the bad packets (in the cases of multiples like 5-10 range often it was the same IP reporting over and over). And now that I have made the keepalive change I've gotten just 2 log entries in the past hour.
Does this give us any more insight as to the root cause of the issue?
With this in mind should I keep the keepalive at 0, or go back to the 30 setting and just don't worry about the log entries?

I do not know iptables, so I can not say what it treats as invalid packets.

nginx can not affect on TCP packets in general, because it's low level TCP
thing. However, there are some cases:

1) MSIE always closes keepalive connection with RST packet,
2) if reset_timedout_connection is on (off by default), then nginx
closes timed out connection with SO_LINGER 0, and it causes RST packet
sent to a client. However, reset_timedout_connection did not affect
keepalive connections.