Page 1 of 1

packet fragmentation on modern routers

Posted: Thu Jan 22, 2015 6:14 pm
by qbism
Is packet fragmentation over wifi still a big issue as it was 5 or 10 years ago? I've been reading about the long-standing 1400byte +/- packet limit and wondered if it is still relevant. For example, do most new routers support frame aggregation and/or 'jumbo frames' of 3000 bytes?

Re: packet fragmentation on modern routers

Posted: Thu Jan 22, 2015 6:41 pm
by Spike
the larger a packet, the more likely you'll get interferance corrupting it, resulting in loss.
one way around this is to just repeat the packet multiple times. of course, this will increase packetloss on wired links.
both FTE and DP have network protocols that resend current state on loss. the server needs a journal of what was sent in each packet and the protocol needs a way to track loss somehow (a gap in ack sequences or something), but the reduction in data can be significant while large scenes can send entity data round-robin or so (as the previous frame is not instantly bad). on the other hand, the quakeworld protocol will keep resending data until its acked, and may have the best performance on a lan, but as it deltas from the last-known-received state, both ends need quite extensive state logs.

the ethernet packet limits remain as de-fragmentation can be used for ddos attacks.
in theory, routers are not meant to need to defrag packets, only the final receiver does. however, any port numbers are present only in the first fragment, and thus NAT boxes *MUST* have some way of tracking that, and the cheesy way to handle that breaks if you have packets coming from multiple sources/routes.
With wide-spread carrier-grade NAT on the horizon (combined with home/private NAT boxes), expect the worst.

minimum ipv4 fragment size is something around 576(-headers) bytes. the standard permits silently dropping any fragments smaller than that regardless of how they arrived, and fully blaming the sender for it never arriving. hurrah. any link that requires fragmentation smaller than that *MUST* support defragmenting before sending over ethernet or whatever (typically such extra fragmentation happens at the hardware/driver layer rather than the router's kernel).
if an application generates packets larger than this limit, there is a chance that such packets will be fragmented and a chance that the receiver/NAT will drop all fragments.
ipv6 has a higher minimum, but I don't remember it.

Re: packet fragmentation on modern routers

Posted: Fri Jan 23, 2015 4:38 am
by qbism
Thanks, I'm starting to understand the difference between the MTU (1500 bytes for ethernet), the minimum reassembly size (576 bytes IPV4 and 1500 bytes IPV6), and the absolute minimum datagram size (64 bytes).

Some protocols use various types of compression to squeeze as much as possible into the limit but I'm not clear on whether one scene/frame can be split into smaller chunks before sending. 'Not instantly bad' - referring to large messages that are likely fragmented? Or are these frames chopped into multiple packets by the server?

Re: packet fragmentation on modern routers

Posted: Fri Jan 23, 2015 3:15 pm
by Spike
in terms of quake payloads 1448 is the lowest I've seen needed with quakeworld servers (1446 for qw clients).
1448 should also apply to both nq reliables and unreliables (either direction).
which is 16 lower than the theoretical ethernet cap

1500(ethernet payload limit)-20(ip)-8(udp)-8(quake) = 1464
additional costs:
pppoe(oa) = 8
ipv4 options = 40 (max)
vpn = 36-80

so this 1448 may have been caused nested pppoe headers, or pppoe combined with some ipv4 option. or something else, I have no idea. all I really know is that 1450 (vanilla quakeworld's default) wasn't enough.
for those few packets where the server was trying to resend 1450 bytes of data, the connection died.

Re: packet fragmentation on modern routers

Posted: Fri Jan 23, 2015 7:41 pm
by Baker
There is a minimum size?

That seems counter-intuitive ... the sign-on handshake sends very small amounts of data.

If so, how does a client that sends only a small amount of data (wish velocity, buttons, impulse, angles) work? 99% of the time this is going to be under 50 bytes or very nearly so.

Or am I misinterpreting this? Or is something hidden going on or something I didn't notice happening (padding somewhere? An unnoticed delay somewhere?).

Re: packet fragmentation on modern routers

Posted: Sat Jan 24, 2015 4:36 am
by qbism
To be more accurate, it's a 64-byte minimum ethernet frame (based on he ability to detect collisions in a certain minimum length of cable). I happen to have q2pro source open and notice

Code: Select all

#define MSG_TRESHOLD        (64 - 10)   // keep pmsg_s 64 bytes aligned
Perhaps the '10' is the expected minimum header size. I haven't dug deep enough to know if it's related. I also read that typically the network driver will pad smaller packets before transmission. So is it even a concern?

I'm thinking of an ad-hoc coop match over wifi and one player is the listen server. Gets messy! The maximum possible edict count is higher in coop. And on a listen server the processor is splitting time between client, graphics, and server.

Re: packet fragmentation on modern routers

Posted: Sat Jan 24, 2015 7:33 am
by Spike
ethernet runt packets are not a real concern. they should common enough that hosts will pad as required.

if you're trying to keep things aligned, I'd aim for 43-byte alignment as that's the ATM cell size.
10 sounds like quakeworld/q2's netchan protocol size - two 32bit sequences and a 16bit 'qport' value for client->server packets.

Re: packet fragmentation on modern routers

Posted: Sun Jan 03, 2016 2:26 pm
by Jay Dolan
Sorry for necro-posting, but I've actually been experimenting with these values in my engine (Quetoo) just recently.

Interestingly, I've found that my own home network (ADSL NAT'ed behind Apple Airport) is happy to transparently fragment and reassemble UDP packets all the way to Rackspace hosting.

To test this, I increased MAX_MSG_SIZE to 16KB, and added some goofy logic to the game module that spawned hundreds of giblets, and deployed a dedicated server on Rackspace. I then connected from my LAN, and proceeded to blast the giblets with the BFG, which produces BFG tracer lasers and causes the giblets to bounce and fly all over the map.

In my tests, messages as large as 12KB (the largest I could generate) were handled completely gracefully. I didn't expect that. Crazy, right?

So where does this leave us? The IP spec claims that the transport layer _should_ do this transparent fragmenting for us. Is it finally safe to assume that, in most environments, it will?

Re: packet fragmentation on modern routers

Posted: Sun Jan 03, 2016 2:44 pm
by Jay Dolan
Just as a follow-up, I even connected two separate clients from my Macbook to the remote server, to ensure that qport was working correctly, and indeed, everything held up for the same test of 200 giblets + BFG nonsense. Food for thought!

Re: packet fragmentation on modern routers

Posted: Sun Jan 03, 2016 7:00 pm
by Spike
there'll always be someone with a connection that can't handle fragments.
typically with multiple levels of router that needs different mtus - 8-byte fragments are probably going to trip all sorts of DOS protection and paranoia.
there's a lot of paranoia out there, and when it works just fine for tcp you'll always have someone complain that facebook works so your game should too, despite their router being badly configured like having udp completely firewalled or whatever...

so it works for you. that's great... but that's really all your test demonstrates.

Re: packet fragmentation on modern routers

Posted: Mon Jan 04, 2016 1:07 pm
by Jay Dolan
Sure. I saw this same experiment fail 10 years ago when I tried it behind a Linksys WRT54 on cable.

Nothing works 100% of the time, not even gravity ;)

You're not really offering much in the way of solutions. And even the caveats you call out contradict the net code we've all been using for 20 years: 1450 > 576, yet QW and Q2 do not handle internal fragmentation.

I guess the point I was making is: Perhaps it's time to revisit acceptable limits for MAX_MSG_SIZE? And as an aside, in older engines, where the legacy behavior is to immediately drop any client requiring a message > 1400 bytes anyway, frankly what's the harm in attempting to deliver a larger payload? It'll either get there (yay!) or it won't, and the client will timeout or drop on a malformed / out of sequence message. What's the difference?

Re: packet fragmentation on modern routers

Posted: Mon Jan 04, 2016 5:58 pm
by Spike
that's because any time I try replying in detail, this forum logs me out and swallows my posts in the process.
1450 comes from ethernet, which is very abundant.

fte+q3 support their own netchan fragmentation. dp only does for reliables, it depends upon prioritisation and packet rates for unreliables.
fte+dp use some kind of round-robin or prioritisation in order to limit packet sizes somewhat. both use nack-based networking. both also run at more than 10fps too, of course...

imho clients should be willing to accept large payloads, but servers should try to avoid needing them. its probably better to try than to kick outright. users will feel that it was malicious either way, but an error message saying that its not personal can help a little on a psychological level.
if you're unsure what a client supports, then either you're stuck updating q2's ents at only 5 or 2.5 fps, or you're depending on fragmentation. Neither are ideal, but at least fragmentation can be 'solved' by getting the user to buy decent hardware/connectivity.