A new standardised protocol?

[metlslime]

there are two ways to go with protocols i think, first is to keep the basic architecture of netquake protocol, and just patch on little things -- like using shorts instead of bytes for models, sounds, ammo, etc, adding a field for entity alpha, and so on. This is pretty much what I have been doing myself recently, though without any idea of a community standard.

The second approach is to remove even basic assumptions and produce a much more modern protocol, taking lessons from quakeworld, tribes, and others. The primary basic assumption, which makes netquake so simple, is that all entities the client cares about are included in every packet, so there are no synchronization problems. This hits an inevitable wall in that UDP packets can only be so big to get past most routers. We could do a lot with partial updates, splitting into multiple packets, sending only important entities, delta compression, etc, but these all rely on somehow keeping the client and server synced up.

Is there a middle ground? It's possible, but I can't think of exactly what it would be. Partial updates but without any synchronization attempt... maybe with the server randomly re-sending unchanged data just to improve the chances that the client gets it?

[mh]

I'd tend to favour keeping as close to protocol 15 as possible. It may not be the best option technically, but it does (hopefully!) have the advantage of making widescale adoption a lot simpler, and therefore more likely.

[Spike]

FTEQW's extended QW protocol strips out any additional features that clients do not support, on an individual feature basis. This keeps it extensible, but if I use bits badly, the only way to fix it is to remove features again later...
You'll find that many things the server sends are sent on a per-client basis anyway. Entities, for example, are the bulk of the data sent over the protocol, they're the thing that most engines modify first. And they're always sent on a per-client basis (with the exception of baselines).
Its fairly trivial to do this really.
If you use different svc ids, then the client says what it supports, and never even has to check to see which version is being used.

Yes, FTEQW supports DP5 through DP7 protocols in its client, but only 6 and 7 on the server. Or something like that. It probably doesn't support the more recent changes to DP7 though.
Unlike the differences between QW and NQ, DP7 vs NQ has no major incompatibilities. The major differences are simply limits, stats, and entities. Although yes, there are a couple of other twiddles around the place.

If the mod doesn't use the extra features of a protocol anyway, then it really doesn't matter if the protocol is downgraded.

FTEQW demos have an additional bitmask in the svc_serverdata (svc_serverinfo for you NQ people), which specifies the extensions used in that demo. This is a re-statement of what the client already knows, of course, but its useful to at least warn clients that there are features in the protocol that it might not recognise. You can only play that demo with a client that supports all the original features. This is a bit of a hinderance but not a major issue.

Just think... If all engines supported csqc and the same protocol extensions to support it, then any other protocol limitations could be worked around... *cough*

[mh]

Point me at the info and documentation on CSQC and I'll happily support it.

Right now I'm having a hard time finding anything useful; the best I can find is this: http://www.fteqw.com/wiki/index.php?title=EXT_CSQC which looks like a journey through trial and error - not my idea of fun. Same applies to attempting to reverse-engineer something from another engine.

If it's so cool and useful and solves all problems, it really should be made more accessible to people.

[Urre]

mh: you know, people keep saying that all the time. I guess it's different when it comes to integrating it to your engine, but learning the qc side of it probably isn't as hard. But I can understand people not being entirely keen on learning how to create an entire codebase based on incomplete lists of system globals, builtins and functions. Dresk did most of the hard work, and I just expanded on that. If you know qc, and feel adventurous, it's kind of fun actually.

I've promised to write up some tutorials or something of that kind for csqc, will have to get started on that soon...

[mh]

I think this slight sidetrack has highlighted one absolute must-have for any proposed new protocol. Whatever comes out of this discussion absolutely NEEDS to be documented properly. It might be the greatest protocol in the world, but if it can't be implemented quickly and easily in existing codebases (including stock GLQuake), it won't really get much uptake.

[LordHavoc]

Havoc, didn't you say a while ago that there was something you wanted to add to dp7 that justified making a dp8?

Yeah, embedding reliable messages in the unreliable stream, and resending them if packetloss occurs, just like I do in DarkWar, which would get rid of the extreme lag with reliable messages (in the Quake protocol a reliable packet is sent, then when it gets confirmation that it was received, it sends another if there's anything queued up - the frequency gets worse the higher your ping).

I think I wanted to redesign the packet headers too.

Hard to remember now.

[LordHavoc]

I think this slight sidetrack has highlighted one absolute must-have for any proposed new protocol. Whatever comes out of this discussion absolutely NEEDS to be documented properly. It might be the greatest protocol in the world, but if it can't be implemented quickly and easily in existing codebases (including stock GLQuake), it won't really get much uptake.

I could take a day off and document the entire DP7 protocol sometime - after IronGrip: Warlords ships at least, then I'll have some time again.

[Urre]

Maybe make DP8 first, and document that instead?

[LordHavoc]

Maybe make DP8 first, and document that instead?

DP8 is a lower priority.

[scar3crow]

I remember reading about how Quake 4 is even more aggressive about optimizing such far updates, not sending info like models or animation frames. This could definitively be used also for not directly visible entities into the fat PVS (provided the computational cost to detect this is reasonable ), like entities completely behind walls.

I am reminded of DarkPlaces cullentities cvar which basically defeats wallhacks and saves on bandwidth (with few issues, an example of one would be the shifting of light with rtlights in e3m6, the low hallway behind the gold door - you can see shadows appear on the far wall when the engine decides to no longer cull the func_plat entity). Just something to maybe keep in mind regarding that functionality of being quite so aggressive.

[Lardarse]

Possibly off-topic here, but I can't see a more suitable topic for it, and I don't know if it deserves a topic of its own:

How to include .alpha support in an engine when nothing about it is mentioned in the QuakeC: http://www.celephais.net/board/view_thread.php?id=60190 posts 11 and 14.

[frag.machine]

/me bumps the thread with a slightly different but related topic...

I was thinking about how sending coordinates values as longs eats so much network bandwidth when I stumbled in a really simple idea to avoid it. It's so simple that either it's likely to be wrong and just don't work (although I must confess I cannot see why not) or somebody else already had this insight and I didn't realize it . Anyway...

Let's say that we split the world in a three-dimensional grid of cubic sectors with 8192 units side, the first sector starting at the lowest coordinates values ('-maxx -maxy -maxz') and the last sector starting in the highest (that's it, 'maxx-8192 maxy-8192 maxz-8192'). Then, when we want to send a coordinate, instead of sending 3 longs (12 bytes) we do some math and send instead the corresponding sector index, and 3 shorts representing the coordinate offsets inside this sector. In the client side we apply the inverse calculation and restore the x, y and z values corresponding to the sector and add the offsets to this to get the final coordinate values. I did a quick math, and assuming a 256 sectors grid (so the index would fit in a single byte) we can have coordinates ranging from '-32768 -32768 -8192' to '32767 32767 8191', the biggest drawback in this case is that Z values couldn't be as big as the other axis (but frankly, I don't think it's a big deal, and one willing to expand it could just use a short to send the sector index instead and still would save a good ammount of bandwidth).

Anyone here can see a reason why this would not work ?

[Spike]

Alternatively, you could send things relative to the view origin. Then you wouldn't have the limitations imposed by sectors nor the added bandwidth.
(although admittedly you would also need to send the view origin relative to the view origin, which is always fun but isn't that esentually what a delta is?).

Really if you want a standard protocol, you need it to be easy to implement. A drop in replacement. Without that, any new standard is just an incompatability with older versions of the same single engine.

[metlslime]

I was thinking about how sending coordinates values as longs eats so much network bandwidth when I stumbled in a really simple idea...Anyone here can see a reason why this would not work ?

I was playing with different ideas like this recently, too. I mainly had the goal of having a way to support larger coords without making packets (much) bigger for maps that don't need the extra range. My ideas were:

- send all coords as 3 bytes instead of 2. (20.4 fixed point instead of 13.3 or something.) Savings over longs, but still bloats packets.

- send one more byte after the coords, with 3 bits for X, 3 bits for Y, and 2 bits for Z. Basically the same thing you suggested. Problem: quakec can send coords directly without calling any builtin functions (mainly for tempents, but also some progs send hand-crafted svc_sound messages.) There's no clean way for the engine to know about that.

- new SVC messages that support larger coords, old messages still supported (and used whenever the coords are small.) Problem: quakec-generated messages will always be small coords, so no temp entities will ever make it outside the 8192^3 box.

- writecoord/readcoord functions actually devote bit 15 to be "there is an extra byte attached after this one", which means coords between -2048 and +2048 are sent as 2bytes, and anything past that is sent as 3. This works invisibly to the rest ofthe code, including quakec. Problem: even the smallest quake maps are usually off-center and, for example, e1m1 is mostly past +2048 on the y axis. So this still bloats packets even for small, old, stock maps.

And the other problem is all these tricks are too complicated for the modest efficiency they provide. I ended up postponing large coords to a future protocol. Probably the best method is really just to consistently send larger numbers, and solve the packet size problem some other way, or just saying "works in single player, good luck playing COOP on big maps on the internet."