Proposal - Protocol Extensions

[mh]

OK, lot of things floating around that need a new protocol number, and it's a one-way ticket to a horrific mess. Engine #1 ends up supporting protocol 15 and 16, engine #2 ends up supporting protocol 15, 16 and 17, but the two implementations of protocol 16 are incompatible, so a client/server session between the two engines will crash and burn.

We need to clean this up, so my proposal is that we define a "protocol extensions" mechanism.

Rather than bumping to a new protocol number every time, instead we implement a single "protocol 666" which tells the engine "I'm using something other than stock protocol 15 here, so watch out".

Each supported extension is a #define, so the items we need to standardise are (1) that this is the way we'll do it, and (2) the list of defines.

If using protocol 666, we read a series of ints from the server in CL_ParseServerInfo (it could be only one, but I prefer to build this to support more than 32 extensions from the outset) and store them in an extension flags member of the client struct.

The last thing we need is a way for the client to tell the server what extensions it supports. This can be set up in a similar manner to the above, and sent during CL_SignonReply.

So now we have a client which knows which extensions the server supports and a server which knows which extensions the client supports, so we take the common items which gives us which extensions we actually use. The server checks with a bitwise and when sending the data, and adjusts it's behaviour accordingly. The client checks with a bitwise and when reading the data and adjusts it's behaviour accordingly. Both can fall back to protocol 15 if any given extension is unsupported by both.

This then puts us in a position where the proposed "new standard protocol" can be implemented as quickly and easily as possible in as many engines as possible, doesn't have to be an "all or nothing" implementation (engines can pick and choose which features they want), preserves compatibility between protocol version numbers, and will always have a fallback to standard protocol 15 behaviour if something isn't there.

Comments welcome.

[frag.machine]

I've been playing around with this idea, and implemented a special message to send a extension string very similar to the DP_CHECKEXTENSION mechanism (which in turn was cloned from the OpenGL extension mechanism). The integer list is a smarter, network-wise variant. There are a couple of problems here, though:

a) somebody needs to centralize the extensions codes/string names, or the whole thing won't work. Since LordHavoc already volunteered to do a similar work with the QC extensions, I vote on him to assume the task ;

b) everyone needs to agree about the meaning and behaviour of every extension, what can become a real problem. We can end up with either radically different versions of the same extension, or with lots of very similar extensions, polluting the definition table.

Otherwise, the extension check during handshake and selective fallback is great idea.

[mh]

On (a), it's a pity that we don't have a "Quake Standards Group" anymore; would have been perfect for such a central registry.

I'll concede (b) with the observation that everyone is going to end up doing their own thing anyway. It's something of a miracle that there is (almost) unanimous agreement on even things like use of a "textures" directory...

(b) is also where proper documentation can make it live or die; if an extension is poorly documented or not documented at all, there's more danger of multiple variant implementations.

Anyway, in the spirit of "show, don't tell", i'm going to build a very simple "sample implementation" on stock Quake which should demonstrate things better.

_______________________

Some further thoughts - the server absolutely must control the protocol. My suggestion above of letting the client downgrade extension support is not a good one; it's feasible that a hypothetical protocol extension could be for anti-cheat mechanisms, for example.

[Spike]

Extension flags are good, just don't forget to allow negotiation of supported extensions.

FTE supports something like this, but tuned for QuakeWorld obviously.
In the connection protocol, the client sends a vendor/flags pair. The server masks it with what the server supports and sends the resulting vendor/flags back to the client in the svc_serverdata (svc_serverinfo is the nq name). It actually inserts the vendor/flags before the version, so the list is terminated with the protocol version that its extended from.
This mechanism allows future expansion and independant expansion.
vendorids would be stuff like LittleLong(*(int*)"FTEX") for example.

[metlslime]

an issue with this negotiated protocol is that a demo is going to be frozen on whatever featureset the client and server in that one instance agreed upon. Now, i suppose the worst case is that only the original client should be able to play it, and any other client that supports a superset of that original client's feature set would be able to too, depending on how it's done.

------

An alternate approach I've been thinking about and might as well add here for discussion purposes:

Basically each protocol extension would be a new SVC or CLC number, rather than messing with existing SVC contents, and the second byte of any new message would be a "size" so that engines that don't understand the message type can skip it by reading the correct number of bytes.

That's the easy part. For extensions to the entity update message, we can reserve the last flag of each flags byte as a "more flags in the next byte" indicator, giving us plenty of flags. Each flag could indicate the presence of some new chunk of data, in the same order as the flag bits are read, and the only issue is how to indicate the size of that chunk to engines who don't know what it is. If we didn't care about bloat, each new chunk could have a size byte in front of it. On the other hand, we could simply say that all chunks are the same size and then there's no need to encode the size. (probably 1 byte, but 2 or 4 might be better depending on how people plan to use it.) If you need to send more than that amount of data, you'd just need to reserve multiple flag bits for that extension. (on a side note, choosing 1 byte might be a good way to encourage people to be frugal. I can't believe some protocols are sending floats for alpha, for example, when 255 gradations of alpha are more than enough.)

The advantages of this approach:

1. it's dead simple to implement "ignore stuff i don't understand", therefore any engine could be patched to ignore all of it in a few lines of code.

2. the demo contains all of the features the server supports, and a different client can play it later, and even unlock some of the latent goodness in it that the original client might have ignored.

Disadvantages:

1. Not the most efficient use of bandwidth, since more data is sent than the client may use. Also, a size byte on every SVC could be wasteful (depends on how many SVCs are generally sent in a frame.)

[metlslime]

one more general note:

there are some extensions that, I think, would have to be required for the game to be meaningful. For example, if you have 2-byte modelindexes, how do you send that to a client that doesn't support that extension? At best, the client will have invisible entities, or entities with the wrong model.

So, should there be a way to tell the client to abort if the server decides one of the extensions is non-optional, but the client doesn't have it?

[mh]

one more general note:

there are some extensions that, I think, would have to be required for the game to be meaningful. For example, if you have 2-byte modelindexes, how do you send that to a client that doesn't support that extension? At best, the client will have invisible entities, or entities with the wrong model.

So, should there be a way to tell the client to abort if the server decides one of the extensions is non-optional, but the client doesn't have it?

That's pretty much where I was coming from too, although I was thinking more along the lines of anti-cheat and ensuring all clients are on a level playing field.

The client would have to tell the server which extensions it supports, and following from that the server could either decide to kick the client or to rely on the good old "illegible server message" to cause to kick itself.

[frag.machine]

Some further thoughts - the server absolutely must control the protocol. My suggestion above of letting the client downgrade extension support is not a good one; it's feasible that a hypothetical protocol extension could be for anti-cheat mechanisms, for example.

Agreed. Also, the server must have some way to tell the client which extensions are mandatory (like large coordinates support, sound/model messages using short/int values, etc), and which could be ignored (fancy particle effects and other non-vital eye candy).

[goldenboy]

whatever you do, please document it.