DarkPlaces Protocol 7

[Baker]

Now running NQ mods in a way that supports QW clients, does have some appeal to me, but the network protocol is quite poor for what NQ mods expect to be able to do.

I have a few netQuake mods I would like to port to DP, as a server,which could support FTE/QW clients via dp7, i woul love some sort of rough outline for DP7 support for netQuake strictly as a baseline non qw prediction layer that is truely netquake compatible.

I wish I had time to write up a tutorial for DP7, but you can find everything dealing with it by searching for PROTOCOL_QUAKEDP in the darkplaces source (PROTOCOL_DARKPLACES7 is never checked for, because it is the "default" mode, the code represents differences from DP7 rather than differences from PROTOCOL_QUAKE or PROTOCOL_QUAKEDP).

PROTOCOL_QUAKEDP is darkplaces' extended quake protocol from way back, it was used by nehahra for multiplayer, it's the most similar to stock QUAKE protocol (it simply added more entity flags and flags for extended sound info and such).

But the important part is that PROTOCOL_QUAKEDP will show you all the places that DP7 DIFFERS from QUAKE/QUAKEDP protocol.

The major differences are in these areas of networking:
cl_input.c CL_SendMove - the clc_move message this writes has higher precision, and a great deal of extra data (for the prydon gate cursor extension, reporting what is at the crosshair), it also has a "move sequence" which represents the current move number in the game for prediction, this is 0 if cl_movement is off (no prediction), otherwise it is used to synchronize the prediction to what the server has parsed from the client (how many moves have been executed on the server side), so that the prediction knows what data has been used and what data is still unconfirmed (and thus needs to be predicted).
After clc_move is sent it sends some clc_ackframe messages which correspond to all the server frames it has received since the last clc_move was sent out, this is critical for the server to function properly (it is how it knows what data the client has received).
cl_parse.c CL_ParseClientdata - SU_ITEMS and other stats are not networked here in the case of DP7, all stats are updated by the svc_stat* messages which are sent as unreliable messages (their receipt is confirmed by the clc_ackframe handling).
common.c MSG_WriteCoord* and MSG_WriteAngle* - these functions represent alternate encodings of coordinates and angles used by the rest of the DP7 protocol code, and have matching MSG_ReadCoord* and MSG_ReadAngle* functions.
netconn.c - this contains some packet layer functions for the QW-like networking that is used instead of the legacy quake protocol for connecting to servers, do a search for CCREQ to find things here, basically connect messages in darkplaces begin with 0xFFFFFFFF and then a text string (like in QW, Q2, Q3) rather than the specialized CCREQ_ numbers, so a different connect process is required.
The DP7 connect process begins with a getchallenge message and the server replies with a challenge response, then the client gives back that challenge code to the server in its connect request, then it proceeds normally like in Quake, the reason for this is to prevent qsmurf attacks (where someone slams a list of quake servers with connect requests spoofed to point to an intended victim of the flood, and the servers spend 5 minutes trying to talk to this person, which also makes it hard for legitimate players to connect), this connect process was inspired by Quake3 network documentation.
You can find the code for this process in NetConn_ServerParsePacket and NetConn_ClientParsePacket, where getchallenge, challenge, connect, and other commands are handled.
protocol.c EntityFrame5_ functions - this is the heart of DP7 networking, it is the loosely synchronized Tribes-like entity network code, and stat code which operates in the same way.
The heart of the Tribes-like entity/stat networking is this:
Each frame a packet is emitted containing some differences from the previous frame (I say some because often not all data fits, so this is only a partial update, this is why there are no packetoverflow problems in huge levels full of monsters in one room), along with these updates being put into the packet, it also builds a "packet journal" describing the same differences (such as which fields on an entity were sent this frame, and also which stats were sent).
Overall this is a straightforward delta of the entity/stat database for this player, and each frame it makes an attempt to fit all the differences in the database into the packet, thus nullifying the differences in the database.
Some may call this "dirty flag" networking.
Now the reason for the packet journal is simple - when the client input comes in, it contains some clc_ackframe messages, each one of these refers to a packet journal by frame number, any messages for packet journals that have been deleted (already confirmed as received) are simply ignored, any messages that correspond to packet journals will cause those packet journals to be deleted (as they have been confirmed), if a packet journal is older than the frame number specified it is now known to be a LOST packet, and its differences (entities sent, what fields were sent on each entity, and the stats sent) can now be flagged as dirty again, then the packet journal for the lost frame is deleted as it is no longer useful.
So server->client packet loss is handled by this clc_ackframe mechanism, where any lost update is restored so that it will be sent again.
There is an extra detail to this process - when a packet journal is lost, it only needs to restore the entity field flags and stat flags that have NOT been sent in any later packet journal in the database, because any later update would have already arrived by the time this repeat does.
It is worth noting that all networking uses the latest data available on this entity, which is why packet journals do not save the value sent, only the flag for the lost data itself.
If there are too many packet journals, the server can simply mark them ALL as lost (rolling their dirty flags back into the database) and start over, this mostly happens when someone is experiencing a multi-second lag spike and not confirming any packets at all.
sv_main.c SV_SendServerinfo - this calls EntityFrame5_AllocDatabase for the client, initializing the database for networking entities and stats for this level to this player
sv_main.c SV_WriteClientdataToMessage - this does not send the various SU_ flags when using DP7 protocol, because they would be redundant, however it does build a stats array which will be networked later by the EetityFrame5_WriteFrame function.
sv_main.c SV_SendClientDatagram - this uses different packet size limits based on the protocol used, for DP7 1400 byte unreliable packets are used instead of Quake's 1024 limit, and for a local client (singleplayer/listen server) the limit is 65536 bytes, so that singleplayer has the best experience in crowded levels.
sv_main.c SV_ReadClientMove - this is where the upgraded clc_move message is parsed, the cursor details could be ignored along with the move sequence (but you still get better aiming precision), the client will not predict as long as the server tells it that its last received move sequence is 0.

Now an overview of the entity networking, NOTE: all of this is in protocol.c and can be used as-is, this explanation is only informative.

The client plays dumb and simply reads updates, and echoes back frame numbers received, the updates are applied immediately to the entities, and any entities not mentioned in this packet are assumed to be identical to before (where as in QUAKE protocol they are assumed to be removed), interpolation is done between the two latest states received.

The server keeps track of which frames have been sent but not yet confirmed by the client, essentially this boils down to 3 states for each frame in the packet journal: pending (sent, no confirmation yet), lost (sent, later frame was confirmed by a clc_ackframe message but not this one), received (sent, this frame number was specifically referenced by a clc_ackframe message)

Whenever a packet is lost the server restores the update flags it represented (but first clearing any update flags that match later packet journals, no point in sending it redundantly), so that the next packet will attempt to fix the state.

For each player an entity_state_t array (snapshot) is built representing the entities the player can see, the player also has an entity_state_t array representing the current state of everything, and an update flags array (which fields need to be sent), and an entity priority array, the snapshot is compared to the entity database for the player and update flags are set accordingly for any differences detected.

Entities are prioritized by distance from the player and other attributes, how this is done is not particularly important, but the priority must continually increase - in DP it increases priority by 1, it also checks if the entity is within 1024 units and increases priority by 1, it also checks if it is a full update, attachment, has a colormap change - like a team change on a player - or changed model, and increases priority by 1, the net result is that priority is increased every frame by 1, 2, or 3, it is capped at 31.

Entities are added to lists by priority number and the lists are walked from highest to lowest, any entity that is put into this packet has its priority drop to 0, and it is added to the packet journal (to handle packet loss).

Stats changes are simply networked individually, but their receipt is confirmed by the same packet journal system, this is done by comparing the stats array built for this player against the stats database for the player and update flags are set.

There you go, that is an overview of the DP7 protocol code in DP.

Wanted to mark this exceptionally useful post for future use.