Describing the Quake Command System

[Baker]

This is a boring topic, it is also the primary backbone of the Quake engine. There are other sub-systems that are important, but this is the one that makes the Quake engine flexible and powerful.

Memory could be done another way, game logic could be Quake 2 DLL style or hardcoded, but everything in the engine depends on this to give it flexibility. It also makes the engine very easy to expand, gives users many tools, make everything configurable.

The Quake command system consists of 2 buffers (text storage memory):

1. The command execution buffer. Commands waiting to be processed. (8192 bytes in stock Quake)
2. The console. Output from commands already processed. (16384 bytes in stock Quake, almost always larger in a modified engine)

Both of the above are essentially plain text. When you start up Quake, more or less config.cfg gets read and copied into the command execution buffer. When you type something in the console, that line gets added to the command execution buffer. When you type "exec myconfig.cfg", that file is tossed into the command execution buffer.

Each frame -- a frame being every time the engine runs one cycle of code --- the engine will run everything in the command buffer one line of text at a time until there is no more text to process. The exception is the "wait" command, which will tell the engine to stop processing commands for this frame.

In code, the way the command buffer works is this:

1. Commands. Essentially a mapping of text to a function. Think of it as a 2 column table. key = "path" function = COM_Path_f (code address).

Code: Select all

Text                      Function 
 (char * name)             void (*xcommand_t) (void)
-----------------------   ----------------------------
"centerview"              V_StartPitchDrift
"map"                     Host_Map_f
"path"                    COM_Path_f
"quit"                    Host_Quit_f

There is no set number of commands. It throws every command on to the hunk (the memory Quake allocates for operation) during engine start up and alphabetizes it as each command is added. Since after engine startup (host initialization), the engine bookmarks a point in memory and says "everything after this will be cleared every map" and as a result, no new commands can be added when host initialization, and therefore the memory bookmark of the "clearing point", is set.

2. Console variables (cvars). Essentially a mapping of text to a function. Think of it as a 3 column table. key = "volume" string representation of value = "1.0" machine representation = (floating point stored interpretation of value). A cvar can't have the same name as a command.

Code: Select all

Cvar Name Text            Text Value    Floating Point Numeric Value of Text
-----------------------   -----------   ---------------------------------------------
"cl_forwardspeed"         "200"         200.0
"name"                    "player"      0    (since "player" as text doesn't convert to a numeric value)
"volume"                  "0.7"         0.7

Any time a console value is changed, the floating point value is recalculated from the text. So you do "cl_forwardspeed 400", the "400" is evaluated to be the number 400.0 in floating point --- which is the data type that supports decimal places. Cvars in Quake are fixed and unchangeable and known when compiling the engine source code, so unlike commands they don't go on the Hunk (but the string value does).

3. An alias. When found on the command buffer, it is expanded to its full value .... alias +quickattack "impulse 7; +attack; impulse 2" ... and interpreted.

4. Command parser. Takes a line of text [or up to a semi-colon] like ---> "maxplayers 16" . Splits on spaces (argument #1 "maxplayers" argument #2 "16" ...) and counts the number of arguments (1: "maxplayers" 2: "16" <--- count = 2) and as commands are executed, the function checks the argument count and can check the different argument values if the command uses them or ignore them if the command doesn't use it.

The command might refuse to do anything if the arguments it requires aren't available and might print a message complaining and explaining how to use the command.

The command parser splits on spaces and a few other characters, actually. Notably, if part of current line is within quotes like if the current line is: name "john smith" is doesn't split on a space in quotes (Argument #1 "name" Argument #2 "John Smith").

5. The special command "cmd". Instead of interpreting this locally, sends it to the server you are connected to (which may or may not choose do anything with what you sent depending on the command). "rcon" --- remote console -- operates in a similar manner, but isn't actually part of the stock [Net]Quake engine

6. Executing a command on behalf of a client (clc_stringcmd) --- this is a curveball. You are connected to a server playing coop (the server might be another computer on a LAN) and type "god". Well ... this is a bit involved ...

1. These commands in their code know they are only valid in-game

Code: Select all

void Host_Noclip_f (void)
{
	if (cmd_source == src_command) <---- "src_command" means came from the local command buffer (i.e. you typed in the console or pressed a key)
	{
		Cmd_ForwardToServer (); <-------- Even in single player, you have to be "connected to a server" (in single player, you are the server too).
		return;
	}
	....
}

2. Functions that sends the command to the server.

Code: Select all

void Cmd_ForwardToServer (void)
{
	if (cls.state != ca_connected)  <------------- you need to be connected to a server (single player, you are the server too).
	{
		Con_Printf ("Can't \"%s\", not connected\n", Cmd_Argv(0));
		return;
	}

	if (cls.demoplayback)
		return;		// not really connected

	MSG_WriteByte (&cls.message, clc_stringcmd);  <----- write the command out in the message we send to the server

	if (Q_strcasecmp(Cmd_Argv(0), "cmd") != 0) <----- "cmd" isn't something we send, it is something we ignore.  We are "in-cmd" (see above "cmd")
	{
		SZ_Print (&cls.message, Cmd_Argv(0));
		SZ_Print (&cls.message, " ");
	}

	// A modified Quake engine might have "talk macro expansion right here"

	if (Cmd_Argc() > 1)
		SZ_Print (&cls.message, Cmd_Args()); <------ sent the command arguments if there are any
	else
		SZ_Print (&cls.message, "\n"); <----------------- hrm?  This is obvious, but why not present in other "if" condition
}

3. Server reads message (host_client is a global, it is the current player we are getting messages from ...)

Code: Select all

/*
===================
SV_ReadClientMessage

Returns false if the client should be killed
===================
*/
qboolean SV_ReadClientMessage (void)
{
	...

	case clc_stringcmd:  <--------------- we received a command from a client
	s = MSG_ReadString (); <------------- read it in
	if (host_client->privileged) <------------- if special client, we do this command no matter what (idgods)
		ret = 2;
	else
		ret = 0;

	(check to see if we know what this command is)

	if (Q_strncasecmp(s, "status", 6) == 0)
		ret = 1;
	else if (Q_strncasecmp(s, "god", 3) == 0)
		ret = 1;

	(check to see if we know what this command is)

	if (ret == 2)
		Cbuf_InsertText (s);  <------------------ idgods, run this command like we typed in the console.
	else if (ret == 1)
		Cmd_ExecuteString (s, src_client);  <-------------- run this command noting this is for a specific client
	else
		Con_DPrintf("%s tried to %s\n", host_client->name, s);  <----- we don't know what this command is, log it
	break;
}

4. Command is run, noting it is being applied as the current client ...

Code: Select all

void Host_God_f (void)
{
	...

	if (pr_global_struct->deathmatch && !host_client->privileged)  <-------- Don't let someone do this if progs is in deathmatch mode.
		return;

	sv_player->v.flags = (int)sv_player->v.flags ^ FL_GODMODE; <---- modify player
	if (!((int)sv_player->v.flags & FL_GODMODE) )
		SV_ClientPrintf ("godmode OFF\n");
	...
}

Code: Select all

Memory Block (The Hunk)

Command buffer (our 8192 sized place to text to process)
Commands (The above table of text/function pairs)
Network connection status table (1 per max players, which is as many as 16)
String Memory (by default 256 KB)
-------------------   
BOOKMARK --- *Clear Everything after here on new map* (host_hunklevel)
-------------------
Server progs.dat (compiled QuakeC game logic, interpreted as byte code)
Server QuakeC interpreter fields stuff (QuakeC can add fields per entity, the engine doesn't know in advance what these are)
Map data (the .bsp file)
Anything else

I didn't know the command buffer gets complex from a client-server perspective (I hadn't thought of it before), but even the client-server aspect is rather straightforward.

[Baker]

A re-construction of sorts of the command, name, table.

Wrote up this so I can re-use for console variables system. Called like this: Link_Alphabetically ((link_t**)&topitem, (link_t*) newitem); // This function may change the topitem to newitem.

Code: Select all

void Link_Alphabetically (link_t** head, link_t *newitem)
{
	link_t* cursor   = *head;
	link_t* headitem = *head;
	
	// Update alphabetical link list.  First check to see if we are top item
	if (headitem == NULL || strcmp (newitem->name, headitem->name) < 0 )
	{
		newitem->next = headitem;
		*head = newitem;
		return;
	}
	
	// Not top ... run through list and stop at end or when we alphabetically beat an item
	while (cursor->next && strcmp(newitem->name, cursor->next->name) > 0)
		cursor = cursor->next;

	newitem->next = cursor->next; // We take over the next item spot
	cursor->next  = newitem;		  // And become the next item
}

Code: Select all

#include "engine.h"
#include "command.h"

#define MAX_NUM_COMMANDS 512

static cmd_t *topitem;
static cmd_t commands[MAX_NUM_COMMANDS];

cmd_t* Command_FindName (char* name)
{
	int i;
	
	for (i = 0; i < MAX_NUM_COMMANDS; i ++)
		if (commands[i].function && StringMatchCaseless (name, commands[i].name) )
			return &commands[i];

	// Didn't find
	return NULL;
}

void Command_Execute (char* name)
{
	cmd_t* myCommand = Command_FindName (name);

	if (!myCommand)
		Host_FatalError ("Command_Execute: \"%s\" not found");

	// run it
	myCommand->function ();
}


void Command_Add (char *name, voidfunc_t function)
{
	int i, newslot  = 0;
	cmd_t* newitem  = NULL;


	if (host.startup_complete) 
		Host_FatalError ("Command_Add:  Can't add command \"%s\" after host startup complete", name);

	if (Command_FindName (name))
		Host_FatalError ("Command_Add:  Can't add command \"%s\" already exists", name);
	
	// Find empty slot
	for (i = 0; i < MAX_NUM_COMMANDS; i ++)
		if (commands[i].function == NULL)
		{
			// Found slot ... fill it in
			newitem = &commands[i];
			newitem->name = name;
			newitem->function = function;
			break;
		}

	if (!newitem)
		Host_FatalError ("Command_Add: Can't add \"%s\" no empty slot (slots full?)", name);

	Link_Alphabetically ((link_t**)&topitem, (link_t*) newitem); // This function may change the topitem to newitem
}

void EngineCommand_MsgBox (void)
{
	System_MessageBox ("Title", "Hello");


}


fbool Command_Init (void)
{
	Command_Add ("msgbox", EngineCommand_MsgBox);
	Command_Add ("msgbox2", EngineCommand_MsgBox);
	Command_Add ("msgbox3", EngineCommand_MsgBox);

	Command_Execute ("msgbox");

	return True;
}


void Command_Shutdown (void)
{


}

[taniwha]

QF uses hash tables for the commands and cvars. When a sorted list is needed (tab-completion, cmdlist, cvarlist), the hash-table keys (cmd/cvar names) are extracted into an array and passed through qsort. The hash table code is used all over the place, so it's quite well tested .

[Baker]

Console variable system ... using sort of a Quake II method to access values directly (without having a mess of places with global extern).

Code: Select all

#include "engine.h"
#include "variables.h"		// Coutesy include

typedef struct var_t
{
	const char*		name;
	struct cmd_s*	next;

	const char*		defaultstring;
	text64			string;
	float			value;
	int				integer;
//	vec3_t			vector;
	voidfunc_t		afterChangeFunc;	// What to do if change occurs.  If anything.
} var_t;


#define MAX_NUM_VARIABLES 512

static var_t *topitem;
static var_t variables[MAX_NUM_VARIABLES];

var_t* Variable_FindName (const char* name)
{
	int i;
	
	for (i = 0; i < MAX_NUM_VARIABLES; i ++)
		if (variables[i].name && StringMatchCaseless (name, variables[i].name) )
			return &variables[i];

	// Didn't find
	return NULL;
}


// Central clearing house for any change of value
void Variable_ChangeValue (var_t* myVar, const char* newStringVal)
{
	StringLCopy (myVar->string, newStringVal);
	myVar->value = (float) atof (newStringVal);	// Convert string representation to floating point
	myVar->integer = atoi (newStringVal);

	if (!myVar->afterChangeFunc)
		return; // Doesn't have a change function
		
	myVar->afterChangeFunc ();
}

void Variable_Add (const char *name, const char *stringdefault, voidfunc_t afterChangeFunc)
{
	int i, newslot  = 0;
	var_t* newitem  = NULL;


	if (host.startup_complete) 
		Host_FatalError ("Variable_Add:  Can't add console variable \"%s\" after host startup complete", name);

	if (!name || Variable_FindName (name))
		Host_FatalError ("Variable_Add:  Can't add console variable \"%s\" already exists", name);
	
	// Find empty slot
	for (i = 0; i < MAX_NUM_VARIABLES; i ++)
		if (variables[i].name == NULL)
		{
			// Found slot ... fill it in
			newitem = &variables[i];
			
			// Note that this won't trigger the change function, afterChangeFunc not wired up yet
			Variable_ChangeValue (newitem, stringdefault);

			newitem->name = name;
			newitem->defaultstring = stringdefault;
			newitem->afterChangeFunc = afterChangeFunc;
			break;
		}

	if (!newitem)
		Host_FatalError ("Variable_Add: Can't add \"%s\" no empty slot (slots full?)", name);

	Link_Alphabetically ((link_t**)&topitem, (link_t*) newitem); // This function may change the topitem to newitem
}



/*
////////////////////////////////////////////////////////////////////////////////
  These just filter into Variable_ChangeValue
////////////////////////////////////////////////////////////////////////////////
*/

// Just pass the string along after determining which cvar it is.
void Variable_SetString (const char* name, const char* newStringVal)
{
	var_t* myVar = Variable_FindName (name);
	
	if (!myVar)
		Host_FatalError ("Variable_SetString: Unknown variable \"%s\" ", name);

	Variable_ChangeValue (myVar, newStringVal);

}

// Convert float to string and set.  Clean up string represent.
void Variable_SetFloat (const char* name, float newValue)
{
	char tempstr[128];
	size_t	i;

	SNPrintf (tempstr, sizeof(tempstr), "%f", newValue);

	// Strip off ending zeros
	for (i = strlen(tempstr) - 1 ; i > 0 && tempstr[i] == '0' ; i--)
		tempstr[i] = 0;

	// Strip off ending period
	if (tempstr[i] == '.')
		tempstr[i] = 0;

	Variable_SetString (name, tempstr); // That func does error handling, we don't need 

}

/*
////////////////////////////////////////////////////////////////////////////////
  Retrieve values.  Low performance functions.
////////////////////////////////////////////////////////////////////////////////
*/

const char* Variable_GetString (const char* name)
{
	var_t* myVar = Variable_FindName (name);
	
	if (!myVar)
		Host_FatalError ("Variable_GetString: variable \"%s\" failed", name);


	return myVar->string;
}


int Variable_GetInteger (const char* name)
{
	var_t* myVar = Variable_FindName (name);
	
	if (!myVar)
		Host_FatalError ("Variable_GetInteger: variable \"%s\" failed", name);


	return myVar->integer;
}

float Variable_GetFloat (const char* name)
{
	var_t* myVar = Variable_FindName (name);
	
	if (!myVar)
		Host_FatalError ("Variable_GetFloat: variable \"%s\" failed", name);


	return myVar->value;
}

/*
////////////////////////////////////////////////////////////////////////////////
  Retrieve values.  High performance method.
////////////////////////////////////////////////////////////////////////////////
*/

// Convert float to string and set.  See variables.h header for usage description
void* Variable_Hook (const char* name, const char* type)
{
	var_t* myVar = Variable_FindName (name);
	
	if (!myVar)
		Host_FatalError ("Variable_Hook: variable \"%s\" failed", name);

	if (StringMatch(type, "string") )
		return (void *)&myVar->string;
	if (StringMatch(type, "float") )
		return (void *)&myVar->value;
	if (StringMatch(type, "integer") )
		return (void *)&myVar->integer;

	Host_FatalError ("Variable_Hook: \"%s\" unknown hook type request", name);
	return NULL;
}





fbool Variable_Init (void)
{

	Variable_Add ("name", "player", NULL);

	return True;
}


void Variable_Shutdown (void)
{


}

Code: Select all

/*
////////////////////////////////////////////////////////////////////////////////
  To access the cvar, a function should hook into it.
	static float* fov_var_value;
	
	 if (!fov_var_value)
		fov_var_value = Variable_Hook_Float ("fov");

	printf ("%f\n", *fov_var_value);

////////////////////////////////////////////////////////////////////////////////
*/

[Baker]

QF uses hash tables for the commands and cvars. When a sorted list is needed (tab-completion, cmdlist, cvarlist), the hash-table keys (cmd/cvar names) are extracted into an array and passed through qsort. The hash table code is used all over the place, so it's quite well tested .

I haven't ever quite gotten console command and cvar completion work entirely "right" in anything yet --- what I mean is that it is completely intuitive and never does something annoying and completes everything within reason even after semicolons and such .... although maybe DirectQ does now and maybe QuakeForge.

I've have to see how QuakeForge does it

[p.s. I changed my opinion on curly brackets, hehe. I saw some ways FTE does parsing and tried to draft up a way to achieve something I want to try here ... kind of complex to explain at the moment ... and realized the curly brackets way is really the right way.]

[taniwha]

QF's completion isn't very smart yet. It certainly doesn't complete parameters, and I'm pretty sure it doesn't work after semicolons. It does, however, provide a list of all possible cvars and commands (including aliases?) that match the current "head". Certainly less annoying that id's very basic version.

Re parsing: may I suggest QF's plist code? It's actually quite self-contained (only dependencies are dstrings and hash tables, and all three modules are documented). There are some dependencies on sys (Sys_Printf etc) and mathlib/compat (min/max and strequal), but they're trivial to replace.

Property lists (include/QF/qfplist.h and libs/util/qfplist.c) (format docs)
Hash tables (include/QF/hash.h and libs/util/hash.c)
Dynamic Strings (include/QF/dstring.h and libs/util/dstring.c)

[Baker]

Re parsing: may I suggest QF's plist code?

I'm sure I'll be looking that over ... [Noticed you have some docs on the QuakeForge site that I want to read .... lucky +1. Also ... a joke ... an engine with any kind of documentation? No way. Well, FTE has some on the wiki. The ezQuake essentially has documentation via the QW wiki. I do understand why most engines don't have any kind of real documentation except some words in the readme ... the old limited time issue and "are you going to get stuff done or what?" factor. ]

Meanwhile, mild headache inducing look at code ...

Code: Select all

void SZ_Print (sizebuf_t *buf, char *data)
{
	int             len;
	len = Q_strlen(data)+1;
// byte * cast to keep VC++ happy
	if (buf->data[buf->cursize-1])  <------------------------- if cursize is 0 (empty buffer, that can't be right!  Can it?   It is reading outside the buffer!!)
		Q_memcpy ((byte *)SZ_GetSpace(buf, len),data,len); // no trailing 0   <--- ok, previous byte from cursor is not null.  Continue from cursor location.
	else
		Q_memcpy ((byte *)SZ_GetSpace(buf, len-1)-1,data,len); // write over trailing 0  <--- overwrite starting at the null terminator point.  The reason this should work is that "end-of-line" is a "\n" in the command buffer.  So we shouldn't encounter null characters --- maybe ever.
}

^^ I just don't like the way that is written one bit. Thankfully SZ_Print is called few times in the actual code and only for local stuff (command buffer, never network code).

It probably works. Maybe always by sheer coincidence? I'm referring to the "reading outside the buffer" part. That bugs me.

[Baker]

Woohoo! My own command interpreter + command buffer is working.

It doesn't do a hell of a lot (no parsing except grabbing a single line at a time and then removing it). But the code is short, easy to follow and by building from the ground up I have to understand every inch of what I am doing.

Still, 3 years ago ... even a basic command interpreter was a sci-fi concept to me. The nice little bonus from doing this is an increased understanding of client<-->server communications. Not much difference between the command buffer and filling up the buffer to send stuff to clients.

[mh]

DirectQ puts them all (cmds, cvars, aliases) into individual lists, then builds a master list containing pointers to the originals. This is sorted alphabetically (also using qsort) and then is searched using a binary search at runtime. There's a more elegant way of doing the original lists using C++ inheritance, and an STL container would be a nicer way of managing the whole thing, but right now it's not that big a problem.

The runtime binary search is actually important - otherwise executing the command buffer involves a lot of linear array walking and string comparison. With id1 this isn't so bad, but when you start adding a heap more cmds and cvars and when you run a mod that's stuffcmd-happy it can become a slightly noticeable problem.

Autocompletion is a bit of a nuisance; having it added to map, save, load, playdemo, timedemo and game is nice for the player, but it involves a lot of opening files, parsing them to get info out, validation, autocomplete list building, and other crap when a game is loaded - one of the reasons why DirectQ's game changing code is painfully slow compared to other engines. (Quakespasm has it on maps but it doesn't validate the map, so ammo box models get included in the list.)

[taniwha]

Yeah, the hash tables made a noticeable difference 12 years ago, and that was for overkill. They make an even bigger difference for alias-heavy mods (eg, TF).

The hassles you mention for tab-completion is one reason I never took QF beyond just command/cvar completion. Although a bit of extra work for the programmer (and potentially annoying for the user) is to not pre-scan the lookups (especially map and skybox), but to cache them individually on first usage. And then, when the gamedir changes, do a partial flush of the caches (this would be hard to get right and possibly not worth the hassle: full flush would be more robust). However, most commands and cvars would have fixed parameters (if any!) and thus wouldn't be an issue. For certain commands (and most cvars), just a simple value is expected: completion could just print a help message.

Handling tab completion for commands and cvars seems pretty obvious to me: have a "complete" callback in the command/cvar struct that either returns a completion string or dumps a message. Or both, even.

[mh]

have a "complete" callback in the command/cvar struct that either returns a completion string or dumps a message. Or both, even.

That's the way Doom 3 does it and it's quite nice. When I originally implemented it the idea didn't occur to me, unfortunately, so instead I trapped the "if (key == K_TAB)" condition and did some evil stuff there. It wouldn't be too difficult to change that part of the code, but it would be quite a job of work to modify all of the registration/etc and I'd probably prefer to unify the different types first.

The pre-scan to build the lists would remain painful. I had thought of a build-on-demand scheme, but the idea of some hitching while it searches for the next file, opens it, validates it, possibly decides to skip it (e.g. a map that doesn't contain an info_player_start) and restart the process if necessary was something that didn't appeal to me.

I had partially implemented on-demand in the menus, so for maps it would just initially get the name from the filesystem (which was pretty fast) then when you hover over it in a menu it would open the map and load all the necessary info for display to the user. That gave me a good idea of how significant the hitch in the console would be (it's bad enough to be unacceptable).

Currently my prescan loads everything for display in both the console and the menus. That's where the main slow down in game changing comes from. Parsing demos in particular is quite horrible.

I got rid of skybox autocompletion some time ago because of having to collapse up to 6 textures into one name for the list (some engines allow incomplete skyboxes and some mods rely on it), and all of the variant paths that skyboxes can be in. It was just too fragile. Ideally skyboxes would be always specified as a single .dds file containing a cubemap texture which would fix the worst of that, but it's unfair on others to expect them to support that too, and we're in a legacy data situation so it's too late to change now. Best we can hope for is not making things worse.

[taniwha]

That bad, huh? What about caching the file lists in the gamedirs? Each gamedir is fully indexed and any overlapping entries (caused by gamedir inheritance) are appropriately pruned. Rescan checks are done by checking the timestamp of the cache file against the timestamps of the directories in the gamedir tree.

[Spike]

thread it?

[taniwha]

Most of the hit is probably IO. If so, threading will only make it worse. Unless you mean putting it in a background thread: that certainly should ease the pain.

[Spike]

aye, put the prescan in a separate thread, so you can still actually load a map/join a server/etc while its busy waiting for the disk.
I hate loading screens. You'd best have a VERY good excuse for them if they last longer than 3 seconds.