A new standardised protocol?

[Spirit]

Hi Engineers!

I am clueless about network stuff, protocols, code etc. But I had the urge to post this. Surely I forgot to write down this or that and some things might sound stupid. Well, it's basically just an attempt to invoke a discussion.

Maps are getting bigger and bigger in terms of pure size, details, monsters and probably other things. The problem is that the common engines are not able to load those maps. Of course you can say "well, these are Quake's limits, if you break them I could not care less if my engine breaks". I totally understand this and actually I feel a bit like that myself (meaning I somehow like being limited).

However, people are making these maps, some of them are simply great and the players expect any map to work in any engine. Unless I overlooked some there are only 2 (3-4) engines that have introduced support by using an optional less limited network protocol so far.

Darkplaces supports "QUAKE, QUAKEDP, NEHAHRAMOVIE, DP1 and up (DP2, DP3 etc)" via sv_protocolname.
aguirRe's engines support "15 (std Q1) and 10000-10002" via sv_protocol.
Fitzquake is going to have a protocol for such maps in the upcoming release and it's another proprietary one.
MHQuake might be getting one too, at least it would not surprise me when I look at the to-do list.

As I briefly talked to metlslime (Fitzquake) about it, the probably biggest problem quickly came up:
<Spirit> it would be pretty cool if you engine coders could get together and decide on a common protocol
<metlslime> every engine coder agrees with you, and furthermore every engine coders thinks it should be their own protocol that becomes standard

While the internal limits (lightmaps, clipnodes etc I think) don't really affect anything "outside" your engine, the protocol limits (such as entities) surely do. It wouldn't be that much of a problem if we were talking about singleplayer only, it sure is a mess if you take multiplayer (obviously different protocols do not talk to each other) and demo recording/playback (obvious to coders, not so much to a player) in mind.

It's sad enough that things like fog, full- and overbrights or skybox loading are so differently supported, but those are faults made in the past I guess. These protocols are a pretty current thing so right now is the chance to set a common standard to benefit everyone:

The coders can brainstorm and create a truly kickass protocol.
The mappers and modders can let their creativity float more freely.
The players just see it work.

As a "benchmark" the Warpspasm would be perfect I guess.

PS: I am not sure about Nehahra but I guess if there was a standarised protocol with "higher limits" (and whatever else has to be changed) then Nehahra would be supported?

PPS: MH currently seems to be working on exterminating/increasing the internal limits, maybe he can put this into a nice tutorial/guide/article so others can catch up?

So to conclude:
How to create a protocol that more engines could adopt?
How to name it?
How to use it (variablename)?

[FrikaC]

Quick way to make a standard = make a good tutorial.

[mh]

Removing limits is a good thing, but I think you've hit the nail bang on the head when you mention two types of limits. While it can be done on both client and server side in isolation, getting them to talk together over the standard Q1 protocol is the challenge.

Ideas I have:

* Discard the standard protocol and write your own for Single Player games. No reason not to, there's no issues with communicating with other engines to consider, so far as client/server interaction is concerned you're effectively in a sandbox environment and can do what you want. You can even ignore the protocol altogether and just directly access the structs. The engine would of course detect when you're running Multi Player and revert to the standard protocol.

* Adapt the standard protocol without breaking it. Some things that spring to mind include sending multiple updates instead of just one, but I haven't done anything practical in terms of how this would work yet.

* Agree on an already standardised protocol. JohnFitz is right about everyone having their own ideas about what's best (my way is, of course ), and unfortunately it applies across the board. An already standardised GPL protocol seems to be the best way of getting everyone agreeing.

* Always always always ALWAYS keep the standard protocol in the engine as the "ultimate fallback".

If I can personally avoid a proprietary protocol, I will. Actually scratch that, I've no intention of implementing a proprietary protocol. Any engine will live or die by how well it interacts with what's already out there, and there's enough arch-traditionalists in the mapping and MP communities to kill anything new dead in the water before it even has a chance to show off how good (or bad) it is.

Regarding tutorials on removing limits, it's quite difficult to do for a number of reasons. Some of them involve some fairly extensive re-engineering in parts of the code that probably only LordHavoc has ever seen before, and also I've switched the engine over to C++ a while back, so some of my code would be pretty much unusable to anybody else. Finally, and as a specific example, removing the "-heapsize" limit requires a good understanding of how Q1 handles memory in the first place, as well as very careful consideration of every single memory allocation in the game; it's not the only one where this kinda thing is the case. Here's the guts of lightmaps, however, which dates back a few years and was originally done in C:

Code: Select all

// returns a texture number and the position inside it
lightmap_t *AllocBlock (int w, int h, int *x, int *y)
{
	int		i, j;
	int		best, best2;
	lightmap_t *lm;

	for (lm = lightmaps; lm; lm = lm->next)
	{
		best = LIGHTMAP_SIZE;

		for (i = 0; i < LIGHTMAP_SIZE - w; i++)
		{
			best2 = 0;

			for (j = 0; j < w; j++)
			{
				if (lm->allocated[i + j] >= best) break;
				if (lm->allocated[i + j] > best2) best2 = lm->allocated[i + j];
			}

			if (j == w)
			{
				// this is a valid spot
				*x = i;
				*y = best = best2;
			}
		}

		if (best + h > LIGHTMAP_SIZE)
			continue;

		for (i = 0; i < w; i++)
			lm->allocated[*x + i] = best + h;

		return lm;
	}

	// allocate a new lightmap
	lm = (lightmap_t *) malloc (sizeof (lightmap_t));

	// link it in
	lm->next = lightmaps;
	lightmaps = lm;

	// init it
	memset (lm, 0, sizeof (lm));
	memset (lm->allocated, 0, sizeof (lm->allocated));
	lm->modified = false;
	lm->texnum = GL_AllocTextures (1);

	// call recursively to use the new lightmap
	return AllocBlock (w, h, x, y);
}

Most of the rest of it involves changing the lightmap info over from individual globals to struct members, and destroying the structs on a map load.

[frag.machine]

Ok, let's start from the begin.

Let's put the problems properly pointed out by MH and metslime apart and try to define a clear goal to achieve with this new protocol - namely, to support bigger, far more populated and detailed maps.

So, in order to support this new protocol, an engine must also:

1) supports a bigger number of dynamic entities (at least how many ? 1024 ? 2048 ? as many as the available memory allows, like Darkplaces ?);
2) supports a bigger number of precached models (again, at least how many ? the current limit is 256) and sounds (same limit as models);
3) obviously, stretchs the internal engine limits for vertices, faces, leafs, possibly number of textures, dimensions, etc;
4) and of course, all this limit breaking eats up a lot more of memory, specially stack memory space, and depending of how many of this the engine already consumes we can start to bumping the head on the floor, requiring a major rewrite of some parts of the code to compensate this;

And so on...

As you see, the problem goes far beyond just defining a new protocol: there is a lot f work to do in the plumbing department to reach the point where one can say: "now, let me think about a new protocol". And of course, even if we manage to define a minimal set of messages to expand the current protocol or simply create a new one from scratch, there will always someone whinning like "waaaahh engine XYZ suxx0r cos doesnt support <insert non-trivial feature here> and cant run map blah".

However, that said I agree: we shouldn't be limiting ourselves by standards defined 13 or more years ago, when bandwidth, memory and CPU raw power were far more scarce. And if enough people agree to "sit" and define a minimal message set to this goal - running bigger maps like warpc.bsp - I surely would try to implement it in my engine.

[Spirit]

* Discard the standard protocol and write your own for Single Player games. No reason not to, there's no issues with communicating with other engines to consider, (...)

Demos! I think you forgot about demos here. Those would be incompatible, right?

Shame that the internal limits are so hard. :\

frag.machine: As I understand it, a new protocol could be chosen/created without caring about the "internal" limits. Of course it would be rather pointless if an engine would support a map's entities and models but not the actual brushwork, but I think small steps are possible (with the protocol being the first).

And if enough people agree to "sit" and define a minimal message set to this goal - running bigger maps like warpc.bsp - I surely would try to implement it in my engine.

Bringing the chairs was the intention of my post.

[frag.machine]

Demos! I think you forgot about demos here. Those would be incompatible, right?

Yeah. But a neutral demo format, detached from actual network protocol, could be created. I believe Quake 3 already does something like this.

frag.machine: As I understand it, a new protocol could be chosen/created without caring about the "internal" limits. Of course it would be rather pointless if an engine would support a map's entities and models but not the actual brushwork, but I think small steps are possible (with the protocol being the first).

Yes, you can always implement it without all required stuff. But user experience will vary a lot (from working flawlessly to not loading the test map at all) , and this defeats the objective of a common extended protocol.

[LordHavoc]

* Discard the standard protocol and write your own for Single Player games. No reason not to, there's no issues with communicating with other engines to consider, so far as client/server interaction is concerned you're effectively in a sandbox environment and can do what you want. You can even ignore the protocol altogether and just directly access the structs. The engine would of course detect when you're running Multi Player and revert to the standard protocol.

You completely forgot demo recording - people like demos to play back in multiple engines too.

* Adapt the standard protocol without breaking it. Some things that spring to mind include sending multiple updates instead of just one, but I haven't done anything practical in terms of how this would work yet.

You can avoid packet overflows in the standard quake protocol by sending mulitple packets with entity updates but only the first one having svc_time in it.

However this is just a way to use more bandwidth - people don't like games using huge amounts of bandwidth (and at the default server sys_ticrate of 0.05 it's possible to use 20KB/s in standard quake protocol, or 28KB/s in darkplaces protocol, without using multiple packets per frame).

* Agree on an already standardised protocol. JohnFitz is right about everyone having their own ideas about what's best (my way is, of course ), and unfortunately it applies across the board. An already standardised GPL protocol seems to be the best way of getting everyone agreeing.

All the protocols I've seen have been vastly inferior to DP7 protocol, which has a redesigned entity protocol that allows tens of thousands of entities in one room without collapsing, it just updates the nearby ones more often.

Note that FTEQW also supports DP7 protocol, which makes it more standard than the others.

* Always always always ALWAYS keep the standard protocol in the engine as the "ultimate fallback".

A useless fallback when it comes to extended mods however.

If I can personally avoid a proprietary protocol, I will. Actually scratch that, I've no intention of implementing a proprietary protocol. Any engine will live or die by how well it interacts with what's already out there, and there's enough arch-traditionalists in the mapping and MP communities to kill anything new dead in the water before it even has a chance to show off how good (or bad) it is.

Nehahra showed that not to be the case - it requires its own protocol for the demos, and another protocol for multiplayer.

Warpspasm has also shown that to not be the case - its demos require Arguirre's protocols, and Arguirre's modified bsp support (he broke the limits, nearly doubling the limits on certain signed shorts by treating them as unsigned with some special cases).

DarkPlaces supports all engine protocols except:
FTEQW's extended QW protocol - I do not know the details of this, but it uses the same QW protocol ID, it negotiates extra features on connect, I don't know how well demo playback works.
TomazQuake - this unfortunately does not match up with the QUAKEDP protocol it claims to use, it has some different entity encodings, and it's not easy to fix this (as identifying the demos is not easy without having a unique protocol number).

I have not seen many TomazQuake demos and it was never used for servers very much, so I'm not overly concerned about it.

FTEQW's demos haven't been seen in the wild either to my knowledge, there are occasionally servers however - but they serve up different protocols to each client based on client identification (basically all other engines receive the stock QUAKE protocol from it).

[Baker]

All the protocols I've seen have been vastly inferior to DP7 protocol, which has a redesigned entity protocol that allows tens of thousands of entities in one room without collapsing, it just updates the nearby ones more often.

Note that FTEQW also supports DP7 protocol, which makes it more standard than the others.

A couple of years ago you mused about the possibility of creating a DP7 tutorial.

I'd be willing to take such a tutorial an create prototype implementations of the protocol with ProQuake, JoeQuake, FitzQuake and Qrack. I've probably privately modded each of those engines 70 times each over the last year. Provided that it worked, it would ease the transition for the respective authors to implement it into official builds if they so choose (and I can't see any reason why they wouldn't).

[LordHavoc]

A couple of years ago you mused about the possibility of creating a DP7 tutorial.

That somehow frightens me now

Although the protocol part is really easy to copy over, it's the slight differences in entity handling elsewhere that make it annoying.

[Baker]

A couple of years ago you mused about the possibility of creating a DP7 tutorial.

That somehow frightens me now

Haha

[frag.machine]

Provided we have some good public documentation on DP7 and some eventual help from LordHavoc to dismiss doubts, it's good enough to me. If documenting the whole DP7 protocol is a problem for any reason, we could define a minimal subset of DP7 messages to add to the original protocol, and thus derivate an hybrid solution. What do you think, LH ?

[Baker]

I wasn't kidding at all about my enthusiasm for assisting however I can. Getting commonality among the engines in a way to raise the baseline minimum has always been something that I thought was important.

Perhaps as no surprise, every commonly used engine is still being developed (including JoeQuake where a quiet new release was done I'm think around the fall of last year).

[frag.machine]

Meanwhile, I think we could discuss some techniques to optimize bandwidth usage. LordHavoc already mentioned sending less update packets to far entities, which already should make significant difference. I remember reading about how Quake 4 is even more aggressive about optimizing such far updates, not sending info like models or animation frames. This could definitively be used also for not directly visible entities into the fat PVS (provided the computational cost to detect this is reasonable ), like entities completely behind walls. And I agree with MH about keeping the original protocol as a last resort fall back (at least for demos recording/playback and connecting to legacy engines).

[mh]

I've two ideas coming right up!

Gonna create a new thread for them though, which seems a sensible thing, so as not to clutter up one discussion with another and end up with a confusing mess of cross-posting.

[metlslime]

Havoc, didn't you say a while ago that there was something you wanted to add to dp7 that justified making a dp8?