Quake engine random observations

[Baker]

Note: This thread will probably bore anyone advanced.

I pulled the Quake engine apart into the various pieces and you really have to admire how it was put together and how well it was designed.

GLQuake in particular isn't actually as nice and clean as WinQuake. GLQuake do everything tidy like it should have, like unloadiing textures.

When looking at the engine, it really goes like this ...

1. Main program loop
---- 2. Collect various startup info (command line args, working dir)
---- 3. Initialize well split out components, like the memory, video, sound, input, etc.
----------- 3a. Operating system specific goes to sys_ whatever (generally)
----------- 3b. Non-operating system specific stuff lives about everywhere else
---- 4. Main Loop
---------- 4a. Run a Host_Frame (time check to regulate frames per second)
---------- 4b. If system event (key, mouse, ALT-TAB) delegate to MainWndProc (on Windows)
---------- 4c. If client and/or server is active, do those things (client: send intentions to server, read messages, draw things ... server: do physics, giving every entity a turn)
---- 5. Exit initiated if "quit" or a Sys_Error

Random topic: Host_Error versus Sys_Error

Until recently, I wasn't absolutely clear on the difference between a Host_Error and a Sys_Error.

A Host_Error shutdowns things that have been initialized, then calls Sys_Error (which does an unabbreviated shutdown).

A Sys_Error is meant to be used either when something so bad that a Host_Error is unreliable (out of memory!, a recursive error or such) or trouble happens so quickly into progression of start up that nothing of substance has happened.

Random topic: The Hunk

I thought I'd describe Quake memory management real quick, even though this isn't really something I've been thinking about lately. When I started engine coding, since the engine isn't documented it took a while to understand what the hunk was and how it works.

The Hunk is just a big memory buffer --- like 32 MB or 128 MB in engines without dynamic memory management --- that gets allocated. Some important stuff gets allocated into the hunk early (including other special allocations). Then after all the loading of substance is done, Quake makes a marker of the top point.

All game related data starts at this top point. And since Quake is designed to be a complete reset between maps, Quake just discards everything above that saved "top marker" on a new map.

When loading, sometimes Quake needs a bit of memory temporarily, will make a new top marker, load stuff and do things and then reset back to that top marker point as if it never happened.

The model cache is a bit trickier, Quake goes through the models and appears to mark the ones that are still needed. The same with sounds.

Random topic: Platform independence

Although I have no knowledge how Quake's source code might have looked prior to the source code release, Quake looks like it was built for platform independence from day 1.

All the system specific stuff is nicely shielded away in non-system specific functions. Although there are a ton of #ifdef _WIN32 items throughout the source code.

Maybe the most interesting thing about the Quake source is the use of "video modes" to serve as a platform independent way to communicate video mode availability, with mode 0 being the current settings for whatever a -window windowed mode window is using. Or this could be a legacy approach for DOS that carried forward or could even be just the standard way operating systems like to talk about fullscreen modes (aka dos mode 13 -- VGA 320x200 256 color mode).

DarkPlaces

One of the most interesting things about DarkPlaces historical development from my perspective is that it is rather clear that LordHavoc thought of Quake as a game engine.

In even super-old builds of DarkPlaces, LordHavoc addresses a ton of issues that are small ... yet very annoying to proper behavior.

For example, DarkPlaces actually resets the settings when a user does "set defaults" in the menu. If you do this in GLQuake, it doesn't actually reset but does something worse and although not random, combined with a users preferences, it changes the settings to things that are neither the defaults nor what the user selected but somewhere in-between.

DarkPlaces has a lot of behavior corrections like that all over the place. Like the ability to play multi-map demos by ignoring reconnect. Or like having shadows equally affect brush sub-models (lifts, platforms ... things that are not part of the unmoving world).

Or clearing the demo queue. Startdemos is annoying and likes to restart itself in certain situations.

What Really Sets Quake Apart

Quake in many ways is just a simple game engine with some components that would be very difficult and require broad mastery for a single person to write from scratch.

Quake's inclusion of the progs interpreter, which runs a little program inside itself is, that runs the progs.dat compiled from QuakeC is one thing that really makes Quake different than previous or future game engines. (I've read enough about Quake 2 incompatible game DLLs in this forum that only work with a single Quake 2 engine and how many are closed source.)

It wouldn't be hard to take the Quake engine strip out some things and make it in a 2D board game engine (as an example).

If anything from inspecting all the little components of the Quake engine, the one thing I'd like to learn more about in the future is peer-to-peer networking and how the heck MMORPGs do their networking to support 10,000 players.

The network code in Quake is basic, obsolete and inefficient. But even efficienciently, I'm wondering how an MMORPG server can support 1,000 players or 10,000 players. My initial guess would be a couple of layers of prediction distributed out amongst a few servers. Maybe I am wrong. (I don't have any interest in making MMORPGs, I'm just curious about the networking now ...)

I might post some more random things in here from time to time. Might be interesting. Might not be.

[mankrip]

Thanks for all of this, specially about the hunk.

Quake's memory management is something I still suck at.

[daemonicky]

Quake looks like it was built for platform independence from day 1.
----
So does Wolfenstein and Doom source codes. I think Quake is iteration of Doom, and Doom is of Wolf. It may be that Carmack did similar style at Commander Keen ... And Carmack IIRC learned lot of things in the beginning from Abrash http://www.firingsquad.com/features/carmack/page7.asp .

[mh]

Quake as an interation of Doom - yes and no.

Carmack himself has said that each new tech generation starts with an empty editor window. Now, saying it and actually doing it are two different things for sure, but in the absense of any other info we must work with what we have.

It's obvious that when Quake was being written that Carmack didn't really know what he was doing a lot of the time. The switch from 2D (or 2.5D) to real arbitrary 3D is non-trivial, and not many trailblazers had done it before. The switch again to hardware acceleration (which uses a completely different set of rules) was more or less something completely new.

Quake was the first attempt, Quake II was taking that first attempt and cleaning out all the cruft and mistakes. Odd little hangovers like the WAD file format quite likely wouldn't have survived another few months of development time if ID had had that luxury. That's one reason why the Quake engine doesn't get an "ID Tech" number - it's in the same technology generation as Quake II but is really a weird intermediate stage, an experimental first cut that didn't quite work out.

For MMORPGs, while I don't know the exact details, I would expect that a multi-master approach is the way to go. Each server manages a "region" which contains fewer players, and which needs more frequent updates, updates of info on other "regions" occur between servers on a less regular basis and can be significantly more lightweight. Quake (and it's descendents) have created this obsession with single-master dedicated servers, but they're not really the best way to handle things in real life. Other large distributed networks use multi-master (read up on how MS Active Directory replication works for a good example, and try to view it in terms of "if this was a game...") so it's the logical way to go.

Quake's memory management is really kinda neat. The hunk system is more or less immune to memory leaks, greatly simplifies the who memory management process, and the only thing you really need to be careful of is to clear any pointers that refer to regions of it between maps. A fixed size for it does suck, and - at least on Windows - replacing it's initial malloc with a VirtualAlloc is highly recommended. A portable virtual memory library that does all the cool stuff that VirtualAlloc does and Just Works on other platforms is sorely needed - mmap andd sbrk are shockingly primitive by comparison.

Cache memory was done away with in Quake II; the only purpose it serves is to make loading faster by not having to reload certain object types on a map change. It does complicate things somewhat, but I guess it was an acceptable tradeoff given the spec of machine that Quake had to run on in 1996.

One of the annoying things is how much intertwining, interdependency and entanglement there can be between the various subsystems in Quake. It can be difficult to make a clean separation between subsystems, which opens lots of doors for subtle (or not so subtle) bugs as you evolve your own code. Abuse of globals has a lot to do with that. Quake II's enforced separation of the renderer from the rest of the engine code helped a little there too.

Quake's VM is also neat if a little primitive. It works fine under moderate load, but very quickly becomes the single major bottleneck as entity counts and code complexity ramp up. I've seen framerates in some maps come close enough to quadrupling as soon as the server is paused (by e.g. dropping the console when running SP). Again, there are tradeoffs involved here, and using non-portable DLLs does remove this bottleneck. The progs interpreter itself could use a little JITting and it's something I'm personally interested in exploring in the future.

[frag.machine]

Quake's VM is also neat if a little primitive. It works fine under moderate load, but very quickly becomes the single major bottleneck as entity counts and code complexity ramp up. I've seen framerates in some maps come close enough to quadrupling as soon as the server is paused (by e.g. dropping the console when running SP). Again, there are tradeoffs involved here, and using non-portable DLLs does remove this bottleneck. The progs interpreter itself could use a little JITting and it's something I'm personally interested in exploring in the future.

That's an interesting - and not explored enough - topic: the Quake VM performance.

I've made some measurements using VS2008 profiler over a unmodified FitzQuake 0.85 engine with a custom SP map (1024 monster_knight inside a box), and got some surprisingly data. The VM runtime itself is actually quite fast; however, the engine spends a lot of CPU cycles inside a bunch of builtins - specifically, traceline(). Diving into this builtin will show us that most (~80% or so) of the CPU cycles are spent running SV_RecursiveHullCheck(). So, long story short: improve SV_RecursiveHullCheck and you'll see the effects in QuakeC.

[Irritant]

Carmack himself has said that each new tech generation starts with an empty editor window. Now, saying it and actually doing it are two different things for sure, but in the absense of any other info we must work with what we have.

Quake was the first attempt, Quake II was taking that first attempt and cleaning out all the cruft and mistakes.

Also, Q3 shares a ton of code from Q2 as well. I kind of think that Q1-Q3 really should have been considered one "iteration", which each engine showing a natural evolution and cleanup. It will be interesting when Tech4 is released, so see if that is still the case or that it is truly completely new.

[mh]

Also, Q3 shares a ton of code from Q2 as well. I kind of think that Q1-Q3 really should have been considered one "iteration", which each engine showing a natural evolution and cleanup. It will be interesting when Tech4 is released, so see if that is still the case or that it is truly completely new.

Doubly so as it more or less started out as Q3A with a new renderer. It's the one ID code release I've been looking forward to the most for some time now.

[r00k]

You mean the Quake 3 TEAM ARENA?

[leileilol]

Team Arena isn't different in engine at all - all the different that brought was a broken new ui qvm. IT should also be noted the Q3 engine started around mid-97 before Quake2 was even finished. It was Carmack's "okay, i've been doing things wrong all this time, now its time for me to Do The Right Thing" attempt. It's existence was definitely not evolutionary from Q1 or Q2.

I absolutely loathe it when Team Arena is referred to as a new Q3 updated engine! It's like claiming Source engine is from scratch.

[mh]

http://www.team5150.com/~andrew/carmack ... _1997.html

Just search for the word "Trinity" - that's what became the Q3A engine (hence the tr_ prefix on all of it's renderer source files).

[leileilol]

trinity idea: model light haze around every emiter

haha, there's commented out functions for flares on dynamic lights that do exactly this. though the way he might mention it might be like the funny volumetric fog flare effects Doom3 had. Hardwar did this a LOT in 1998 so i don't know...

trinity: continuous textures, surface cache on all

megatexture ambition that early?

trinity: shimmering heat atmospheric effects

shelved for Doom 3, though it should also be mentioned Chasm: The Rift semi-faked this effect in their smoke transparency lookup routines.

I am going to rapidly wean myself off of working with quake so I can concentrate fully on new directions. The evolution of the Q2 codebase will be left to John Cash (until the mission pack ships) and Zoid.

Yeah, not the same iteration.

[Spike]

so that's what the tr_ stuff was for!

the key archetectural difference between q2 and q3 engines is that q3 made no attempt to support software rendering.

the key difference between q1 and q2 is that... erm. yeah, well, there's no hull issues in q2. urm. that's basically it. Okay, so the netcode is a slight reversion from QW, making it smoother like NQ, but also a bit more laggy. Some alpha blending, I suppose, but...
The rest is just code cleanup (switching renderers on the fly, no ms2 generation for models, smoother netcode, no weird skys, etc, shame neither work without disabling/fixing NX permissions).
Sure, Q2 fixes some bugs, but it feels very much like a minor release between Q1 and Q3, closer to Q1.

[Irritant]

the key difference between q1 and q2 is that... erm. yeah, well, there's no hull issues in q2. urm. that's basically it. Okay, so the netcode is a slight reversion from QW, making it smoother like NQ, but also a bit more laggy. Some alpha blending, I suppose, but...
The rest is just code cleanup (switching renderers on the fly, no ms2 generation for models, smoother netcode, no weird skys, etc, shame neither work without disabling/fixing NX permissions).
Sure, Q2 fixes some bugs, but it feels very much like a minor release between Q1 and Q3, closer to Q1.

Well Q2 also ditched QC, moved the game code into a DLL, added colored lighting, frame lerping, etc. While it's technically closer to Q1(and both are considered idTech2 anyway), you can see a progression towards what Q3 was going to be, even if Q3 was started parallel to the end of Q2's dev cycle.

[Spike]

gamecode in a dll is not always an improvement, for instance, its less of an abstraction than qc. Yes, the gamecode changed significantly, but not the engine itself. the irony with dlls vs qvms in q3 is the engine licensees that just used dlls regardless, much like valve did.
coloured lighting wasn't supported in software rendering. its actually fairly straight forward with gl, although the lighting tool was significantly different, the tools don't directly distinguish the engine itself. :P
frame interpolation was piggybacked on the movement interpolation. this requires the gamecode to run at a fixed 10 fps, as well as the server->client packetrate (the reversion I mentioned).

multiple renderers without restarting the engine are imho more significant to the engine itself than the changes you mentioned, especially if you consider the code cleanups that went with it (thank god they removed mgl in the process).

[mh]

The interesting thing about Q2's gamecode is that it essentially started as a straight port of QC to C.

Might be interesting to experiment with a converter sometime; that might make the JIT stuff from Q3A usable in Q1. All theoretical talk, and maybe there are valid reasons not to, but interesting nonetheless.