Engine Philosophy Discussion

[frag.machine]

From what I read in "Masters of Doom", it was quite the opposite. Carmack resented a lot the absence of Romero during the engine development, because he was too busy running Doom tournaments, supervising the development of Hexen and playing the rockstar to the press and fans. Brian Hook was hired to help Carmack in part due this situation, and later Romero was kicked because of this, too.

Well, I would be curious as to the origin of what seems like a superfluous quantity of new features in the Quake engine.

Like what ? Apart #ifdef'd stuff like skyboxes and fog (much likely added during Quake2 pre-development phase), the original source looks quite... basic to me. Actually, there were a lot of easy to implement improvements (like alias interpolation) left out, to the point you could call Quake a "spartan" 3D engine. Let's not forget that we are talking about a pioneer approach to a truly 3D engine, born as a software render with an afterthought port to OpenGL. And most of all, let's not forget that Quake's development was anything but "easy". It's too easy nowadays to forget back then how limited was hardware and how hard was to conceive, design and program something so advanced as a 3D engine from scratch.

All one has to do is really start digging around in the code and you quickly run across hordes of features that could be categorized as extraneous. It is almost over the top.

I'm sorry, I fail to see such features. Could you point them ?

[frag.machine]

Who needs goto when you've got:

Code: Select all

do
{
   ...
   ...
   if (condition1) break;
   if (condition2) continue;
   ...
   ...
} while (false);

goto is much cleaner and easier to follow than constructs like that, IMO.

It's easy to find goto's harmless when you weren't a programmer in the ye olde BASIC days. GOTO is and ALWAYS will be evil, trust me. You'll always be able to write clean and elegant code without resorting to goto's. Just think a bit more about what you're doing.

[Baker]

I'm sorry, I fail to see such features. Could you point them ?

QuakeC (the game logic could have been built into the executable.)
The Console. Case in point, Half-Life by default does even let you access the console and you can play start to finish without it.
Aliases. Or even the idea of +alias and -alias being triggered when a key is pressed and released.
Stuffcmds. Mentioned earlier.
Colormapping. I haven't played Doom in a bit, especially not the multiplayer, didn't it only have 4 colors like (red, green, blue and maybe gray?) I suspect they were not colormapped (yeah I am DooM ignorant).
Support for multiple video modes. Some old games supported 320x200 and that was that.
Mouse support. Back then, most games didn't have this.
CD support. Back then, most games didn't have this.
Multi-Platform support. Like endianess checks and so forth.
TCP/IP support. Wasn't IPX really the only way that networking was actually done back in that day. I have more recent 1990s games that only supported IPX. I mean Quake 2 came out in late 1997 and if IPX was the norm, Quake would have been a success during the commericial development cycle like other games from that era that only support IPX (some IPX only games exist well into 1998, for sure).

This fails to mention support for things like lightmaps, the idea of vis, how Quake supports file bundling like via .wad and via .pak.

Quake didn't need to support a complicated config.cfg either. Let alone support comments for it. gamedir support wouldn't be required either. Or a complicated filesystem to find files in multiple .pak files and multiple directories.

Quake in theory, at least if game scope was more limited, wouldn't have needed to support the idea of each level having its own models and sounds. They could have all been precached or something more hard-wired.

If you look at some console games today, you don't even need to support "options" and back in that day many games didn't support "customize controls" ... you used whatever keys the game manual said.

If you go on the above, Quake doesn't need a cvar system or a command interpreter. For starters.

The entire game could have been a commericial success without any kind of support except for DOS. Windows 95 played DOS games (usually?).

[Irritant]

Those look like normal advancements rather than extraneous features.

Also, if you haven't read Masters of Doom, do so. It was definitely more of a case of Carmack coming up with cool things and springing them on the rest of the team than demands being made by the team.

[dreadlorde]

The moral of the story is that the whole "Unix philosophy" thing is really only appropriate for certain kinds of program - specifically command-line tools intended to be invoked from shell scripts, or similar small throwaway homebrew tools where performance isn't a constraint..

I sincerely disagree. "Do one thing and do it right" can be applied to any program, and I think Quake does exactly this. There's no silly level editor in the engine, the game doesn't have a PBX system built in so people can yell racial epithets at each other, &c.

The features mentioned above can be used through the game indirectly with a simple message passing system. The best example of this Plan 9's plumber(4). The plumber receives messages and plumb(6)'s them to other processes. One could easily extend this system to say, start ventrilo or teamspeak or skype when connecting to a server, &c.

my 2¢.

[FrikaC]

Actually, there were a lot of easy to implement improvements (like alias interpolation) left out, to the point you could call Quake a "spartan" 3D engine.

[frag.machine]

I'm sorry, I fail to see such features. Could you point them ?

QuakeC (the game logic could have been built into the executable.)
The Console. Case in point, Half-Life by default does even let you access the console and you can play start to finish without it.
Aliases. Or even the idea of +alias and -alias being triggered when a key is pressed and released.
Stuffcmds. Mentioned earlier.
Colormapping. I haven't played Doom in a bit, especially not the multiplayer, didn't it only have 4 colors like (red, green, blue and maybe gray?) I suspect they were not colormapped (yeah I am DooM ignorant).
Support for multiple video modes. Some old games supported 320x200 and that was that.
Mouse support. Back then, most games didn't have this.
CD support. Back then, most games didn't have this.
Multi-Platform support. Like endianess checks and so forth.
TCP/IP support. Wasn't IPX really the only way that networking was actually done back in that day. I have more recent 1990s games that only supported IPX. I mean Quake 2 came out in late 1997 and if IPX was the norm, Quake would have been a success during the commericial development cycle like other games from that era that only support IPX (some IPX only games exist well into 1998, for sure).

This fails to mention support for things like lightmaps, the idea of vis, how Quake supports file bundling like via .wad and via .pak.

Quake didn't need to support a complicated config.cfg either. Let alone support comments for it. gamedir support wouldn't be required either. Or a complicated filesystem to find files in multiple .pak files and multiple directories.

Quake in theory, at least if game scope was more limited, wouldn't have needed to support the idea of each level having its own models and sounds. They could have all been precached or something more hard-wired.

If you look at some console games today, you don't even need to support "options" and back in that day many games didn't support "customize controls" ... you used whatever keys the game manual said.

If you go on the above, Quake doesn't need a cvar system or a command interpreter. For starters.

The entire game could have been a commericial success without any kind of support except for DOS. Windows 95 played DOS games (usually?).

Oh man... I'm sorry, looks like my fault understanding what you were meaning with "superfluous" features. My bad.

[Baker]

Oh man... I'm sorry, looks like my fault understanding what you were meaning with "superfluous" features. My bad.

Eh? I'm a rather casual easy going and I get the limits of internet communication. I don't worry about stuff like that, besides "failure to communicate" is always the fault of communicator (ME!).

I could name a few more. FOV support even in software Quake, fullbright texture support, waterwarp ... even the ability to swim. The game could have been completed and successfully without any of that.

I think the sheer amount of what looks like a single generation of engine improvement (even if my list is a bit off in the accuracy dept., the end conclusion about the amount of advancement is true).

Those look like normal advancements rather than extraneous features.

At least in my head, the reason I find the feature advancement in Quake vs. "the past id works" fascinating is because it was "non-linear".

It wasn't like there was a goal and they efficiently met the feature set for that goal. You have all kinds of sporadic and whimsical improvements ... it isn't focused.

Also, if you haven't read Masters of Doom, do so. It was definitely more of a case of Carmack coming up with cool things and springing them on the rest of the team than demands being made by the team.

I have that book courtesy of Spirit [thanks Spirit, I enjoyed it more than I could have anticipated] and read it cover-to-cover.

I do understand you feel that the advances between DooM and Quake to be evolutionary. And maybe you are right.

My perspective (as someone six rungs down the engine coding ladder as yourself) could be a little different considering you mostly work with the Q2 engine and I work mostly with the Q1 engine.

I see QuakeC as a remarkable and possibly coding-universe-unique attempt to platform neutralize what is effectively an interpreted language built within an engine (with external tools, of course).

Other games uses .dlls (or equivalent or ... well ... worse like hardcoded behavior) including what for you is so obvious as to not even need to say (but some Q1 people reading this wouldn't know) Quake 2.

If you are working with Quake 1 engine code, you always are aware that the engine has a particular unique feature. Every single day (that you aren't doing something very narrow). For me this weirdness causes me to not take for granted anything that is a feature, I guess since there is that one highly remarkable and unusual Quake feature, I don't take for granted the other features.

/My thoughts on this

[Spike]

quakec isn't unique. UT has uscript, for example. q3's qvms are not really drastically different, and many many many games utilize lua for gamecode type stuff.

[Lardarse]

True, but I'd argue that QuakeC is syntactically unique compared to those others, given its quirks. (No ints being the main one; the way functions are declared is slightly weird, but makes sense once you think about it)

[dreadlorde]

the way functions are declared is slightly weird, but makes sense once you think about it

Not really, when calling a function, you don't pass arguments to the type of the function, you pass them to the function.

[daemonicky]

No need for :
QuakeC (the game logic could have been built into the executable.)
The Console.
Aliases.
Colormapping. I haven't played Doom in a bit, especially not the multiplayer, didn't it only have 4 colors like (red, green, blue and maybe gray?) I suspect they were not colormapped (yeah I am DooM ignorant).
Support for multiple video modes.
Mouse support.
doesn't need a cvar system
CD support.
Multi-Platform support. Like endianess checks and so forth.
This fails to mention support for things like lightmaps
, the idea of vis, how Quake supports file bundling like via .wad and via .pak.
Quake didn't need to support a complicated config.cfg either.
Let alone support comments for it. gamedir support wouldn't be required either. Or a complicated filesystem to find files in multiple .pak files and multiple directories.
didn't support "customize controls" ... you used whatever keys the game manual said.

(I edited the quote)

True.

But, consider how some of them make developing easier (aliases, mouse, cvar, pak, config, quakec).

But it is true, they could build it like Duke 3d.

Actually I would like to know what difference in speed/ease of development would it make if they did just game, not an engine.

I guess they knew, this is going to be engine used by community and I guess they did the most (quake goes gpl, quake c, abrash wrotes about quake development http://www.bluesnews.com/abrash/ in Black Book http://bit.ly/lvoo3h?r=bb ) to bring people to them. I guess they knew gaming is social - there probably were some Doom fans meetings or something. I mean Facebook is built on idea that people do like socializing and sharing; on Youtube you can get movies and whole albums shared ... I just think that modding w

Quake in theory, at least if game scope was more limited, wouldn't have needed to support the idea of each level having its own models and sounds. They could have all been precached or something more hard-wired.

True.

Adding : Tomb Raider had each level with its own sounds and models, and they were packed too.

Comparison : each level of Quake can choose arbitrary models from shared base; in TR each level has its own copy of models, therefore it is harder to make changes across all the levels (unless working version of TR level file is different from release version). Quake is therefore superior.

Anyway, some good database (which neither TR nor Quake has, they have hierarchical ones for models, sounds etc) and version control system would make it smooth.

the way functions are declared is slightly weird, but makes sense once you think about it

Not really, when calling a function, you don't pass arguments to the type of the function, you pass them to the function.

I guess that the reason why functions are declared in Quake this way is :

1. For ease of coding.

I guess he can recognize there is function declaration/definition very fast because after type there can not be '('otherwise. I do not think it is much important, since QC is compiled to bytecode anyway and writing more general compiler to bytecode ... and some QC to bytecode could be probably written in some standard tool like yacc (never used it though).

Another thing is that you can write script to parse QC code and it would be easy one because of these properties. On the other hand parsing some part of C might be easy too ...

Anyways. I think it is not much of a big point, it is just mine guess.



Code: Select all

FUNCTION = FUNCTION_HEADER '=' '{' STATEMENT* '}'
FUNCTION_HEADER = TYPE '(' VARIABLE_LIST ')' NAME
FUNCTION_DECLARATION = FUNCTION_HEADER ';''

2. For clarity.

To me it is cleaner to read function name from "blablbablablababa blablbablablababa my_function" than from " blablbablablababa my_function blablbablablababa ", because the name of function gets lost. It is not a matter if you have good syntax hightlighting.

I think that someone of K&R+Thompson somewhere said that they inteninaly did declaration syntax similar to use syntax ... There are several permutations on how to declare function (permutation of positions of name, body, return type, parameters).

QuakeC (the game logic could have been built into the executable.)

True. I analyzed a little and here is the result (with a flamish name as bonus ). Yes, I am biased, you can tell .
Basically I found these variables
speed ............ how fast it is
portability ...... you can run in anywhere
community .... people who share, play, use, buy, work on cool stuff, socialize because of quakec
ease of coding ... obvious
debugability .... ease of finding a mistake
testing ............ looking how it works. Is thre a mistake? ease of testing
+- means how is it improved/worsened in red there is subject, that is DLL or QUAKE C.

0.1 loss (10 percent loss) is serious bussiness

http://www.team5150.com/~andrew/carmack/johnc_plan_1997.html#d19970313 (DLL)
"... considering dropping qc in Quake 2 ... wasn't an option when we had to support dos ...I could improve the language, or just adopt a real language like java, but the simplest thing (for JC to do) to do would be just use native code ... be more efficient as a dll (+speed)... As we do more sophisticated game logic (What? Where? Is he serious?), efficiency becomes more and more important ... but for full size game levels it will likely be at least a 5% to 10% overall speed improvement (NOTE: same difference does not matter in case of z-buffering) ... It would be non-portable(-portability). I am dreading the reaction to this from the linux community (-community)... it is really only an issue for servers (-admins)... it would cut out a number of people that currently enjoy hacking quake (-community)... You could debug your patch in a real debugger! (+debugging) (-testing)Yipee! (yahoo!)"

http://www.bluesnews.com/abrash/chap70.shtml ( QUAKE C):
"The Quake server maintains the game’s timebase and state, performs object movement and physics, and runs monster AI. The most interesting aspect of the server is the extent to which it’s data-driven. Each level (the current “world”) is completely described by object locations and types, wall locations, and so on stored in a database loaded from disk. The behavior of objects and monsters is likewise externally programmable, controlled by functions written in a built-in interpreted language, Quake-C. Quake is controlled by its external database to the extent that not only have people been able to make new levels, but they’ve also been able to add new game elements, such as smart rockets that track people, planes that can be climbed into and flown, and alerters that stick to players and screech “Here I am!”--(+community)all without writing a single line of C or assembly code(+ease of writing). This flexibility not only makes Quake a great platform for creativity, but also helped a great deal as we developed the game, because it allowed us to try out changes without having to recompile the program. Indeed, levels and Quake-C programs can be reloaded and tried out without even exiting Quake(+testing)."

http://www.bluesnews.com/abrash/chap70.shtml (10 percent difference)
"There’s a cost, to be sure; z-buffering is slower than non-z-buffered drawing, and the z-buffer has to be initialized to match the visible world pixels, at a cost of about 10% of Quake’s performance (-speed)[. That cost is, however, more than repaid by the simplicity and accuracy of z-buffering, which saves us from having to perform complex clipping and sorting operations in order to draw entities properly, and gives us flawless drawing under all circumstances."

http://www.phatcode.net/res/224/files/h ... 70-09.html (DLL)
""Another likely change in Quake 2 is a shift from interpreted Quake-C code for game logic to compiled DLLs. Part of the incentive here is performance(+speed)—interpretation isn’t cheap—and part is debugging, because the standard debugger can be used with DLLs. The drawback, of course, is portability(-portability); Quake-C program files are completely portable to any platform Quake runs on, with no modification or recompilation, but DLLs compiled for Win32 require a real porting effort to run anywhere else. Our thinking here is that there are almost no non-console platforms other than the PC that matter that much anymore, and for those few that do (notably the Mac and Linux), (-community) the DLLs can be ported along with the core engine code. It just doesn’t make sense for easy portability to tiny markets to impose a significant development and performance cost (FIXME: 10 percent?) on the one huge market. Consoles will always require serious porting effort anyway, so going to Win32-specific DLLs for the PC version won’t make much difference in the ease of doing console ports."

http://www.bluesnews.com/abrash/chap64.shtml (speed)
"When I came in on Monday, John had the look of a man who had broken through to the other side--and also the look of a man who hadn’t had much sleep. He had worked all weekend on the direct-BSP approach, and had gotten it working reasonably well, with insights into how to finish it off. At 3:30 AM Monday morning, as he lay in bed, thinking about portals, he thought of precalculating and storing in each leaf a list of all leaves visible from that leaf, and then at runtime just drawing the visible leaves back-to-front for whatever leaf the viewpoint happens to be in, ignoring all other leaves entirely ... The first time John showed me his working prototype, I went to the most complex scene I knew of, a place where the frame rate used to grind down into the single digits, and spun around smoothly, with no perceptible slowdown. "
Each to each visibility with RLE, O(n) (<= one has to clip to frustrum anyway and edge case is all other leaves are visible from that leaf). NOTE: It is in core brute force O(n^2) visibility. Awesome lesson on how simple stuff can be compressible. NOTE: Interesting is that, to my knowledge, he did not try any hashing like in Doom or 8-tree (octree). Point is - you CAN make it fastby chaning algorithm and a lot .

Some smart guy made simple changes to Q2 renderer http://quake2world.net/?q=node/69 (speed)
Quake 2 was made 4 times faster just by simple optimizations like :

1 lookup pvs on change .... “almost all programming can be viewed as an exercise in caching,” http://xona.com/ramblingsinrealtime/chapter1.html
2 sort drawing by texture ... setting new texture has big cost in opengl
3 vertex arrays ... I do not know if JC knew about them by time he wrote Q2
4 multithreading (decoupling quake into more/less mutualy exclusive parts) ...


Compare : Let say Z buffer has 10 percent, lets say QC has another 10 percent. If i do correct math here, that is 0.9*0.9 = 0.81 = 81 percent; so Quake 2 could run at about 80 percent with having z-buffer and QC if they would ditch z-buffer, but since they did not it is 90 percent of its speed.

Compare : TMK Quake 3 is first in id-techs to use hashing for cvar lookup. It is O(n) to O(1). I do not know how big part of performance cvar lookup was.

Why can scripting language be better than Cee?
http://stackoverflow.com/questions/1193 ... age-over-c ( QUAKE C)

Conclusion

It makes me feel bad sometimes that games are just about graphics. It is of course overstatement, but I hope You get the point ...

I do not know for sure, but there was adn probably is some kind of stupid "i can has more fps than you" meme in gamers community, toghether with "i can has direct3d x i can has opengl" and "i can has better shaders". It is stupid because fps does not matter after 20 or so (rate at which fastly changing pictures make smooth change for brain) and opengl is superior just because it is not for one plaform only.

10 percent loss can be forgotten by 400 percent gain elsewhere. Interpreted is better than compiled for community, portability, testing ... and maybe not for debugability (well, I dont know about this point much, but I belive it is somewhatt nonsense that interpreted Quake C is worse to debug than compiled... I mean ... there is no IDE etc for it like for C++ ).Conclusion?There is no reason to ditch Quake C from Quake. It was a mistake.

Questions :

What could be other forces which caused decision of JC to ditch QC?
What can be the X in "more sophisticated X in Quake2's game logic"?
Unreal does octree/8-tree to speed up rendering? What does Bethesda use for rendering?

[mh]

That's quite a lot. Masters of Doom is essential reading here as it fills in many gaps and answers many questions.

Quake was designed with the intention of being user-modifiable from the outset. Both Wolf3D and Doom had recieved some pretty basic modifications - School Doom, Star Wars Doom, Barney Doom, etc. Both Romero and Carmack thought that this was The Coolest F*cking Thing Ever and decided that Quake would be even easier to mod. Not everybody at ID necessarily felt the same way.

No sinister external forces caused the ditching of QC for Q2. Just practical pragmatism. A lot more of the server moved to game code for Q2 (physics for example) and QC was just not up to the job. You've been reading the .plan archives I see, so do continue reading. You'll learn that other options were considered, including extending QC and using Java. In the end a DLL was chosen (despite some concerns about it), and the reasons why are outlined in the archive.

It's notable that the next engine (Trinity/Q3A) reverted back to an interpreted bytecode VM type thing. It was also capable of JITting the .qvm, and could use compiled DLLs if you wished. So Q2 was the abberation in the middle, whereas Q1 and Q3A showed a clear line of continuity.

Don't underestimate the power of being able to debug stuff properly. I get the sense that far too many members of the Q1 community subscribe to a "hack at the code, try it, see if it works, hack at it some more, and continue in that vein until something comes out the other end" ethos. 30 seconds in a really good debugger can save weeks of hacking and slashing at code, trying stuff at random, and hoping you get things working. All non-trivial programs have bugs, so a good debugger is an essential tool.

If your overdraw is low enough (less than 2 x or so) then disabling Z-testing and drawing in strict back to front order can be the fastest way, yes. But it requires strict back to front order, so with a hardware renderer you don't get texture sorting, you don't get primitive batching, you get thousands of state and texture changes per frame, and performance will go to hell. Z writing is still essential as entities need something to Z test against.

A lot of the Abrash stuff is mostly relevant for a software renderer. A hardware renderer changes the rules, hardware T&L changes them again, and programmable hardware changes them yet again. You need to know the context that things were written in.

I don't believe that guy got 400% in the general case. Not for one minute. Maybe under certain idealised cases, and maybe under heavy load, but not in the general case. I can say that from a position of authority because I've done the same with DirectQ and I know the performance characteristics of this kind of code.

A lot of what he talks about actually did exist in 1997; Q2 did use vertex arrays as released, it did use multitexture, it even did sort by texture (but in the single textured path only). The single biggest performance gain in Q2 (and Q1) in multitextured mode comes from decoupling your lightmap updates from your surface drawing. Combine that with the correct choice of format and type, and you'll be getting dynamic light updates almost for free.

For Quake (and Q2) style rendering vertex buffers are useless without shaders. If you're animating water textures, cycling MDLs/MD2s through frames, etc, you need to be updating a dynamic vertex buffer each frame, in which case you may as well abandon the vertex buffer and just use a vertex array.

The R_RecursiveWorldNode trick was something I had in unreleased code for a short while, but I abandoned it because it made framerates uneven. Given a complex enough BSP tree you could jump from 200 FPS to 400/500 FPS and back many many times over the course of 1 second. The gameplay experience was unpleasant, so raw speed isn't everything. Sometimes you accept lower than a theoretical best because it's better for gameplay (which is the whole point of the exercise).

Simplifying the BSP tree for the purposes of rendering probably has more mileage in it. Collapse it so that no node goes below a certain minimum volume, then amalgamate all surfaces in the node sharing the same properties into a triangle soup. Remove duplicate vertexes, draw with indexes, and you'll do better. There's some extra overhead at load time, but Quake already loads fast enough; so long as it's under half a second or so for ID1 maps it should be acceptable.

[Baker]

I get the sense that far too many members of the Q1 community subscribe to a "hack at the code, try it, see if it works, hack at it some more, and continue in that vein until something comes out the other end" ethos.

I am wondering where web based standards and languages might fit into this as an influence. I'm talking about PHP, Perl, Javascript, HTML/CSS/DHTML and such.

I had bad habits before starting to play with the Quake engine in large part due to that.

[Stops before accidentally creating a super-tangent sub-topic.]

[revelator]

early on it might have been just that way later on a few members pushed through the hack phase and started developing
true extensions and improvements like LH and MH (the oldest in the engine scene i know and i been around a loooong time).

im still very much in the hack and see if it works phase mostly due to lacking vital knowledge (my math fu is seriously broken hehe), hell back when i started the only language i ever touched was basic, in the amiga and commodore days (before that even, my first computer was a ZX80). but i did learn quite a few things watching and asking on the original QSG forums and here. even though ill probably newer get to a point where i can create my own engine truely from scratch.