Engine Philosophy Discussion

[Baker]

I read some of this from an article at Slashdot.

http://www.aosabook.org/en/index.html

"Architects look at thousands of buildings during their training, and study critiques of those buildings written by masters. In contrast, most software developers only ever get to know a handful of large programs well—usually programs they wrote themselves—and never study the great programs of history. As a result, they repeat one another's mistakes rather than building on one another's successes.

This book's goal is to change that. In it, the authors of twenty-five open source applications explain how their software is structured, and why."

I've read Spike and Metlslime's and LordHavoc's and well --- obviously --- Carmack's engine code. [And Tonik's and Fuh's and the ezQuake guy's and R00k's and Jozsef's and Grossman's and the QIP Quake people's and MH's and leileilol's and qbism's and frag.machines's and the Flash Quake guy's and the PSP people's and Sleepwalker's SDL stuff ...]

Since I don't have anything useful to contribute to the topic of the thread, I'm going to note things I have noticed about other people's code:

1. Spike uses #ifdefs everywhere. They cannot be relied upon fully, but as far as I can tell, he uses them to mark experimental code or code that he did not fully trust at some point. FTE contains some of the greatest engine achievements, but the only one that truly boggles me: How the F did he do 24-bit color in software? Now ... the reason I wonder this ... unlike most implementations, you really cannot write that one piece at a time. Well, actually now that I think about it, I can imagine a couple of ways to phase it in but it still requires massive changes in several places simultaneously to make it happen. FTEQW has a dramatically high amount of radical feature experimentation compared to other engines.

2. Metlslime types a lot of notes into his code. He'll unwind a mystery and then document it into the notes. I suspect that the way the code is designed that he made detailed planning documents in advance because the organization of the code is too "perfect" to have evolved. His engine coding is like his mapping, there are no misaligned textures.

3. LordHavoc is a fan of delegation. He likes formulaic-ally breaking down what is really occurring and outsourcing it into several different functions and structs and arrays. And adding cvars to control the behavior. In the older days, if you look at each version of the code, you find him tackling stupid things that clearly annoyed him [10 years before most people thought of them ... like the option to die with your view angles "normal".] In the more recent versions, he tackles things that annoyed him more recently [true type font, for example]. The code is often hard to follow like complex C++ because operations have been broken down into so many functions and arrays it is difficult to picture the data flow and see the big picture. Nevertheless, the approach to handling several issues is invaluable and if lucky enough ... the comments are very good but they do not last for long from version to version. DarkPlaces is clearly influenced by Quake 3, model formats, server browser and server self-reporting technology, OpenGL rendering techniques in additions to address what LordHavoc clearly perceived as the weaknesses of Quake like memory management.

4. ezQuake peoples. The code is written by many different authors in a very structural and easy to follow way and commented well ... and clearly reviewed by several different eyeballs. Next to FTEQW, probably the most useful codebase to follow. ezQuake suffers from unbridled feature creep, probably because a great many "cool" ideas have been simultaneously implemented leading to an in-cohesive experience.

5. R00k. Comes up with a fancy idea, codes it. Worries about bugs later. This is actually the true spirit of experimentation, which is why Qrack has a lot of unusual features and also unsolved mysteries. It is the highest net experimentation engine for standard NetQuake, with hordes of invaluable ideas brought to fruition.

6. Leileilol. Coding changes from the mind of a true modder. Includes QuakeC extensions that are likely important to unique mods and have changes reflecting a great knowledge of historical tool resources.

7. MH. The most likely source of code that operates without error the first time. The code is tutorial quality in the sense that all options and conditions have been anticipated and dealt with. Usually the first release is also THE final release as a result. No stone is left uncovered when MH is done, barring advertised alpha and beta releases where you know he is fielding for unknown factors to emerge ... but he knew they would emerge.

8. Tonik. When I look at his source code, I have the feeling he is far more conservative than average as he solves functional problems and I think he loves Quakeworld but hates modern Quakeworld. Tonik has a huge laundry list of solved problems and advancements in the early days. He was clearly a proponent of reliability and operational stability, with a side goal of bringing all the feature of NetQuake into the Quakeworld domain (single player, NetQuake mod compatibility but with Quakeworld physics etc.) But his engine work doesn't focus on excessive bells and whistles like, say, ezQuake but feature-set expansion. Vwep support, Quake 3 support, removal of efrags etc. I get an old school Quakeworld feel from his code.

9. PSP Authors. The PSP engines feel heavily influenced by Quakesrc.org posts and tutorials [QuakeSrc.org was fully operationally during the development cycle of the primary historically PSP engines]. The PSP engines have a flare for optimizing use of the PSPs controls (on-screen keyboard, many ways of dealing of with the analog stick ... a mini-joystick in the form of a slightly slidable button).

10. Carmack of the 1994-1999 era, as far as I can tell --- if you view "DOS Quake" as one true Quake, it is clear he wanted it to have it "all". The engine is both a clear demonstration of "group brainstorm" as it contains high quality implementations of several experimental ideas [some of which you can imagine John Romero saying in some pizza meeting insisting the engine has gotta do this or that]. The engine has a ton of different layers of "systems" all over the place and the 3D math to back it up. The ideas of everything from physics, independent entities with properties and think functions and in the GL version --- shadows, mirrors and transparent water --- to framegroups and skin groups -- a command console and scripts and multiplatform design and the support for several different "model formats" --- even if these were brush models, alias models and sprites --- and client side particles. Historically, thinking of the difference between Doom II and Quake ... and the completeness of the network protocol (even if bugged and dialup inefficient) ... it is bit difficult to grasp the idea that all of this change was in a single generation of engine advancement --- and that it worked and was operationally bug-free. And that it runs on DOS! Or that Carmack made the GL version in just a month.

[frag.machine]

10. Carmack of the 1994-1999 era, as far as I can tell --- if you view "DOS Quake" as one true Quake, it is clear he wanted it to have it "all". The engine is both a clear demonstration of "group brainstorm" as it contains high quality implementations of several experimental ideas [some of which you can imagine John Romero saying in some pizza meeting insisting the engine has gotta do this or that].

From what I read in "Masters of Doom", it was quite the opposite. Carmack resented a lot the absence of Romero during the engine development, because he was too busy running Doom tournaments, supervising the development of Hexen and playing the rockstar to the press and fans. Brian Hook was hired to help Carmack in part due this situation, and later Romero was kicked because of this, too.

[qbism]

I enjoy reading comparisons between architecture and programming. "Learning from mistakes" speaks to me about preparation and documentation before diving into project production.

A few concepts of good architecture process that resonate with some of Baker's points-

1. Respect the uniqueness of individual designs but open up underlying frameworks. Long ago, masonry guilds protected construction secrets. Nowadays, the masonry industry publishes detailed tech notes for anyone to read and gives free seminars. I imagine that improvements flow back to the industry as builders review and test techniques. Open up redundant silos of wheel reinvention.

2. Modularity. Isolate and standardize components as much as possible. Easier to delegate work, swap-out parts, and isolate systems.

3. Durability. Good buildings last 100 years or more... and so will Quake code. Program not for today, but for your grandchildren.

4. Building codes. A few countries have adopted the "International Building Code", which standardizes performance rules for structure, emergency egress, energy use, etc. Imagine that programs are required to meet a highly detailed standard, and a permit and inspections are required to release it. (The approval process may be something like that for certain proprietary hardware platforms.)

5. Rigorous study, critique, and defense of design. I guess that's the gist of the book. In architecture school, designs are pinned-up for public presentation to peers and professors. Famous buildings have volumes written to document, denounce, or praise the design. One may obtain a similar experience posting code here on inside3d, maybe that's why I like it!

6. Focus on knowledge and technique rather than tools. A more general and rounded education- become educated in design, history, and the design process. Let that have priority over the quirks of a particular toolset or platform.

[daemonicky]

Btw. Quake is event oriented and quite similar to Javascript (I apologize because I do not remember correct names):
window.onload = function ~= when map is loaded handler = { function }
setTimeout(function, nextthink); ~= nexthink; think = function;
And these tiny DOM nodes can have some handlers, almost those of Quake entites. Heh, that would be cool to see DOM dancing before you, wiggling and some evil banners would be smashed by br tags. But I guess they dont have collision detection, it is a pity.

I dont know, it seems too general for me. In praxis, I think, the best way is to be inspired by some nice techniques of extreme programming and agile development.
- Build working thing ASAP. Decide what is the most important stuff and get it working, even by a hackish way but have something to build upon.
- Make a good desing first. Get to know some UML.
- Pair programming. I tried it only few times and it was cool. But it really depends how much the skill of you to match and how you complete each other in the pair.
- Make atomatical tests. If you add something new run some of these tests.
- Standards are good. When I used to program in C I was looking up manpage of each stdlib function.
- Make use of wide range of approaches to solve problems. Solve problems geometrically, by graphs, sets, databases, regular expressions, recursion, lambda calculus and so on ... so pretty much get to know your toolkit and make a good one.

I enjoy reading comparisons between architecture and programming. "Learning from mistakes" speaks to me about preparation and documentation before diving into project production.

Then you might enjoy what Knuth wrote http://www.paulgraham.com/knuth.html .

Learning from mistakes" speaks to me about preparation and documentation before diving into project production.

http://alistair.cockburn.us/Incremental ... +reworking

[Baker]

From what I read in "Masters of Doom", it was quite the opposite. Carmack resented a lot the absence of Romero during the engine development, because he was too busy running Doom tournaments, supervising the development of Hexen and playing the rockstar to the press and fans. Brian Hook was hired to help Carmack in part due this situation, and later Romero was kicked because of this, too.

Well, I would be curious as to the origin of what seems like a superfluous quantity of new features in the Quake engine.

All one has to do is really start digging around in the code and you quickly run across hordes of features that could be categorized as extraneous. It is almost over the top.

The game could have been shipped with maybe 2/3 of those features cut out and saved a lot of time, which is of course one of the great things about the flexibility of Quake.

2. Modularity. Isolate and standardize components as much as possible. Easier to delegate work, swap-out parts, and isolate systems.

I've really like C because I like low-level coding, but in reference to this ... I am growing to hate headers. After you try to isolate pieces and get them disconnected, you end up with a sprawl of headers and eventually seem to hit the wall where you have to throw

Code: Select all

#ifndef MYHEADER_H
#define MYHEADER_H
...

to avoid getting lost into an abyss of time wasting.

Although one thing that becomes a bit interesting is finding out how old the "C" language actually is and well it was standardized and how long ago that was. I didn't really suspect the roots of C emerged in the late 1960s, with some of the design from precursors in the late 1950s [ALGOL ... like what were they programming on the in 1950s, heh. Univac? Did they even have punch cards (or whatever they were called back then?]

Then you might enjoy what Knuth wrote http://www.paulgraham.com/knuth.html .

Checking that out.

[daemonicky]

I think it might be better to make Quake in higher level language like Python. You dont need to make it object oriented code, but you would be able to write foreach, some functional code, easily make new complex data structures. C is pain to work with from my experience, Python is much more pleasant.

Well, I would be curious as to the origin of what seems like a superfluous quantity of new features in the Quake engine.

I wanted to ask Carmack something / how his older engines were, whether it is possible to release source, history of desing of his games/ but sort of gave up and forgot about it. So. Why not ask Carmack? Or Romero? I think they would answer ... that would be simplest way to know for sure how it was

Checking that out.

Enjoy. Khuth is somebody to check. He wrote also some great stuff about GOTO http://pplab.snu.ac.kr/courses/adv_pl05 ... -knuth.pdf .

[mh]

Who needs goto when you've got:

Code: Select all

do
{
   ...
   ...
   if (condition1) break;
   if (condition2) continue;
   ...
   ...
} while (false);

goto is much cleaner and easier to follow than constructs like that, IMO.

And once you've been exposed to true crawling horror like Duff's device or negative array indexes (or Duff's device using negative array indexes) you really begin to appreciate goto by comparison.

I actually have a valid use case (well, two actually, but this one amused me more) for Duff's device in my current codebase: transferring indexes to a dynamic Index Buffer where each group of indexes may have a variable offset added:

Code: Select all

__inline void D3DBrush_TransferSurface (msurface_t *surf, entity_t *ent, brushpolyvert_t **verts, unsigned short **ndx, int ndxofs)
{
	unsigned short *srcindexes = surf->indexes;
	int n = (surf->numindexes + 7) >> 3;

	// we can't just memcpy the indexes as we need to add an offset to each so instead we'll Duff the bastards
	switch (surf->numindexes % 8)
	{
	case 0: do {*ndx[0]++ = ndxofs + *srcindexes++;
	case 7: *ndx[0]++ = ndxofs + *srcindexes++;
	case 6: *ndx[0]++ = ndxofs + *srcindexes++;
	case 5: *ndx[0]++ = ndxofs + *srcindexes++;
	case 4: *ndx[0]++ = ndxofs + *srcindexes++;
	case 3: *ndx[0]++ = ndxofs + *srcindexes++;
	case 2: *ndx[0]++ = ndxofs + *srcindexes++;
	case 1: *ndx[0]++ = ndxofs + *srcindexes++;
	} while (--n > 0);
	}

[Baker]

Why not ask Carmack? Or Romero? I think they would answer ... that would be simplest way to know for sure how it was

I'm kind of talking about something else here, but I generally think it is more considerate to "leave people alone" and take what they give you.

Can you imagine how many emails they get probably asking the same questions over and over again with many of the answers already known?

Carmack is one guy. Romero is one guy. The ratio of email quantity to body count of 1 is probably staggering.

Which is why I think forums are great. People willingly volunteer to help, but are never specifically obligated as an individual.

I thought it was wonderful in 2006 when scar3crow got Romero to release the Quake map sources, which Romero had stated previously that he had intended to do.

But aside from that, I don't believe in bothering people especially ones that have given so much already.

/Just my opinion. My opinions, like all opinions, are ... just opinions.

I think it might be better to make Quake in higher level language like Python. You dont need to make it object oriented code, but you would be able to write foreach, some functional code, easily make new complex data structures. C is pain to work with from my experience, Python is much more pleasant.

Python is an interpreted language = slow.

C is just a rung or 2 above assembly language that insulates you from how different architectures do things.

Interpreted languages are like going to the grocery store and buying a box of cereal. Languages like C are like planting the crops and getting your hands dirty to grow the wheat.

It is more work and requires a far greater knowledge overhead, but you have more freedom to control end results and don't have to live inside of a containment box.

There are advantages to both higher level languages and lower level ones. But the potential of lower level ones is that you don't have so much limiting you except yourself versus interpreted languages which often run into "damn, that can't be done" or more likely "damn, that can't be done in a way that is fast".

[Spike]

1. Spike uses #ifdefs everywhere. They cannot be relied upon fully, but as far as I can tell, he uses them to mark experimental code or code that he did not fully trust at some point. FTE contains some of the greatest engine achievements, but the only one that truly boggles me: How the F did he do 24-bit color in software? Now ... the reason I wonder this ... unlike most implementations, you really cannot write that one piece at a time. Well, actually now that I think about it, I can imagine a couple of ways to phase it in but it still requires massive changes in several places simultaneously to make it happen. FTEQW has a dramatically high amount of radical feature experimentation compared to other engines.

ifdefs are bad.
of course, if they're always defined, they act as comments. and if you see them in related code, you have a tendency to think they're needed and to propagate them between various affected locations.
and modularity is good.
but ifdefs suck. use comments instead, and keep them up to date. comments rock. but don't use them too much or they get in the way too.
24bit software renderers don't need to be implemented all at the same time. get the mode set up properly, get the console background and text drawn, get bored, come back, get other stuff to stop crashing. if you make it into a map, then you get a quater of the screen drawn with a screwed up palette but its not fatal, testing it is fine, you just have to remember what's not written yet.
writing it one piece at a time is fine, if noone knows that they can actually activate it and find all the lingering omissions then you never get yelled at to fix things. put simply, 24bit colour doesn't mean the 8bit renderer cannot work.

so yeah, keep functions around 30 lines. shorter and its too much whizzing around trying to find the right bit of code that does what you're looking for, longer and its not modular enough. add comments for awkward logic.

[daemonicky]

You are right. It would be nice to know whether he offered some free chat or something similar.

Python is an interpreted language = slow.
C is just a rung or 2 above assembly language that insulates you from how different architectures do things.
Interpreted languages are like going to the grocery store and buying a box of cereal. Languages like C are like planting the crops and getting your hands dirty to grow the wheat.

Yes. I know it is slower because it is interpreted. But you can always measure what is slowest at you engine and replace it with C, while being very productive with the rest. And that is the main idea for doing something in Python or some othe high level language. It is not the speed of program, it is the speed of writing it/speed of programming. And if you wrote a better algorithm you may be faster than C coder given same time (there are languge constructs which allow you to write something in one line in language which takes dozen in the oter ...).

I agree with You on the rest. But I think than because you built faster with high level one you can build more, and because you build more you may learn what is wrong with your desing and your approach so you may change it early ... and in the end program it in C

I dont know. THey are both tools and tools have pros, cons and are good for something and bad for something else.

and once you've been exposed to true crawling horror like Duff's device or negative array indexes (or Duff's device using negative array indexes) you really begin to appreciate goto by comparison. Laughing

I liked Duffs device when I was younger. I grow up. No, I am kidding I just forgot about it.

You can look at more horrors there :
http://www.hackersdelight.org/HDcode.htm
http://www.hackersdelight.org/
I actually found some of them beatyful and extremely expressive. In that Baker is absolutely right, sometimes some constructs are more powerful in low level languages. Like bitfields. But deciphering code heavily made with it hurts the head. I mean, I looked at Duke Nukem 3d Source Code, it was full of tricks.

[daemonicky]

ifdefs are bad.

It remindned me of a rant of people who are making Plan 9 and made Unix http://doc.cat-v.org/henry_spencer/ifde ... ed_harmful .

so yeah, keep functions around 30 lines. shorter and its too much whizzing around trying to find the right bit of code that does what you're looking for, longer and its not modular enough. add comments for awkward logic.

It remindned mi of RISC and its huge amount of instructions VS CISC and it huge and slow generic instructions.

[Baker]

It is not the speed of program, it is the speed of writing it/speed of programming. And if you wrote a better algorithm you may be faster than C coder given same time (there are languge constructs which allow you to write something in one line in language which takes dozen in the oter ...).

There aren't any right or wrong answers to the speed versus control thing. Sometimes both matter.

I personally started to dislike being shielded inner operations because the barriers stopped me from some things I wanted to do.

[mh]

...and there are different kinds of programs. If you're running a large overnight payroll, for example, that absolutely has to finish by 8AM, then you are going to be very concerned about speed and won't give a damn if $LATEST_FAD_LANGUAGE allows you to do the same thing in one tenth the code.

Likewise if you're trying to render a Quake scene with 30,000 epoly.

The moral of the story is that the whole "Unix philosophy" thing is really only appropriate for certain kinds of program - specifically command-line tools intended to be invoked from shell scripts, or similar small throwaway homebrew tools where performance isn't a constraint.

Little-known-fact: the vast majority of Quake engines are CPU-bound. Badly CPU-bound (that and they suffer atrociously from CPU/GPU sync issues). As soon as you've got the basics right in your renderer, you're going to find very quickly that CPU is your primary bottleneck, so that's where the most intense focus needs to be for future optimization, and where the most extreme potential for loss exists by doing something the wrong way, choosing inappropriate technology, or whatever.

[Spike]

I personally started to dislike being shielded inner operations because the barriers stopped me from some things I wanted to do.

if you write functions with gotchas, expect to be got. :)
special cases are always bad for that reason.

ifdefs are not only special cases, but they result in two separate builds to test.

[mh]

As a corollary - if you find yourself writing lots of extra code to handle more and more special cases, then you were most likely doing it the wrong way in the first place.