RMQ Engine ... Sheesh!

[Baker]

When I first used the RMQ engine, I noticed a lot of things and I knew a lot of the planned features and a number of them were noticeable in the game, of course.

Maybe just because of the blog, I expected the RMQ engine to be a minimal change mostly to accomodate BSP2, all the GL extensions and use of them.

But there are a hell of a lot of improvements in the engine aside from those changes. I thought I was done pulling in code that I didn't fully understand from another engine just to see the result of a specific set of changes, but I guess I am wrong.

I also see evidence of at least one feature I didn't know could be done. It looks like RMQ *might* have video capture with SDL somehow. I'm also a bit surprised to see demo rewind in the engine. I thought MH hated that, haha.

While talking about MH's code, I was looking at frames per second again (to measure speed improvements) and loaded up a few engines and did timedemos. On this machine, DirectQ 1.9 did the timedemo so fast it was barely on-screen and this Windows 7 laptop isn't fancy at all. It did like 541 frames per second, well over double of the best results with the other engines I tried. It wasn't the 541 fps, it was the "over double" of other engines that was the shocker.

I opened up the RMQ engine to check on 3 things (to see if dynamic light was correct on rotated brushes, it is. I never noticed rotated brushed were improperly lit by dynamic lights in my engine, never thought to check) ... check the model loader for anything of interest ... and I noticed that in particular FitzQuake is really slow switching between fullscreen and windowed and I knew that both RMQ and Quakespasm have some sort of improvement for that.

I hadn't really expected so much of the RMQ engine had been combed over/upgraded. In a way, it also doesn't surprise me ... but the extent of the changes are way more significant than I would have guessed.

[Baker]

I also see evidence of at least one feature I didn't know could be done. It looks like RMQ *might* have video capture with SDL somehow.

It isn't actually AVI capture, writes a series of TGA files to disk.

[Baker]

[Disclaimer: Hey this stuff might be in DirectQ too but I discovered it here ...]

RMQ Gem #1: Stalled save games

host_cmd.c -> Host_Savegave_f

for (i = 0; i < pr_progs->num_edicts; i++)
{
ED_Write (f, GetEdictForNumber (i));

// eeep!!! this was why savegames stalled!!!
// fflush (f);
}

fflush forces the I/O buffer to finish writes from the file stream. It was doing this after writing every server edict to file (there are usually several hundreds of these).

Oh man ...

[mh]

I do have a tendency to do that - even while working on something specific, if I see something else I don't like or think could be better I fix it up to the way I think it should be. Bit of a bad habit really, too easily side-tracked (and makes keeping any kind of changelog a total bitch).

The series of TGAs was agreed on as the most portable solution. Portable audio is nasty enough; I'm not touching video. It does sync with sound correctly though.

I still hate demo rewind.

[Baker]

RMQ Gem #2

Determine lighting for static entities only once. RMQ does this in CL_ParseStatics (or whatever).

[Note to self: Don't even bother with that if model is marked with noshadow or fullbright rendering like torches].

RMQ also generates a 4x4 matrix here and does it only once.

[Baker]

RMQ/DirectQ Gem #3:

(An oldy to me since I saw this in DirectQ over a year ago).

Generate smax and tmax for brush model surfaces on load rather than recalculate those all the freaking time

RMQ Gem #4

Player skin code includes a lot of software stuff. Kill.

RMQ Gen #5

Most engines appear to keep copies in memory of map texture data it doesn't actually use anywhere.

RMQ Gem #6

Resample 8-bit textures while still 8 bit textures to gain speed. (I wouldn't do this with alpha 8 bit textures, but I think RMQ special handles those).

[mh]

The static entity lighting thing actually sounds like a bug - it should determine the surface and cache that in the entity, because the lightmap data used by the surface may change. A neater way would be to store out the RGBA values used in the lightmap for each surface style on the point it hits, and run those through the standard d_lightstylevalue[surf->styles[maps]] calculation, but that would inflate the entity size a little. Not too much, just a little. dlights should still be determined every frame, of course.

The matrix remains valid.

These two were necessary on account of the huge numbers of static entities in some maps - what was normally not a bottleneck suddenly became significant.

The 8-bit resample expands to 32-bit at the same time so it's just a single pass over the data whether the texture needs to be resampled or just expanded. I thought it would have been an obvious thing to do.

[Baker]

Yeah, good point. It sounded good though. Most statics are brush models or torches that don't get affected by lighting. Although in RMQ I noticed possibly an unusual number of non-solid surface entities that I believe to have been statics. I don't know if you had implemented your "brush models get affected by dynamic light" change in RMQ. I guess I could shoot a rocket and see.

Code: Select all

// static entities never move so get an initial R_LightPoint for them here and store it out so that we don't have to
	// trace the BSP tree at runtime
	R_LightPoint (ent, discard);

RMQ Gem #7

Fast load TGAs.

The original TGA loader loads images byte by byte from disk. RMQ reads the entire file into memory and then walks it byte by byte instead of involving a hundred thousand different single byte reads.

I thought it would have been an obvious thing to do.

I don't know. It isn't that it isn't obvious in retrospect, more that things like the TGA loader .... I've read that code many times. I've fixed things in it. It wasn't until now that I noticed the fgetc everywhere (I saw them. I knew they were there. I just wasn't think of load speeds so I wasn't think of the implications.). I've read the save game file function several times. I never thought about the fflush.

I think bottlenecks get fixed when someone notices them. Like that one thread where I was wondering why external textures on map load were so slow and I discovered the pak files got walked thousands of times for textures that aren't in the pak.

I think awareness is what solves problems. I know you had map load times and FPS in mind when working with the RMQ engine (because those maps are so damn big).

[mh]

Regarding the performance note in your first post, DirectQ can't really be compared fairly to other engines in timedemos because it bases it's damage/bonus flashes on client time rather than realtime. In a timedemo client time is running faster so these last for less (real)time on-screen, and because they have huge fillrate overhead it naturally runs faster as a result.

If you make the same change to any other engine (cl.time - cl.oldtime instead of host_frametime in V_UpdatePalette) you'll get a good speedup in timedemos too. DirectQ will likely still go faster, but not by so large a margin.

The extra performance is maybe 90% achievable by any Quake engine today, but it does require pushing the hardware requirements higher than some people might be comfortable with (I don't think RMQ pushes them high enough). The other 10% comes from D3D-specific things like vertex buffer usage and locking flags that are clear, sensible, and actually work (GL3.x + has these too, but it still goes to hell with UBOs).

It's a very rough rule of thumb but I'd classify gfx hardware into 4 broad categories: prehistoric, ancient, early-modern and modern. There are some oddities (I'd make Voodoo 2 prehistoric on account of the mini-driver and weird texture memory setup for example) and some omissions (I don't classify the likes of GeForce 3 and 4 as they were interim steps on the way to full high-level shader capabilities), but it breaks down like so:

- prehistoric typically has a single texture unit, much missing functionality (such as blend modes), typically a mini-driver, requires lots of hardware-specific hacks.
- ancient is full GL1.2 to 1.5 support, D3D5 to 7, no shaders, may have hardware T&L, usually has multitexture. Full GL ICD.
- early modern is GL2.0 and 2.1 or D3D9 class, shader-model 2 or 3, full capabilities at that level.
- modern is GL3/4, D3D10/11, GPGPU stuff, 1gb of video RAM, etc.

This is a bit of a long-winded preamble leading up to the main observation, which is that if you aim at a specific hardware level, you can usually expect hardware one level up from it to run well enough. Hardware two or more levels up will be highly unlikely to be running at anything near it's full capabilities. Some ad-hoc informal profiling shows me that most Quake engines are doing good if they hit 60% GPU utilization during a typical scene with uncapped framerates and just letting everything run flat-out. There's just so much stuff piled on the CPU, and so much data constantly being sent over the command buffer, that the GPU is spending so much of it's time relatively idle, and when it's not idle it's always stalling so that the CPU can catch up. And it's normally the faster processor that's capable of doing a better job of what's been instead loaded onto the CPU.

DirectQ hits 100% GPU utilization during the same scene, but lately I've been noticing that I may have gone a little too far in that direction - anything extra added (such as my aborted attempt at GPU lightmaps) has tended to have a disproportionate performance impact. There's some better workload balancing to be done here for sure.

[Spike]

the problem with gl, which isn't much of an issue with d3d, is that even moderatly recent intel cards are in the 'ancient' grouping that you mention. which sucks.
if you're okay with it, you can always have alternate paths for different levels of hardware. doom3 supposedly did a chunk of that with its arb/nv/ati specific backends, for instance.

a major major speedup for bigass1 is running particles in clienttime instead of realtime (not just flashes).
another sneaky trick you can potentially do, which fte does, is to throw the audio mixer into another thread. this gave me a 10%ish speedup in fte, but I doubt other engines would see quite the same rise. 10% is a really disturbing amount, but it all builds up.

reaching 100% gpu utilization is bad, if you're running at 640*480.
If they're only reaching 60% at 1920*1080 then that would be really bad.

I remember getting my laptop to render at 1100fps or so. yay for water shaders. add in a lightmap and the fps drops to half of that. the laptop was okay for drawing stencil shadows, but yeah, texture lookups were a killer. the support for rbsp lightmaps I recently added has me worried enough that I refuse to try it on that laptop.
gpu-based lightmaps are even more fun when you're trying to get them working with q3 shaders and a fixed-function gpu. Its really not pretty.

[Baker]

RMQ Gem #8

A really nice no-cost colored deadbodies solution

Throws a *scoreboard var in cl_entity_t pointing to the appropriate player. Use that player's skin (likely if the model matches.)

A really nice 99% solution. To hell with the 1% super-fringe cases.

[Actually it doesn't appear to check that, I grabbed a ring and my dead body was funky. Maybe throw a last known model pointer in the scoreboard and last known skin # in the scoreboard and have R_TranslatePlayer skin update that.]

---------------------------------------
RMQ Item of Curiousity #9 and #10

snd_dma.c -> void S_StaticSound (sfx_t *sfx, vec3_t origin, float vol, float attenuation)

Code: Select all

	// hack for non-spatialization of global sounds
	if (attenuation > 0)
		ss->global = false;
	else ss->global = true;

Too lazy to look at QuakeC right now to think about how this works or if it can break anything existing, what it does is obvious.

And there is this (looks like from DarkPlaces, but maybe not) ...

Code: Select all

	// fixme - do we actually need to update ch->origin here or can we just use a different origin vec (a local?) purely for spatialization?
	if (ch->entnum > 0 && cls.state == ca_connected) // && cl_gameplayfix_soundsmovewithentities.integer)
	{
		if (cl_entities[ch->entnum] && cl_entities[ch->entnum]->model)
		{
			// brush model entities have their origins at 0|0|0 and move relative to that so we need to use trueorigin instead
			// update sound origin
			VectorCopy (cl_entities[ch->entnum]->trueorigin, ch->origin);
		}
	}

RMQ Item of Curiousity #11

A to-do putting forward the idea of frustum culling on the server. In multiplayer and possibly predictive protocols, this would be bad.

But in single player, server-side frustum culling wouldn't seem to be a problem, reducing some calculations needed on the client side and making demos smaller, etc.

RMQ Item of Curiosity #12

DarkPlaces Lerping fixes for movetype step (the U_NOLERP in original Quake). Changes in CL_RelinkEntities to support this too, of course. Comments "1.05 server-side move lerping" ... so basically DarkPlaces has only had this 10 years +/- before getting into another engine . (I just sort of think it is cool that the old DarkPlaces builds, even the 2002 ones and such, LordHavoc was noticing and fixing really annoying stuff that no one had thought about very much at that time.)

RMQ Item #13

PushRotate and PushMove don't allocate and release memory for edicts every frame. Instead creates a buffer for it.

[Baker]

RMQ Item #14

sy_phys.c: Implements allowing moving on entities like normal map, except only allows this for SOLID_BSP and MOVETYPE_PUSH.

This allows moving like normal on top of exploding boxes and whatever other external bmodels the RMQ guys are using.

But doesn't allow heresy like walking on fiends or using the poor Chthon on E1M7 as a bridge (*).

Code: Select all

				if (mod->type == mod_brush && mod->name[0] != '*')
				{
					// we have a brushmodel that (1) doesn't have a touch function, and (2)
					// is an instanced BSP model, so we switch the solid and movetype
					ent->v.solid = SOLID_BSP;
					ent->v.movetype = MOVETYPE_PUSH;

					// flag that we're hacking the model
					ebjump = true;
				}

* A few years ago, me and Bam were working on a CTF mod and Bam made the Chthon appear on E1M7 despite being a multiplayer CTF mod. The Chthon didn't do anything, he just sat there. But he was good for blocking rockets. Skilled players would use him as a bridge, like the SDA guys. It was hard because you slid all over the place on the Chthon.

RMQ Item #15 - A change to SV_ClipToLinks adding a goto. I'm afraid I don't know what this does, but I am betting it is better collision with something in some circumstance.

It could be something like keeping fish and scrags from moving through solids in rare circumstances. I notice the RMQ engine plays the water splash sound throwing grenades through water, so it could be that. It could also somehow be necessary for the rain effect to collide properly with surfaces (maybe it resulted in the grenade/water thing as a side effect). But really, I don't know what this does.

[goldenboy]

It's like the Incredible Hulk of Quake engines.

Hardware requirements, btw, must reflect the hardware that is prevalent among users. What good is DirectX 11 if only 10% of mankind has hardware that can run it.

[taniwha]

What, 600 million isn't enough?

[goldenboy]

Depends what your standards are.

Within the RMQ team itself, not everybody has DirectX11 capable hardware. Not everybody on the team can even run the mod.

I hope this doesn't come as a shock to you.

Developers have to figure these things into their work. You can't just ignore this, or you're developing inside a bubble.