Lazy Texture Reload Question

Discuss programming topics for the various GPL'd game engine sources.
Post Reply
Baker
Posts: 3666
Joined: Tue Mar 14, 2006 5:15 am

Lazy Texture Reload Question

Post by Baker »

I'm a few days away from a new client release (ProQuake 4.95) and a new server version on a test server (and ... well.. plus a new mod :D).

The Client: Mostly @MH here ...
1) I've absorbed many of my FitzQuake Mark V changes (hehe :D ) and well --- back 2 years ago my coding sucked, was a somewhat hacky, etc. I've rapidly largely cleaned up the codebase (viva WinMerge efficiency). Except I haven't tackled the 2 thorniest issues (video and input cleanups).
2) The D3D and OpenGL builds are in a singular binary, even more so than Engine X. Still not as slick as, say, FTE. Requires engine restart to switch. Like Engine X, I abuse the wrapper to give it vsync ability and such :D Which it doesn't really support, but hey where is the fun with living within clearly defined limits.
3) Graphical niceties? Zero. I've played Black Mesa and after maybe 15 minutes, I dug through a stack of CDs to find the REAL Half-Life. I kind of see ProQuake as the last refuge of archaic GLQuake. ProQuake feels like ugly old GLQuake in a way that even DirectQ or FitzQuake don't give off --- and every engine needs its specialty and ProQuake's is gonna be ugly 1996 GLQuake smudgy feel. Kind of like how Doom and Doom II should be dated.

The Server: Twenty kinds of administrative ease. Where administrative ease = if there is trouble, be able to handle it with ease. And with options more gentle and creative than bans. And loaded with stats. (Thanks libcurl!!! :D ) The admin controls are rather "robust" --- even by rather imaginative standards of "robust". :D Kinda Robocop-like :mrgreen:

Here's the real question: For anyone that knows (I suspect Spike does) ... if I want to swap renderers, any thoughts on the most hassle free way to reload all the textures? Right now it requires an engine restart, and I don't want to spend 2 weeks to write a texture manager.

/Yeah, sure long-winded post for such a simple question. Enough said. :D
The night is young. How else can I annoy the world before sunsrise? 8) Inquisitive minds want to know ! And if they don't -- well like that ever has stopped me before ..
Spike
Posts: 2914
Joined: Fri Nov 05, 2004 3:12 am
Location: UK
Contact:

Re: Lazy Texture Reload Question

Post by Spike »

your d3d renderer is just a gl wrapper. thus assuming your wrapper is written correctly and can shutdown/restart, and you can shutdown/restart opengl cleanly too, switching between d3d and gl at run time should not be a problem.

The most hassle-free way to reload all the textures is to share the textures from one context to the next, then you don't have to destroy any of them. Note that I wouldn't trust the drivers to not mess things up, but hey, the idea is there.
The second-most hassle-free way is to just purge everything and restart from scratch. You'd be doing this on a gamedir change anyway. Models can be flushed with... well, with the 'flush' command. Maps are purged on map changes. Cachepics are generally queried by name before use. You might have to reload the menu but that's it. Wad, sbar, conchars, maybe palette will each need special care. Presumably you'll need to reload the wad itself from disk. In place or whatever, then just rerun the hud init code.
The trick is doing all of that without destroying the current game.
Write proper startup and shutdown code in the first place and the entire program becomes easy to start and shut down on demand. Grr @ iD software.
NPAPI support basically mandated that I get FTE into a state where it can start+shutdown cleanly multiple times in a row, without leaking (too much) memory. When you're in someone else's process, calling exit(0) is evil. Yeah yeah, I know, I shoulda just started a new process using the browser's window as a parent, but blurgh, my way is cooler, even if it doesn't allow multiple plugin instances at once.
mh
Posts: 2292
Joined: Sat Jan 12, 2008 1:38 am

Re: Lazy Texture Reload Question

Post by mh »

You could begin by reusing the video mode changing code for this. You've a clear heritage from FitzQuake here so you've a decent (ish) headstart on it as that just totally destroys and recreates the GL context (and main window) so you're starting from a clean slate each time.

What will cause you trouble is that mode creation and changing under D3D is quite different than under GL, so you'll need to beat around the code a little to get it working (which may end up taking you longer than the two weeks to write a texture manager....) - a true native implementation of mode changing under D3D8 (or 9) looks absolutely nothing like the contortions that wrapper needs to go through... Anyway, a quick look over your VID_Restart_f function shows that the best place to start is probably by dropping the "#ifdef DX8QUAKE" stuff in there and using the GL path for both; of course the D3D8 wrapper also intercepts ChangeDisplaySettings, the wgl calls, etc, so you'll need to jump into the appropriate place in your code and call the correct function.

Having both renderers coexist in a single binary and be switchable at runtime means that you'll also need to resolve naming conflicts; e.g. you'll be linking to opengl32.lib which has it's own glEnable, and the wrapper also has a version of glEnable. Something like Quake 2's qgl system will be needed throughout your code to get that working clean.

Either way it's messy and involves a lot of work.

Personally I think that a higher level of abstraction is the real way to go with this, and will cause you much less trouble longer-term. So instead of going down to individually wrapped GL functions you'd make e.g. "GL_DrawSurface" and "D3D_DrawSurface" routines and they look after the drawing of a surface with real native code in each. The D3D8 wrapper was never really designed for this level of robustness/abuse (it has more in common with the old 3DFX mini-GL than it does with a full OpenGL implementation) so that may be your best option (and should give you significantly higher performance under D3D8 too). It does mean a complete write-from-scratch of a D3D8 renderer though, but in the long term it will pay you back.

The other option I'd favour is to junk the D3D8 renderer. I know the reasons for wanting to keep it, but the OpenGL situation is considerably less grim than it was 2/3/4 years ago, and most of the trouble-spots on the likes of Intel hardware are well known and can be avoided, so maybe it's time to re-evaluate those reasons?
We had the power, we had the space, we had a sense of time and place
We knew the words, we knew the score, we knew what we were fighting for
Baker
Posts: 3666
Joined: Tue Mar 14, 2006 5:15 am

Re: Lazy Texture Reload Question

Post by Baker »

mh wrote:The D3D8 wrapper was never really designed for this level of robustness/abuse (it has more in common with the old 3DFX mini-GL than it does with a full OpenGL implementation) so that may be your best option (and should give you significantly higher performance under D3D8 too). It does mean a complete write-from-scratch of a D3D8 renderer though, but in the long term it will pay you back.

The other option I'd favour is to junk the D3D8 renderer. I know the reasons for wanting to keep it, but the OpenGL situation is considerably less grim than it was 2/3/4 years ago, and most of the trouble-spots on the likes of Intel hardware are well known and can be avoided, so maybe it's time to re-evaluate those reasons?
I've considered that as well, still the overhead to maintain the DX8 wrapper is minimal and I've got the performance so high (as you witnessed firsthand, sometimes it can actually beat OpenGL --- on my ATI mobility I get 50% higher FPS). Considering every once in a while someone has headaches with OpenGL drivers, that is a worthy reason to keep it, in my opinion.

BTW, my current codebase doesn't have any of those #ifdefs (which wouldn't be helpful in a dual renderer version). The .exe size itself is just 620KB.

Anyway, it sounds like there are no "easy" "easy" answers. I'll do some thinking ....
The night is young. How else can I annoy the world before sunsrise? 8) Inquisitive minds want to know ! And if they don't -- well like that ever has stopped me before ..
Spike
Posts: 2914
Joined: Fri Nov 05, 2004 3:12 am
Location: UK
Contact:

Re: Lazy Texture Reload Question

Post by Spike »

Baker wrote:I've considered that as well, still the overhead to maintain the DX8 wrapper is minimal and I've got the performance so high (as you witnessed firsthand, sometimes it can actually beat OpenGL --- on my ATI mobility I get 50% higher FPS).
tbh that just means you need to batch things a bit better like the d3d wrapper does for you.
mh
Posts: 2292
Joined: Sat Jan 12, 2008 1:38 am

Re: Lazy Texture Reload Question

Post by mh »

Spike wrote:
Baker wrote:I've considered that as well, still the overhead to maintain the DX8 wrapper is minimal and I've got the performance so high (as you witnessed firsthand, sometimes it can actually beat OpenGL --- on my ATI mobility I get 50% higher FPS).
tbh that just means you need to batch things a bit better like the d3d wrapper does for you.
This is more or less it. The D3D8 wrapper does batch things quite effectively; if you look at all the GL_SubmitVertexes calls that happen on state changes, you'll see that it gathers up verts as they're drawn and adds them to an array, then when it needs to change state it submits everything gathered so far and starts again, so you basically get one draw call per bundle of state. That's at the expense of a lot of unnecessary CPU overhead copying and moving data around, which a native implementation (either GL or D3D) wouldn't have.

The same applies to lightmaps. By wrapping glTexSubImage it needs to first build the lightmap update in system memory, then lock the texture rect, then copy it in. A native implementation would just lock the texture rect (with D3DLOCK_NO_DIRTY_UPDATE) in R_BuildLightmap and write in directly, then call AddDirtyRect sometime later on to complete the update, like so:

Code: Select all

	if (SUCCEEDED (IDirect3DTexture8_LockRect (lightmap_textures[surf->lightmaptexturenum], 0, &lockrect, NULL, D3DLOCK_NO_DIRTY_UPDATE)))
	{
		int stride = lockrect.Pitch >> 2;
		RECT *dirtyrect = &lightmap_dirtyrect[surf->lightmaptexturenum];
		unsigned *dest = ((unsigned *) lockrect.pBits) + surf->light_t * LIGHTMAP_SIZE + surf->light_s;

		for (i = 0; i < surf->tmax; i++, dest += stride)
		{
			for (j = 0; j < surf->smax; j++, blocklights += 3)
			{
				// 2x overbrighting is not enough for the full range of lighting so we'll use 4x instead
				if (blocklights[0] > 0x1fe00) blocklights[0] = 255; else blocklights[0] >>= 9;
				if (blocklights[1] > 0x1fe00) blocklights[1] = 255; else blocklights[1] >>= 9;
				if (blocklights[2] > 0x1fe00) blocklights[2] = 255; else blocklights[2] >>= 9;

				dest[j] = 0xff000000 | (blocklights[0] << 16) | (blocklights[1] << 8) | blocklights[2];
			}
		}

		IDirect3DTexture8_UnlockRect (lightmap_textures[surf->lightmaptexturenum], 0);

		if (surf->lightrect.left < dirtyrect->left) dirtyrect->left = surf->lightrect.left;
		if (surf->lightrect.right > dirtyrect->right) dirtyrect->right = surf->lightrect.right;
		if (surf->lightrect.top < dirtyrect->top) dirtyrect->top = surf->lightrect.top;
		if (surf->lightrect.bottom > dirtyrect->bottom) dirtyrect->bottom = surf->lightrect.bottom;

		lightmap_modified[surf->lightmaptexturenum] = true;
	}
Batching is possible in GL1.1 with the gl*Pointer calls and glDrawArrays/glDrawElements, but it does need a bit of heavy reworking of certain parts of the code, and some changes in the way you think about drawing stuff. You can't just shoot off vertexes on the fly from any arbitrary source, instead you need to plan things out in advance more. Without going all the way to there, it's also possible to do a simpler implementation and just set up your surface verts so that you can make one set of gl*Pointer calls per model, and one glDrawArrays (GL_POLYGON, ... per surface, which on it's own would be a huge improvement over what GLQuake currently has. Likewise each MDL can be handled in a single draw call.

Get all of that done and you'll find that the performance advantage of the D3D8 wrapper disappears. There's nothing special about the wrapper; it just tries to beat some sensible drawing out of GLQuake's botch job, but doing it for real in GLQuake itself would be better. So it's not that the wrapper is particularly fast or hyper-efficient, more that GLQuake itself is incredibly poor and it's making the wrapper look good.
We had the power, we had the space, we had a sense of time and place
We knew the words, we knew the score, we knew what we were fighting for
Post Reply