Memory Management Success + FAILS

[Baker]

I am annoyed that when I run my engine it quickly shows in Windows taskmanager that it is using 120MB. It'll say that with DarkPlaces or FTEQW or Qrack as well. Or DirectQ if using external textures.

Now lemme throw this in, load up ARWOP (A Roman Wilderness of Pain) first map ROMAN1 ... which precaches tons of models and sounds. I mean lots. And it busts quite a few engine limits and requires FitzQuake 0.85 or DirectQ to run it.

Now all of the model precaches, sound precaches and all of the memory total is a mere 60MB on the hunk. No kidding, just 60 MB.

So why is it showing as 120MB for an engine with external textures on the start map and then why does it balloon up to some number like 200MB or higher with ROMAN1 running?

Starting an engine to console, you might have 20MB showing in task manager. And if you look at the hunk, you've got something like a measly 3MB being used. The model cache and sound caches are empty ... well ... they might not be since you have the automatic precaches (ambient sounds and stuff like bolt1.mdl).

I rewrote my engine's memory management to essentially 4 hunks ...

1. the "low hunk" ... everything up to host_initialized, which is stuff like the palette, commands, the console background, qsockets).
2. the "level hunk" ... everything that gets cleared on map load.
3. the "models/sound hunk" ... precache guys that might persist from level to level
4. the temp hunk.

And I made commands to erase them each.

Guess what? Virtually NO DIFFERENCE in Windows taskmanager after loading a map it shows 120 MB even if I reduce the hunk allocations. Not even killing off the hunks ...free (level_hunk);

No when I'm sitting at the console at startup with 25MB being used according to Windows Task Manager, I have video, the engine is rendering open GL, opening the menu makes sound, I have mouse input.

Allocating 120 MB hunk? Windows task manager still shows 120MB of memory usage.

Even doing something silly like instead of malloc to allocate the hunk, I did ... LAFF ...

Code: Select all

byte myHunk[128 * 1024 * 1024]

And I used that as the hunk .... 120 MB memory usage on the start map. UH? What? My hunk is larger than that.

Going on the idea that Windows might intelligently decide what memory is being used, I rewrote image processing to COMPLETELY avoid mallocs and actually toss the stuff on the hunk using Hunk_AllocName and carefully saving the hunk mark before the few texture processing steps.

NO DIFFERENCE!

Now ... add to this. I changed cl_efrags[MAX_EFRAGS], lightmaps[MAX_LIGHTMAPS] and those guys to be hunk allocated instead of static arrays. And memory usage was reported higher!

In fact, when I changed lightmaps to the hunk, my frames per second when to hell in addition to the memory usage going up.

I also tried only putting cl_entities and sv edicts on the hunk, to try to reduce memory usage. No difference. I even came up with an awesome way to dynamically allocate entities once level (you know, ARWOP ROMAN1 being a huge map, it only has about 700 entities on map load).

I rewrote the SV_SpawnServer to read the entities as early as possible and count "classname" within the string, went something like this ...

Code: Select all

char *helper = my_entities_string;

for (helper=strstr(helper, "classname"); helper; count++, strstr (helper, "classname")
___ helper += strlen ("classname");

// Written above in an easy way to understand, I'd never write it wastefully like that recalcing strlen("classname") countless times for no reason and it is obviously a function.

Now this will say 385 entities for "start.bsp" but if you check sv edicts or cl_num_entities it'll be like 85 because many of those are lights. Still, in advance you can use the count to estimate a reasonable number of entities and allocate that in SV_SpawnServer and CL_ParseServerInfo. svedicts are always on the hunk, and FitzQuake puts client edicts on the hunk at the start of every map.

So, while all my efforts to defeat Windows Taskmanager reporting on memory have failed, if I ever encounter a lower memory device I want to write for, I'll sure do a better job handling memory scarcity.

I do not know why Windows reports memory the way it does. My engine like every other engine frees the temp image processing stuff after reading it in, mipmapping, etc. and I did 15 checks to verify beyond even an unreasonable doubt that this is always occurring.

DirectQ is about the only engine where you actually see memory usage drop. In some of my efforts, it did drop a few MB here and there, but it always perpetually climbed.

Furthermore, let's say the Windows task manager is true. Add to that the massive ARWOP only uses 60 MB of hunk. Add to that even when I did a static array of the size of 128 MB, it actually showed less memory usage than that. I think all of the above are almost sad proof there is no reason for sophisticated memory management in a stock Quake engine for Windows.

However, if I ever decide to code stuff for, like, say the PSP again ... the above frustrations will prove super-valuable. Fortunately, I only spent about 2-3 days on all the above. In particular, sv edicts are very costly.

[frag.machine]

Hmm, just a wild guess but... Could this memory being allocated by the libc and not your code actually ?

[mh]

The memory usage reported in task manager is more than just that you allocate yourself in your program. It includes allocations made by your drivers, other DLLs that the process loads, etc etc etc. Windows itself might even pre-emptively allocate some memory in case you need to use it sometime soon (so that it's available fast without needing to stall the program while it goes hunting for a free block big enough).

Just to complicate things further - in most cases this won't be actually allocated "real" memory. On a 32-bit PC you can address up to 4GB of memory, right? Now, even if you only have 16 MB in your machine, Windows can actually still address the full 4GB. This - not your swapfile - is what's really meant by "virtual memory". A page table is used internally to map these virtual addresses (which are the ones you see in your program) to real memory addresses.

This 4 GB of potential space is split in two. 2 GB of it (it may be 3 GB on some machines depending on boot.ini/whatever settings) are addressable by your process, and 2 GB by the system (that's not the way it really is but it will suffice for this discussion). Every memory allocation call you make comes from that 2 GB region (that's why you sometimes see pointers with crazy high addresses even on low memory machines). That 2 GB is per-process; each process has it's own 2 GB region to play in. That's (one of the reasons) why you could see big 32-bit servers with more than 4 GB of memory back in the old days - each process could get a different 2 GB region of real physical memory. The other 2 GB is shared by all processes, but that's OK because a process generally can't access it directly (I suspect that on Win9x it was quite easy to circumvent this and cause all kinds of fun to happen).

Ready to go in at the deep end?

On Windows, VirtualAlloc is king, queen and knave of memory allocation. Every memory allocation you make in your program goes through it. malloc goes through HeapAlloc and HeapAlloc goes through VirtualAlloc. Now, the way VirtualAlloc works is that it has this concept of "reserved memory" and "committed memory".

Reserved memory is just a block of address space (not physical memory) marked off; it's obtained by calling VirtualAlloc with the reserve flag. It's a way of saying "this is my block of address space, there are many others like it but this one is mine". Windows promises not to make any other memory allocations from that block unless you call VirtualAlloc again specifying a range in that block and with the commit flag. It's actually kinda like Quake's hunk in a way, in that it's a range of memory that you can draw down from as required. Very very neat API.

It's important to note that reserved memory is NOT NOT NOT physically in use. No matter how much you reserve, your available physical memory will NOT NOT NOT go down (internal bookkeeping structs may knock a few bytes off it). Each process has the full 2 GB to play with. You can reserve 2 GB in one process and if you never commit any of it, you'll still be able to reserve 2 GB in another and you'll still have all of your free physical memory.

(Aside: I more than half suspect that many Weenix Loonies look at the reserved counters and go into paroxysms of joy about how "oh noes, Windoze is teh bloatware!")

Committed memory is regions of that reserved space that are currenly mapped to physical memory. That's the stuff that's actually in use. It gets a little more complex with page sizes and whatnot, but there's the view from a few miles up.

That's why you sometimes see advice about which task manager counters are the ones that really mean stuff. Unless you're looking at the commit size you're getting a false view, and that commit size may include many things that you don't allocate yourself (as well as stuff in the system's 2 GB region, which - because it's shared between processes - may even be the same chunk of physical memory used by many different processes, like a device driver for inst. Think of it as being like multiple entities using the same model. If 300 entities each use a 10k model you don't have 3000k used - you've only got 10k used. If you query the memory used by each entity individually, and if that query includes the model size, you may be seriously misled).

What kind of things? Textures for starters. Sure textures go into video RAM, but each is also backed up by a system memory copy. In ye olde dayes this didn't happen on consumer hardware and doing anything that caused your video mode to be reset trashed the contents of your video RAM. No system memory backup - textures get lost (or replaced by random garbage). That's why GLQuake went nuts when you tried to change the mode without destroying and recreating your context.

The moral of the story is - don't stress too much about task manager counters. Much of what you see there is totally outside of your control. Keep an eye on your own allocations, make sure that you do nothing stupid and let the OS and your drivers look after themselves. It'll be different on different people's machines anyway so not only is there nothing you can do about it, but it's utterly pointless to even try.

Why does DirectQ's memory usage go down? Because my memory manager is so damn awesome? In one sense, yeah. Most everything is dynamically allocated into one of three different types of memory pool (and there may be multiple concurrent instances of each type) and what's not needed doesn't get used. D3D9 obviously works different to OpenGL, so there may be stuff going on there too. Likewise in your driver.

[revelator]

on win7 dwm.exe is notorious for blowing the top of memory allocations if the program in use has even the slightest leak
i had some success using mss to showel most leaks out of code but damn its a bitch.
switching to xp takes quite a load of of the memory so its indeed OS specific as well.

[Spike]

further to what mh stated, you can select different memory displays in task manager. one is 'paged memory' or some such, which is the amount of physical memory which is actually used for that process and will not include unused memory like zero-filled bss sections.

quake's memory usage comes from multiple places. textures, malloc, bss, and some engines use virtualalloc. glquake never destroyed any textures (quake2 does, but not quake), but glDeleteTextures can free up that memory if you wish to use it. free() will tend not to release memory back to the system as its often not page aligned and system calls are expensive in malloc/free intensive code. bss cannot be freed other than by closing the library/program, but it also won't be paged in unless you actually poke it, so you can have a large bss (quake does this a lot) without paying any real penalties other than in coding style. virtualfree is the only way to guarantee that memory is released back to the system, the virtual memory subsystem also a nice way to zero-fill large blocks of memory quickly.
malloc may or may not page the memory in. On linux, malloc does _not_ allocate memory - it allocates address space which is only paged in when its needed. Typically this memory is identical to bss in functionality, except that more address space can be made available for it as needed. The heap provided by virtualalloc for malloc in windows is likely to be the same.

If your system runs low on memory, its quite likely that sections of your exe will be flushed to disk, as they can easily be reloaded without touching the page file. Also its quite likely that the exe wasn't actually fully loaded from disk in the first place, which is how .exe installers with large data payloads can function on a machine with only 8mb ram.
So yeah, the memory usage is a lie.

[Baker]

What kind of things? Textures for starters. Sure textures go into video RAM, but each is also backed up by a system memory copy. In ye olde dayes this didn't happen on consumer hardware and doing anything that caused your video mode to be reset trashed the contents of your video RAM. No system memory backup - textures get lost (or replaced by random garbage). That's why GLQuake went nuts when you tried to change the mode without destroying and recreating your context.

Ok, now I reconcile all of the above together. Which is why with video mode if you keep the same bpp, you don't have to upload the textures again.

Thanks. That ties most of the observations together and I feel like things make sense again.

[mh]

It's worth noting as well that many of the characteristics of Quake's memory management system stem from the fact that it had to run on a Pentium 60 with 8 MB of RAM. Unless you're explicitly targetting that kind of machine, much of what makes it complex can actually go away.

[Baker]

It's worth noting as well that many of the characteristics of Quake's memory management system stem from the fact that it had to run on a Pentium 60 with 8 MB of RAM. Unless you're explicitly targetting that kind of machine, much of what makes it complex can actually go away.

Yeah, I know and at one point when I was getting frustrated, I thought about just using heap.cpp from DirectQ. I had looked over DarkPlaces, FTE and DirectQ and also spent some time thinking about the hunk system in Quake and tried to make a decision.

Some mappers find the 256 color palette of Quake to be a creativity inducing limit and I remember reading some great thread @ Func once on limits.

The Quake memory doesn't get in my way much, I no longer have any mallocs for image loading anywhere.

I'm not "for" the fixed-hunk because of retrograde reasons or anything, Quake is setup to use a fixed block and most of the data gets cleared level to level. Besides most memory allocations are either cvar changes or temps for resizing images, mipmapping them, converting them 8 bpp to 32 bpp or some such thing.

[Spike]

the current heap memory sucks. there's no way to get notifications when stuff gets purged, leading to leaked vbos, glsl, textures, etc.
cache memory sucks for the same reason.

malloc and free are 'easy' in a sense. all that is malloced must be freed. No uncertainty over who owns it, hopefully.

[mh]

On Windows, the Heap- functions rock.

You can create as many heaps as you want - one for permanent data, one for per-game, one for per-map, etc.

One call to HeapDestroy will release all memory in a heap. No need to track and free individual allocations.

HeapCompact will look after memory fragmentation for you.

There are HeapLock and HeapUnlock functions for thread synchronization.

You can retrieve info about a Heap, enumerate all of it's allocations, validate it or get it's total size any time you want.

You would still need to track other objects associated with heap allocations (like wot Spike said) but otherwise the functions are totally awesome.

[Spike]

I meant the quake hunk memory, sorry.

[mh]

I meant the quake hunk memory, sorry.

Yeah, I know. It occurred to me that it coulda been mixed up.

And you're right - Quake hunk memory is horrible.