Quake Memory Manager - Part 1: the Zone

[mh]

I'm going to do a series of tutorials in which we tackle the most maligned of wee beasties, the Quake memory manager. This code will be based on standard Win/GL Quake and will only require changes to be made to the zone.c file. I'm maintaining the exact same interface as the original Quake memory manager so that anyone can easily use it in their own engines.

This code will be usable with both software and GLQuake, and in fact I'm developing and testing it on software Quake to ensure this. Where possible I'm writing it to be completely portable, although unfortunately the Hunk is most likely going to break this objective.

For part 1 we're going to tackle the Zone. This is the region of Quake's memory which is used for allocation of small strings and other didgeridoos, although at the end of this part you'll be able to use it for whatever you want, no matter how big.

Concept

By default the Zone is a very tight region (47 K) and shares it's memory space with the rest of Quake's memory. Sometimes Quake runs out of Zone space and crashes, so you need to restart with the -zone command-line option and replay up to the point where it crashed.

Here we're going to completely remove that limitation (the only limitation will be the address space of your OS) and also remove the Zone from the shared memory area (so that Zone corruptions don't corrupt elsewhere).

Code Changes

We're going to replace every single Zone function with standard C memory functions instead. I'll give them all here, and make comments where required.

Code: Select all

void Z_ClearZone (memzone_t *zone, int size)
{
}

We don't need to use this function anymore, but we'll retain it for compatibility with code that might be elsewhere. This is the general pattern I'm going to follow throughout this series.

Code: Select all

void Z_Free (void *ptr)
{
	int *zblock = ((int *) ptr) - 1;

	if (zblock[0] != ZONEID)
		Sys_Error ("Z_Free: freed a pointer without ZONEID");

	free (zblock);
}

Note the ninja-trickery in this one. I'll discuss what's happening here after I give Z_Malloc.

Code: Select all

void *Z_Malloc (int size)
{
	int *zblock = (int *) malloc (size + sizeof (int));

	if (!zblock)
		Sys_Error ("Z_Malloc: failed on allocation of %i bytes", size);

	memset (zblock, 0, size + sizeof (int));

	zblock[0] = ZONEID;

	return (zblock + 1);
}

Right. What we're doing here is allocating an extra 4 bytes in addition to the requested size. The first 4 bytes of the buffer are set to the ZONEID value, which is used as a guard value in Z_Free. The buffer we return starts following those 4 bytes.

For Z_Free what we do is back up by 4 bytes from the address given (a pointer is just a memory address and you can go backwards from it as well as forwards), checking that the first 4 bytes equal ZONEID (in case we try to Z_Free something that wasn't allocated via Z_Malloc), then freeing it from that point.

Code: Select all

void *Z_TagMalloc (int size, int tag)
{
	return Z_Malloc (size);
}

This function was never actually used in the engine. I guess it might have been experimental QuakeII stuff as QuakeII does use Z_Malloc with tags.

Code: Select all

void Z_Print (memzone_t *zone)
{
}

void Z_CheckHeap (void)
{
}

Likewise we don't use these any more but we keep them to maintain a compatible interface.

Final code change is at the bottom of the Memory_Init function (go find it at the bottom of zone.c). See this block:

Code: Select all

	p = COM_CheckParm ("-zone");

	if (p)
	{
		if (p < com_argc-1)
			zonesize = Q_atoi (com_argv[p+1]) * 1024;
		else
			Sys_Error ("Memory_Init: you must specify a size in KB after -zone");
	}

	mainzone = Hunk_AllocName (zonesize, "zone" );
	Z_ClearZone (mainzone, zonesize);

We don't need it anymore as we no longer have Zone limitations and we certainly don't want to pollute our main memory with something that is never used. Remove it. Delete it. Take it outside and kill it until it is dead.

Cleaning Up

When I'm finished I'll release a fully modified and cleaned up Zone.c, but for now you can go through the file, identify any variables or structures that are no longer used, and delete them.

The next part

For the next part I think we're going to do the Cache. The Cache is heavily dependent on Hunk allocations, so we're likewise going to break that dependency and put it in it's own memory space, giving it room to grow as large as it wants.

When this is posted (in a coupla hours time ) you'll have a very clean and tight zone.c, that looks virtually unrecognisable compared to the old one.

And then we'll do the Hunk.

[metlslime]

...This should be good

[Baker]

I'm enjoying reading these, I'd like to learn more about memory management.

[Tomaz]

What codebase is this based on?
Like the comment about Z_TagMalloc not being used? and the code you pasted there? Thats NOT from the original quake code, in fact, all the Z alloc code does happen in Z_TagMalloc, since Z_Malloc pretty much does nothing but call Z_TagMalloc.

[revelator]

hmm aye tomaz is right Z_Malloc does indeed call Z_TagMalloc

and Z_Malloc is used in the vm and filesystem in some places ? allthough using plain mallloc or VirtualAlloc seems to do the trick just fine

[mh]

What codebase is this based on?

WinQuake. The version from ID.

Like the comment about Z_TagMalloc not being used? and the code you pasted there? Thats NOT from the original quake code, in fact, all the Z alloc code does happen in Z_TagMalloc, since Z_Malloc pretty much does nothing but call Z_TagMalloc.

Of course it's not from the original code, it's a code change. It seems a little unreasonable to me to expect code changes to be identical to the original code, unless I'm missing a point here.

Yes, I know that Z_Malloc calls Z_TagMalloc, but where else is Z_TagMalloc used in the engine? Nowhere.

Tomaz, I respect your work, but this isn't the first time you've made derisive comments about other people sharing their code and ideas here (I'll refer you to your outburst on the alpha tutorial a while back). Could you at least try to be constructive and contribute a little?

[Tomaz]

Of course it's not from the original code, it's a code change. It seems a little unreasonable to me to expect code changes to be identical to the original code, unless I'm missing a point here.

Yes, I know that Z_Malloc calls Z_TagMalloc, but where else is Z_TagMalloc used in the engine? Nowhere.

Tomaz, I respect your work, but this isn't the first time you've made derisive comments about other people sharing their code and ideas here (I'll refer you to your outburst on the alpha tutorial a while back). Could you at least try to be constructive and contribute a little?

I'm sorry that was never my intention. I respect your work as well, I just pointed this out since I found it to be false information. If I had not cared I would not have replied at all.

What I meant was that you made Z_TagMalloc call Z_Malloc, why? Since previously it was Z_Malloc calling Z_TagMalloc, so if Z_Malloc dont need Z_TagMalloc anymore, why not just remove it?

I understand that you made empty functions that does nothing when they functiosn are called from elsewhere, but in this case, there is nothing anywhere calling Z_TagMalloc, and again. I never meant to make a "derisive"?? ( don't even know what that means but it sounds bad ) comment about this code. I was going to add this to CleanQuakeCpp, which is when I found your comments about Z_TagMalloc to be.. weird.. in lack of a better word. My intention with the post was to get an answer to why it was made like that, not do disrespect your work.

And yes I know I'm not liked around here, or anywhere else for that matter. It's not that I am an evil person, I just never bother/have time to put down those extra words when i write comments that change them from just being mean pointing out errors, to be more constructive.

But I'll defenitly try to be more constructive in my comments in the future.





Edit:

But I have to defend myself, what was wrong regarding my comments on Baker's alpha tutorial? I merely pointed out 3 things that was redundant/wrong and explained why on all 3, and then gave a lengthy explenation of why an alpha sorter is needed.

What about that is bad and not constructive?

[Baker]

Edit:

But I have to defend myself, what was wrong regarding my comments on Baker's alpha tutorial? I merely pointed out 3 things that was redundant/wrong and explained why on all 3, and then gave a lengthy explenation of why an alpha sorter is needed.

What about that is bad and not constructive?

Your comments on my alpha tutorial allowed me to fix alpha in my engine. I needed to sort the entities. I don't believe I would have figured it out by myself very quickly.

/quick point, exits conversation ...

[frag.machine]

Guys... From what I understand, there were no intention to be sarcastic, aggressive or something like that from any part. To me, that's one good example on how having a written conversation can be... complicated, sometimes. Specially to people like me, to whom English is not the main language. That's why emoticons are so valuable: they are there to tell to the other side "hey, don't take everything I am writing here so seriously or in the wrong way".

And now, for something completely unrelated...



/me goes back to my coding hole trying to make moving liquid brushes work

[c0burn]

Guys... From what I understand, there were no intention to be sarcastic, aggressive or something like that from any part. To me, that's one good example on how having a written conversation can be... complicated, sometimes. Specially to people like me, to whom English is not the main language. That's why emoticons are so valuable: they are there to tell to the other side "hey, don't take everything I am writing here so seriously or in the wrong way".

And now, for something completely unrelated...



/me goes back to my coding hole trying to make moving liquid brushes work

Engine or QC? Extras_r4 did it well.

[Spike]

Z_TagMalloc is useful for FRIK_FILE and unclean mods. Really really useful. I would not otherwise recommend its use.

[frag.machine]

/me goes back to my coding hole trying to make moving liquid brushes work

Engine or QC? Extras_r4 did it well.

I started using a combined QuakeC-engine solution, now I am trying engine-side only. We were discussing about this here.

[Teiman]

I did some horrible things in Telejano:

- Moved away from the program-managed memory to OS managed memory. Uh.
- Using a closed source proprietary module to detect memory leaks. I was very vigilant at this, maybe obsessive.

The whole game navigated away from that "16 MB" original footprint of quake to a whole universe of bloat.