-zone to prevent crashing

[necros]

so a few months ago, i was getting crashes and mh clued me in that i needed to use -zone to prevent from happening. apparently, the progs.dat things are stored in a different part of memory than textures and models and such and -zone controls that.

ok, is there like a rule of thumb on how large -zone needs to be relative to your progs?

like, should it just be larger than your progs.dat file? or is it more than that?

the stock id progs is about 400kb but i only started getting crashes around (iirc?) 1.2mb-ish and a -zone 2048 fixed it.

is there an upper limit to progs.dat file size?

also, is -zone size related to the amount of entity fields you have vs how many entities are present in the map?

ie: would having 10000 entities require more -zone space than 100?

[Baker]

so a few months ago, i was getting crashes and mh clued me in that i needed to use -zone to prevent from happening. apparently, the progs.dat things are stored in a different part of memory than textures and models and such and -zone controls that.

ok, is there like a rule of thumb on how large -zone needs to be relative to your progs?

like, should it just be larger than your progs.dat file? or is it more than that?

the stock id progs is about 400kb but i only started getting crashes around (iirc?) 1.2mb-ish and a -zone 2048 fixed it.

is there an upper limit to progs.dat file size?

also, is -zone size related to the amount of entity fields you have vs how many entities are present in the map?

ie: would having 10000 entities require more -zone space than 100?

-zone is the small strings allocation block. It is used for loading config.cfg and autoexec.cfg and aliases, cvar stuff and temporary string allocations and other miscellaneous little and short term things.

And I think FitzQuake puts gamedir stuff and max_edicts and allocates the console (65536 bytes by default) there. So FitzQuake needs more zone memory allocation than other engines for the same behavior.

So with FitzQuake, the number of entities DOES matter for your -zone allocation, but other engines use the hunk (-heapsize and -mem ... with a bit of variance from engine to engine but they do the same thing. I do not believe FitzQuake supports -mem which is intended to be MB versus -heapsize which is KB).

/About 85% confident of the answer. MH obviously has far more in-depth FitzQuake experience modifying the RMQ engine and being a vastly more experienced engine coder. Of course, Metlslime might read this too.

[qbism]

Are you getting "Hunk_Check: trahsed sentinal" or similar message, or just crash?

Standard Quake is 48k, FitzQ is 256k. -zone 2048 overrides it to 2048k, obviously much larger. In either case, that memory is carved from the hunk. I don't think memory overruns from zone to the rest of hunk are checked every time unless "PARANOID" is defined when compiling the engine.

Zone is nested in hunk. It will be listed near the top when entering "hunklist" at the console.

I just noticed this function in zone.c that apparently is never called

Code: Select all

void Z_Print (memzone_t *zone)
{
	memblock_t	*block;

	Con_Printf ("zone size: %i  location: %p\n",mainzone->size,mainzone);

	for (block = zone->blocklist.next ; ; block = block->next)
	{
		Con_Printf ("block:%p    size:%7i    tag:%3i\n",
			block, block->size, block->tag);

		if (block->next == &zone->blocklist)
			break;			// all blocks have been hit
		if ( (byte *)block + block->size != (byte *)block->next)
			Con_Printf ("ERROR: block size does not touch the next block\n");
		if ( block->next->prev != block)
			Con_Printf ("ERROR: next block doesn't have proper back link\n");
		if (!block->tag && !block->next->tag)
			Con_Printf ("ERROR: two consecutive free blocks\n");
	}
}

[qbism]

Add zoneprint command mini-tute:

in zone.c above hunc_allocname, add

Code: Select all

/*
===================
Zone_Print_f //qbism
===================
*/
void Zone_Print_f(void)
{
	Z_Print(mainzone);
}

In zone.c, near bottom within memory_init, add

Code: Select all

	Cmd_AddCommand ("zonelist", Zone_Print_f);	//qbism- implement a previously unused function

When I play Marcher Fortress, getting 197K with zonelist. Is that really the zone size? Would it vary depending on the engine or other limits? Needs more research.

[Baker]

After consulting the FitzQuake source, edicts are allocated on the Hunk ... the big block of memory, which doesn't have anything to do with -zone memory. Likewise, FitzQuake isn't allocating the console buffer on the hunk either. My bad x 2.

As the progs.dat is also loaded on the Hunk, the -zone allocation has nothing to do with progs.dat size.

Main factors:

1. Size of config.cfg, autoexec.cfg, quake.rc and aliases. (Command buffer ... it loads all those files into memory)
2. Cvars ... should be a non-issue. But there are a lot of cvars and FitzQuake allocates 2 (usually) small slots of space for each cvar. Should be trivial.
3. Gamedir allocation (trivial).

Zone allocation has nothing to do with the number of entities nor the progs.dat size, not even with FitzQuake. The progs.dat and entities are allocated to the "Hunk" ... which is the main memory block like when you do -heapsize 130000 (size in kb).

Still the Quake default zone allocation is too small a lot of times, and mods with a lot of aliases have historically crashed with the message "Z_Malloc: failed on allocation of xxx bytes".

/Baker slaps self for not giving "right" answer the first time, but I was watching a movie and didn't have Fitz source available. Baker -5.

[mh]

Zone allocation really should move to malloc/free in a modern engine. There's no real reason for it to exist any more as you're no longer dealing with an absolute hard limit of 8 MB which everything must fit into. We have virtual memory now, we have 4 GB of address space now, we have operating systems that can do memory management now.

It's noteworthy that ID themselves switched it to malloc/free for Quake II.

Some might worry about memory fragmentation becoming a problem, but it wasn't a problem in Quake II, was it?

Other stuff that goes into the zone: key bindings, the command buffer, cvar values. COM_LoadFile has an option to load a file into zone memory but it's never used in the engine. Probably a relic from an earlier architecture (Doom used nothing but zone memory). FitzQuake puts PAK file and search path info into zone memory which can generate a bit more pressure.

Finally, FRIK_FILE does some rather evil things with zone memory - opening a file in append mode is not pretty (could Sys_FileOpenAppend not have been written?)

[necros]

hm... ok, now i'm more confused. i guess i'll just keep suggesting putting -zone 2048 then, because it most definitely does fix the crash.
but you guys seem to be telling me it has absolutely nothing to do with the progs or edicts. o.o

[mh]

Double-checked the Fitz source and there's definitely nothing from progs or edicts going into zone memory. The one other thing it puts in there is the GL_EXTENSIONS list, but that's never going to be huge. That really leaves search paths and the PAK file directories as the most likely culprit.

I'd love to reproduce this in a debugger so that I could tell you exactly what was causing it.

[qbism]

hm... ok, now i'm more confused. i guess i'll just keep suggesting putting -zone 2048 then, because it most definitely does fix the crash.

Seems strange to need more than -zone 256, even with large maps and lots of monsters. And fitzquake is already 256 as default.

[mh]

More than about 4000 files in PAKs would overflow it; with Quake's default MAX_FILES_IN_PACK of 2048 and if someone had lots of custom content, it's possible to see it happening.

[Baker]

hm... ok, now i'm more confused. i guess i'll just keep suggesting putting -zone 2048 then, because it most definitely does fix the crash.

Seems strange to need more than -zone 256, even with large maps and lots of monsters. And fitzquake is already 256 as default.

CTF and RuneQuake and many other mods crash the default zone allocation of 256KB and have for a very long time. It is due to the number of aliases. Take into account temp strings are not necessarily done efficiently in the zone allocation (think of a hard drive that really needs to be defragged --- there might be X space available, but not continugously (!!!)) That is Quake's zone memory for you. The hunk, on the other hand, is largely wiped clean from map to map ... like a true "toss everything out and restart".

So it doesn't have anything to do with large maps or lots of monsters, I can run ARWOP or a map with 400 monsters (Masque of the Red Death) in FitzQuake or Quakespasm with the default zone allocation of 256 just fine (in otherwords, I didn't use -zone anything).

Obviously Necros is getting a real crash from the default zone allocation and a larger one is solving it.

MH's pak file comment in above post might be on to something because Necros is very likely using Quoth which uses 3 paks, plus the 2 id paks, PLUS adds hipnotic to the gamedir *and* Necros probably has his stuff in its own separate gamedir.

That is at least 6 pak files at the absolute minimum right there ... and doesn't hipnotic, if Necros has this installed, use several paks? ... I can't recall where/how FitzQuake's autocomplete of map names is stored, but this "lots of files and paks" thing seems to be on the mark.

[mh]

I can't recall where/how FitzQuake's autocomplete of map names is stored

Zone again, 36 bytes each.

Strikes me that moving at least this much (PAK directories and autocomplete) to malloc/free would be of immediate benefit.

[Baker]

I can't recall where/how FitzQuake's autocomplete of map names is stored

Zone again, 36 bytes each.

Strikes me that moving at least this much (PAK directories and autocomplete) to malloc/free would be of immediate benefit.

^^^ That

I'd kind of hope that someone like Necros has a crapton of bsps in his maps folder.

@Necros ... I think this specific problem is just due to FitzQuake features. I bet if you loaded up aguirRe's Enhanced GLQuake it'd load the maps without any modification to the -zone parameter. FitzQuake has little nibbles here and there that chip away at the zone block which are different than other engines and usually don't matter. But once you add lots of pakfiles, gamedirs plus autocomplete space for maps ... if these quantities exist well above the normal, -zone memory can be exhausted with the default.

It isn't progs.dat size, the number of monsters or entities or brushes or the number of textures ... those are not allocated there.

[mh]

OK, I've just gone through the RMQ source and switched everything aside from the command buffer (Cmd_TokenizeString) to malloc (calloc actually)/free. I could probably switch the command buffer, but I'm slightly paranoid about it as it needs to run every frame. Switching it causes a heap corruption anyway, so there's obviously some memory still being referenced after the Z_Free, which is OK with zone memory but obviously not with standard free. It would be nice to fix it up properly but it's not that big a deal overall.

Update: just did the command buffer. If anyone's interested, this is how I handled it:

Code: Select all

void Cmd_TokenizeString (char *text)
{
	int		i;

	// the maximum com_token is 1024 so the command buffer will never be larger than this
	static char argbuf[MAX_ARGS * (1024 + 1)];
	char *currarg = argbuf;

	cmd_argc = 0;
	cmd_args = NULL;

	while (1)
	{
		// skip whitespace up to a /n
		while (*text && *text <= ' ' && *text != '\n')
			text++;

		if (*text == '\n')
		{
			// a newline seperates commands in the buffer
			text++;
			break;
		}

		if (!*text) return;
		if (cmd_argc == 1) cmd_args = text;
		if (!(text = COM_Parse (text))) return;

		if (cmd_argc < MAX_ARGS)
		{
			cmd_argv[cmd_argc] = currarg;
			Q_strcpy (cmd_argv[cmd_argc], com_token);
			currarg += Q_strlen (com_token) + 1;
			cmd_argc++;
		}
	}
}

[Lardarse]

malloc (calloc actually)

What's the difference?