compressed pk3 support tutorial for standard quake soonish
compressed pk3 support tutorial for standard quake soonish
due to work it takes a bit for me to drop this one here so if anyone feels the urge feel free to rip it from realm
its not quite done yet but close only part of the old system i still need to mangle in are the demos everything else can be loaded from pk3's.
i might have to rewrite parts of it since i use newer hunk allocation system mh made so its not quite straight forward the changes are pretty massive tbh :/ but all in the name of good memory handling.
it can litterally take 1000's of textures models etc but the loadtime increases quite a bit if you overdo it it doesnt have a great impact on speed once loaded though.
it also makes it easier for modders to include special parts since only the relevant data gets loaded "if your pk3's are in the same dir as the mod" it wont load textures etc from id1 besides the mips ofc.
its not quite done yet but close only part of the old system i still need to mangle in are the demos everything else can be loaded from pk3's.
i might have to rewrite parts of it since i use newer hunk allocation system mh made so its not quite straight forward the changes are pretty massive tbh :/ but all in the name of good memory handling.
it can litterally take 1000's of textures models etc but the loadtime increases quite a bit if you overdo it it doesnt have a great impact on speed once loaded though.
it also makes it easier for modders to include special parts since only the relevant data gets loaded "if your pk3's are in the same dir as the mod" it wont load textures etc from id1 besides the mips ofc.
If you're worried about loading speed, you could potentially sort the file names somehow. Then do a binary search on them whenever a file is loaded. Or you could build a hash table of them all.
It really depends how many different textures you try to load.
*.pcx, *.tga, *.lmp, *.jpg, *.jpeg, *.png, *_bump.*, *_luma.*, *_norm.*, *_gloss.*, *_glow.*, *_pants.*, *_shirt.*, textures/*.*, textures/$mapname/*.*, override/*.*, textures/exmx/*.*, etc.
id1/*, qw/*, $engine/*, $gamedir/*
And all the combinations thereof.
Really depends how many other engines you try to be compatible with. :)
But yeah, pk3 files don't have to be any slower to search than pak files, they just generally have more files inside.
It really depends how many different textures you try to load.
*.pcx, *.tga, *.lmp, *.jpg, *.jpeg, *.png, *_bump.*, *_luma.*, *_norm.*, *_gloss.*, *_glow.*, *_pants.*, *_shirt.*, textures/*.*, textures/$mapname/*.*, override/*.*, textures/exmx/*.*, etc.
id1/*, qw/*, $engine/*, $gamedir/*
And all the combinations thereof.
Really depends how many other engines you try to be compatible with. :)
But yeah, pk3 files don't have to be any slower to search than pak files, they just generally have more files inside.
It's more a case that the new Hunk_Alloc system is slower, as it's quite prone to "lots of small allocations" syndrome.
This one is better.
This one is better.
Code: Select all
/*
Copyright (C) 1996-1997 Id Software, Inc.
This program is free software; you can redistribute it and/or
modify it under the terms of the GNU General Public License
as published by the Free Software Foundation; either version 2
of the License, or (at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
See the GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program; if not, write to the Free Software
Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
*/
#include "quakedef.h"
/*
========================================================================================================================
UTILITY FUNCTIONS
========================================================================================================================
*/
int MemoryRoundSize (int size, int roundtok)
{
// kilobytes
roundtok *= 1024;
for (int newsize = 0;; newsize += roundtok)
if (newsize >= size)
return newsize;
// never reached
return size;
}
/*
========================================================================================================================
CACHE MEMORY
Certain objects which are loaded per map can be cached per game as they are reusable. The cache should
always be thrown out when the game changes, and may be discardable at any other time. The cache is just a
wrapper around the main virtual memory system, so use Virtual_PoolFree to discard it.
========================================================================================================================
*/
typedef struct cacheobject_s
{
struct cacheobject_s *next;
void *data;
char *name;
} cacheobject_t;
cacheobject_t *cachehead = NULL;
int numcacheobjects = 0;
void *Cache_Check (char *name)
{
for (cacheobject_t *cache = cachehead; cache; cache = cache->next)
{
// these should never happen
if (!cache->name) continue;
if (!cache->data) continue;
if (!stricmp (cache->name, name))
{
Con_DPrintf ("Reusing %s from cache\n", cache->name);
return cache->data;
}
}
// not found in cache
return NULL;
}
void *Cache_Alloc (char *name, void *data, int size)
{
cacheobject_t *cache = (cacheobject_t *) Virtual_PoolAlloc (VIRTUAL_POOL_CACHE, sizeof (cacheobject_t));
// alloc on the cache
cache->name = (char *) Virtual_PoolAlloc (VIRTUAL_POOL_CACHE, strlen (name) + 1);
cache->data = Virtual_PoolAlloc (VIRTUAL_POOL_CACHE, size);
// copy in the name
strcpy (cache->name, name);
// count objects for reporting
numcacheobjects++;
// copy to the cache buffer
if (data) memcpy (cache->data, data, size);
// link it in
cache->next = cachehead;
cachehead = cache;
// return from the cache
return cache->data;
}
void Cache_Init (void)
{
cachehead = NULL;
numcacheobjects = 0;
}
/*
========================================================================================================================
VIRTUAL POOL BASED MEMORY SYSTEM
This is officially the future of DirectQ memory allocation. Instead of using lots of small itty bitty memory
chunks we instead use a number of large "pools", each of which is reserved but not yet committed in virtual
memory. We can then commit as we go, thus giving us the flexibility of (almost) unrestricted memory, but the
convenience of the old Hunk system (with everything consecutive in memory).
========================================================================================================================
*/
typedef struct vpool_s
{
char name[24];
int maxmem;
int lowmark;
int highmark;
byte *membase;
} vpool_t;
// these should be declared in the same order as the #defines in heap.h
vpool_t virtualpools[NUM_VIRTUAL_POOLS] =
{
// stuff in this pool is never freed while DirectQ is running
{"Permanent", 32 * 1024 * 1024, 0, 0, NULL},
// stuff in these pools persists for the duration of the game
{"This Game", 32 * 1024 * 1024, 0, 0, NULL},
{"Cache", 256 * 1024 * 1024, 0, 0, NULL},
// stuff in these pools persists for the duration of the map
// warpc only uses ~48 MB
{"This Map", 128 * 1024 * 1024, 0, 0, NULL},
{"Edicts", 10 * 1024 * 1024, 0, 0, NULL},
// used for temp allocs where we don't want to worry about freeing them
{"Temp Allocs", 128 * 1024 * 1024, 0, 0, NULL},
// spare slots
{"Unused", 1 * 1024 * 1024, 0, 0, NULL},
{"Unused", 1 * 1024 * 1024, 0, 0, NULL},
{"Unused", 1 * 1024 * 1024, 0, 0, NULL},
{"Unused", 1 * 1024 * 1024, 0, 0, NULL}
};
void *Virtual_PoolAlloc (int pool, int size)
{
if (pool < 0 || pool >= NUM_VIRTUAL_POOLS)
Sys_Error ("Virtual_PoolAlloc: Invalid Pool");
vpool_t *vp = &virtualpools[pool];
// if temp file loading overflows we just reset it
if (pool == VIRTUAL_POOL_TEMP && (vp->lowmark + size) >= vp->maxmem)
{
// if the temp file pool is too small to hold this allocation we reset it so that it's big enough
if (size > vp->maxmem)
Virtual_PoolReset (pool, size);
else Virtual_PoolFree (VIRTUAL_POOL_TEMP);
}
// not enough pool space
if ((vp->lowmark + size) >= vp->maxmem)
Sys_Error ("Virtual_PoolAlloc: Overflow");
// only pass over the commit region otherwise lots of small allocations will pass over
// the *entire* *buffer* every time (slooooowwwwww)
if ((vp->lowmark + size) >= vp->highmark)
{
// alloc in 256k batches
int newsize = MemoryRoundSize (vp->lowmark + size, 256);
if (!VirtualAlloc (vp->membase + vp->lowmark, newsize - vp->lowmark, MEM_COMMIT, PAGE_READWRITE))
Sys_Error ("Virtual_PoolAlloc: VirtualAlloc Failed");
vp->highmark = newsize;
}
// set up
void *buf = (vp->membase + vp->lowmark);
vp->lowmark += size;
return buf;
}
void Virtual_PoolReset (int pool, int newsizebytes)
{
// graceful failure
if (pool < 0 || pool >= NUM_VIRTUAL_POOLS) return;
vpool_t *vp = &virtualpools[pool];
// fully release the memory
if (vp->membase) VirtualFree (vp->membase, 0, MEM_RELEASE);
// fill in
vp->lowmark = 0;
vp->highmark = 0;
vp->maxmem = MemoryRoundSize (newsizebytes, 1024);
// reserve the memory for use by this pool
vp->membase = (byte *) VirtualAlloc (NULL, vp->maxmem, MEM_RESERVE, PAGE_NOACCESS);
}
void Virtual_PoolFree (int pool)
{
// graceful failure
if (pool < 0 || pool >= NUM_VIRTUAL_POOLS) return;
// already free
if (!virtualpools[pool].lowmark) return;
// decommit the allocated pool; if it straddles a page boundary the full extra page will be freed
VirtualFree (virtualpools[pool].membase, virtualpools[pool].lowmark, MEM_DECOMMIT);
// reset lowmark
virtualpools[pool].lowmark = 0;
virtualpools[pool].highmark = 0;
// reinit the cache if that was freed
if (pool == VIRTUAL_POOL_CACHE) Cache_Init ();
}
void Virtual_PoolInit (void)
{
for (int i = 0; i < NUM_VIRTUAL_POOLS; i++)
{
// skip over any pools that were already alloced
if (virtualpools[i].membase) continue;
// reserve the memory for use by this pool
virtualpools[i].membase = (byte *) VirtualAlloc (NULL, virtualpools[i].maxmem, MEM_RESERVE, PAGE_NOACCESS);
}
// init the cache
Cache_Init ();
}
/*
========================================================================================================================
ZONE MEMORY
The zone is used for small strings and other stuff that's dynamic in nature and would normally be handled by
malloc and free. It primarily exists so that we can report on zone usage, but also so that we can avoid using
malloc and free, as their behaviour is runtime dependent.
The win32 Heap* functions basically operate identically to the old zone functions except they let use reserve
virtual memory and also do all of the tracking and other heavy lifting for us.
========================================================================================================================
*/
typedef struct zblock_s
{
int size;
void *data;
} zblock_t;
int zonesize = 0;
HANDLE zoneheap = NULL;
void *Zone_Alloc (int size)
{
// create an initial heap for use with the zone
// this heap has 128K initially allocated and 32 MB reserved from the virtual address space
if (!zoneheap) zoneheap = HeapCreate (0, 0x20000, 0x2000000);
size += sizeof (zblock_t);
size = (size + 7) & ~7;
zblock_t *zb = (zblock_t *) HeapAlloc (zoneheap, HEAP_ZERO_MEMORY, size);
zb->size = size;
zb->data = (void *) (zb + 1);
zonesize += size;
return zb->data;
}
void Zone_Free (void *ptr)
{
// attempt to free a NULL pointer
if (!ptr) return;
if (!zoneheap) return;
// retrieve zone block pointer
zblock_t *zptr = ((zblock_t *) ptr) - 1;
zonesize -= zptr->size;
// release this back to the OS
HeapFree (zoneheap, 0, zptr);
HeapCompact (zoneheap, 0);
}
/*
========================================================================================================================
REPORTING
========================================================================================================================
*/
void Virtual_Report_f (void)
{
int reservedmem = 0;
int committedmem = 0;
Con_Printf ("\n-----------------------------------\n");
Con_Printf ("Pool Highmark Lowmark\n");
Con_Printf ("-----------------------------------\n");
for (int i = 0; i < NUM_VIRTUAL_POOLS; i++)
{
// don't report on empty pools
if (!virtualpools[i].highmark) continue;
Con_Printf
(
"%-11s %7.2f MB %7.2f MB\n",
virtualpools[i].name,
((float) virtualpools[i].highmark / 1024.0f) / 1024.0f,
((float) virtualpools[i].lowmark / 1024.0f) / 1024.0f
);
reservedmem += virtualpools[i].highmark;
committedmem += virtualpools[i].lowmark;
}
Con_Printf ("-----------------------------------\n");
Con_Printf
(
"%-11s %7.2f MB %7.2f MB\n",
"Total",
((float) reservedmem / 1024.0f) / 1024.0f,
((float) committedmem / 1024.0f) / 1024.0f
);
Con_Printf ("-----------------------------------\n");
Con_Printf ("%i objects in cache\n", numcacheobjects);
Con_Printf ("Zone size: %i KB\n", (zonesize + 1023) / 1023);
}
// also alloc the old heap_report command in case anyone used it
cmd_t virtual_report_cmd ("virtual_report", Virtual_Report_f);
cmd_t heap_report_cmd ("heap_report", Virtual_Report_f);
We had the power, we had the space, we had a sense of time and place
We knew the words, we knew the score, we knew what we were fighting for
We knew the words, we knew the score, we knew what we were fighting for
A few other things you can do is to keep the number of pools small and allocate to each in batches. I have a 256 K batch size hard-coded into here ("int newsize = MemoryRoundSize (vp->lowmark + size, 256);") but with a smaller number of memory pools that could be easily increased to 1 MB. What this will mean is that instead of committing memory on a regular basis it's only committed in discrete 1 MB chunks, meaning about 7 or 8 commits per map (as opposed to potentially several thousand if you just commit memory as required). A smaller number of pools means less batch size overhead; with 6 pools and a 256 K batch the most "wasted" memory you'll ever have is 1.25 MB.
My current incarnation uses only 6 pools - permanent, this game, cached models and sounds, this map, edicts and temp. Cached objects can be easily amalgamated with this game, but I want to keep it separate so that I can demand-flush it. You could cvar-ize the commit batch size if you wanted; I'm in 2 minds about this one, it's nice to expose it but should regular users be able to mess with something fairly low-level like that?
Precalculating sizes and allocating in bulk is always good too. For example, you can figure out in advance how many glpoly_t structs (and the sizes of their vertexes) you need for non warp-surfaces, so why not just allocate a single big buffer at the end of Mod_LoadFaces and write the polys into that instead of doing a separate allocation for each poly?
You'll notice that I have fixed hard-coded size limits on the pools here, but that's OK. warpc only needs 48 MB from the "this map" pool (or about 85 MB if you don't cache models and sounds) so a 128 MB upper limit is fine. You could increase it to 256 or even higher if you want; because it's only reserving an area of virtual memory for itself there's no actual resource utilization involved in doing so (just make sure that you don't reserve more than 2 GB total though!)
Anyway, I'm getting into areas more appropriate for the Engine Coding Tips thread now, so I think I'll leave it at that!
My current incarnation uses only 6 pools - permanent, this game, cached models and sounds, this map, edicts and temp. Cached objects can be easily amalgamated with this game, but I want to keep it separate so that I can demand-flush it. You could cvar-ize the commit batch size if you wanted; I'm in 2 minds about this one, it's nice to expose it but should regular users be able to mess with something fairly low-level like that?
Precalculating sizes and allocating in bulk is always good too. For example, you can figure out in advance how many glpoly_t structs (and the sizes of their vertexes) you need for non warp-surfaces, so why not just allocate a single big buffer at the end of Mod_LoadFaces and write the polys into that instead of doing a separate allocation for each poly?
You'll notice that I have fixed hard-coded size limits on the pools here, but that's OK. warpc only needs 48 MB from the "this map" pool (or about 85 MB if you don't cache models and sounds) so a 128 MB upper limit is fine. You could increase it to 256 or even higher if you want; because it's only reserving an area of virtual memory for itself there's no actual resource utilization involved in doing so (just make sure that you don't reserve more than 2 GB total though!)
Anyway, I'm getting into areas more appropriate for the Engine Coding Tips thread now, so I think I'll leave it at that!
We had the power, we had the space, we had a sense of time and place
We knew the words, we knew the score, we knew what we were fighting for
We knew the words, we knew the score, we knew what we were fighting for
Yr welcome.
This one should also be safe for making sure that leafs and nodes are allocated in consecutive memory (very important for node = (mnode_t *) &cl.worldmodel->leafs[i + 1]; to work right).
This one should also be safe for making sure that leafs and nodes are allocated in consecutive memory (very important for node = (mnode_t *) &cl.worldmodel->leafs[i + 1]; to work right).
We had the power, we had the space, we had a sense of time and place
We knew the words, we knew the score, we knew what we were fighting for
We knew the words, we knew the score, we knew what we were fighting for
Windows specific I'm afraid. I don't know what the virtual memory APIs for other OSs are, but anyone who does should be able to port it quite easy. Probably the biggest challenge would be dealing with Zone memory - Windows has so many memory APIs that I suspect there may not be an equivalent. You could just replace it with malloc and free and I suppose it would work (be careful though of my Ninja trickery in Zone_Free...)
As for road tested, it's pretty much the one I use in DirectQ so it's definitely been let out in the wild for a good while.
But I agree though, tackling the Quake memory manager is not a trivial task. Cache, Hunk Low, Hunk High, Zone - AAARRGGHHHH!!! Have you looked at the Q2 source code? That has another memory management system that should be easier to port and that also has quite good flexibility.
As for road tested, it's pretty much the one I use in DirectQ so it's definitely been let out in the wild for a good while.
But I agree though, tackling the Quake memory manager is not a trivial task. Cache, Hunk Low, Hunk High, Zone - AAARRGGHHHH!!! Have you looked at the Q2 source code? That has another memory management system that should be easier to port and that also has quite good flexibility.
We had the power, we had the space, we had a sense of time and place
We knew the words, we knew the score, we knew what we were fighting for
We knew the words, we knew the score, we knew what we were fighting for
ill see what i can do
i belive VirtualAlloc support was added later looking at things i see several different support models the original i think is mmap/munmap which btw can be emulated with (yes exactly that) VirtualAlloc/VirtualFree.
funny thing is i just wrote several wrappers for some programs i was porting to my mingw compiler suite and one of them actually emulates the above.
#if defined(__MINGW32__) && !defined(QUICK)
#include <windows.h>
#define QUICK 1
#define MAP_FAILED 0
#define MREMAP_FIXED 2 // the value in linux, though it doesn't really matter
// These, when combined with the mmap invariants below, yield the proper action
#define PROT_READ PAGE_READWRITE
#define PROT_WRITE PAGE_READWRITE
#define MAP_ANONYMOUS MEM_RESERVE
#define MAP_PRIVATE MEM_COMMIT
#define MAP_SHARED MEM_RESERVE // value of this #define is 100% arbitrary
// VirtualAlloc is only a replacement for mmap when certain invariants are kept
#define mmap(start, length, prot, flags, fd, offset) \
( (start) == NULL && (fd) == -1 && (offset) == 0 && \
(prot) == (PROT_READ|PROT_WRITE) && (flags) == (MAP_PRIVATE|MAP_ANONYMOUS) \
? VirtualAlloc(0, length, MEM_RESERVE | MEM_COMMIT, PAGE_READWRITE) \
: NULL )
#define munmap(start, length) (VirtualFree(start, 0, MEM_RELEASE) ? 0 : -1)
#endif // HAVE_MMAP
hmm time to test out windows 7 virtual machines with my suse distro
i belive VirtualAlloc support was added later looking at things i see several different support models the original i think is mmap/munmap which btw can be emulated with (yes exactly that) VirtualAlloc/VirtualFree.
funny thing is i just wrote several wrappers for some programs i was porting to my mingw compiler suite and one of them actually emulates the above.
#if defined(__MINGW32__) && !defined(QUICK)
#include <windows.h>
#define QUICK 1
#define MAP_FAILED 0
#define MREMAP_FIXED 2 // the value in linux, though it doesn't really matter
// These, when combined with the mmap invariants below, yield the proper action
#define PROT_READ PAGE_READWRITE
#define PROT_WRITE PAGE_READWRITE
#define MAP_ANONYMOUS MEM_RESERVE
#define MAP_PRIVATE MEM_COMMIT
#define MAP_SHARED MEM_RESERVE // value of this #define is 100% arbitrary
// VirtualAlloc is only a replacement for mmap when certain invariants are kept
#define mmap(start, length, prot, flags, fd, offset) \
( (start) == NULL && (fd) == -1 && (offset) == 0 && \
(prot) == (PROT_READ|PROT_WRITE) && (flags) == (MAP_PRIVATE|MAP_ANONYMOUS) \
? VirtualAlloc(0, length, MEM_RESERVE | MEM_COMMIT, PAGE_READWRITE) \
: NULL )
#define munmap(start, length) (VirtualFree(start, 0, MEM_RELEASE) ? 0 : -1)
#endif // HAVE_MMAP
hmm time to test out windows 7 virtual machines with my suse distro