[FTE]Load / refresh level performance question

Discuss programming topics for the various GPL'd game engine sources.
Post Reply
toneddu2000
Posts: 1395
Joined: Tue Feb 24, 2009 4:39 pm
Location: Italy

[FTE]Load / refresh level performance question

Post by toneddu2000 »

Hi insideqc engine gurus! I'm here with a question and not an issue, this time. It's been a while (1 year I guess) since I don't use radiant + q3map2 anymore (yeeah!), infact every level I do it's made directly in FTE and I'm pretty happy with it. Now, since I'm creating a game with several levels (like every other game in the planet, I guess), I'd like to know if this could lead to performance problem.

I explain myself: when engine starts, my only-csqc game loads an empty bsp map, with just 6 black walls, a light and an info_player_start entity. Anyone of these elements is mandatory: a closing hull structure is needed otherwise FTE realtime lights won't work (you can't just create a plane and uncheck "Stop on leak" flag in NetRadiant: it will compile but it won't see rtlights in FTE anyway), a light with targetname(even with radius set to 1 it's ok) is needed to stop FTE to print the "No lights" warning at startup and info_player_start is needed because otherwise NetRadiant won't compile.

So fa so good. Now I created an entity which represents the level and a float that represents level current. When CSQC starts hull.bsp, it spawns level entity and setmodel first "map object" model onto it. When "Load new level" command is triggered somehow, I simply change new map object model to level entity, reset flags that needs to be restarted and that's it.

This leads to the question: This continous swap level models (with relative possibly huge textures attached) without any restarting (for restarting I mean the canonical way to issue the "map foo" command) could lead to performance issue? The continuous loading new models and textures replaces old ones or it keeps filling memory space?

Should I use the map approach or the engine will do the memory free by yourself when that model is no longer tied to any entity?

Thanks in advance for any suggestion, I'm pretty sure this conversation could lead to interesting solutions/opinions and different point of views! :biggrin:
Meadow Fun!! - my first commercial game, made with FTEQW game engine
Spike
Posts: 2914
Joined: Fri Nov 05, 2004 3:12 am
Location: UK
Contact:

Re: [FTE]Load / refresh level performance question

Post by Spike »

the renderer and rtlights code ideally wants pvs. this requires a model on the world entity..

rtlights should be set us as static world lights in order for their shadowmesh cache to work. recreating them every frame is NOT a good solution - which is what will happen, as only the worldmodel's surfaces can be cached in the shadowmesh. Large submodels are NOT good for performance as a result - the shadowmesh will instead need to be regenerated every frame.

a pure csqc mod can use the disconnect command occasionally to purge the qc state (and transient data like lightmap textures and unneeded models), but if you want to purge everything then an occasional vid_reload will unload all the models unused or otherwise. they can still be poked, but do note that this can result in stalls as vid_reload is a little lazy. altrnatively try the flush command.
toneddu2000
Posts: 1395
Joined: Tue Feb 24, 2009 4:39 pm
Location: Italy

Re: [FTE]Load / refresh level performance question

Post by toneddu2000 »

Wow thanks Spike for the super-quick reply!
Spike wrote:the renderer and rtlights code ideally wants pvs. this requires a model on the world entity..
I've no problem about it, but is it ok to do setmodel(world,"mymapmodel.iqm");? Is it safe or even possible? Isn't world entity a read-only entity? Or, by "this requires a model on the world entity" do you mean a model embedded in bsp map with misc_model(via netradiant or whatever)?
Spike wrote:rtlights should be set us as static world lights in order for their shadowmesh cache to work. recreating them every frame is NOT a good solution - which is what will happen, as only the worldmodel's surfaces can be cached in the shadowmesh.
So, even using .rtlights file in maps folder instead of dynamiclightadd() is not ok for performance? I'd really like to use rtlights just once and not visualize them every frame in CSQC_UpdateView() but I don't know if it's doable.
Spike wrote:Large submodels are NOT good for performance as a result - the shadowmesh will instead need to be regenerated every frame.
Instead, using several groups of small submodels each, would it be ok?
Spike wrote: a pure csqc mod can use the disconnect command occasionally to purge the qc state (and transient data like lightmap textures and unneeded models), but if you want to purge everything then an occasional vid_reload will unload all the models unused or otherwise. they can still be poked, but do note that this can result in stalls as vid_reload is a little lazy. altrnatively try the flush command.
Well, I'll try both disconnect and flush and I let you know, thanks a lot. What's the difference between vid_reload and vid_restart, by the way?
I know that in ssqc there's a way to save parms between a map and another but I don't remember nor the name neither where to use them.

Thanks again Spike!
Meadow Fun!! - my first commercial game, made with FTEQW game engine
frag.machine
Posts: 2126
Joined: Sat Nov 25, 2006 1:49 pm

Re: [FTE]Load / refresh level performance question

Post by frag.machine »

You may avoid some of the problems a huge submodel creates by breaking it in smaller, modular sections (this also opens the possibility of artwork reuse). But the engine will always ressents the lack of PVS data that allows to prevent unseen stuff to be rendered. I believe FTE (like every other Quake engine under the sun) assumes submodels always change position or angle and thus light info must be recreated every frame, killing the performance. The bigger the submodel, the greater the hit.

EDIT: in SSQC you can use the parm1 to parm15 cvars to store info between maps.
I know FrikaC made a cgi-bin version of the quakec interpreter once and wrote part of his website in QuakeC :) (LordHavoc)
Spike
Posts: 2914
Joined: Fri Nov 05, 2004 3:12 am
Location: UK
Contact:

Re: [FTE]Load / refresh level performance question

Post by Spike »

the worldmodel must be .bsp (true pvs) or .map/terrain (distance culling). don't bother trying iqm.

dynamiclightadd allocates a dynamic light (or reuses an existing one).
static lights start at 32, which is messy. and you'd need to scan for new ones or something. which is ugly. and override ALL settings, which sucks too. you might also be able to get away with makestatic. not sure if I remembered to code that for static lights in csqc, but its a bit more limited.

lots of small submodels suck too. there's a tradeoff here. try to stick to 1ish texture per submodel, if possible.

vid_restart tears down everything, including the entire rendering context and window.
vid_reload destroys models+textures+shaders, but doesn't obliterate the rendering context itself. this avoids video mode switches or the desktop appearing, or other programs getting input focus, etc. its much less jarring.
flush flushes the models+sounds, and reloads them when they're next poked. doesn't destroy anything that would require any internal pointers to change - sadly this includes textures.

iirc, my purecsqc mod was lame and just used a cvar to carry info from one map to the next. resetting the entire game was also a simpler way to flush all the ents and static ents/sounds. that said I did initially have a version that just directly swapped the map over (as well as precaching it during intermission for pretty much instant map changes).
toneddu2000
Posts: 1395
Joined: Tue Feb 24, 2009 4:39 pm
Location: Italy

Re: [FTE]Load / refresh level performance question

Post by toneddu2000 »

Spike wrote:the worldmodel must be .bsp (true pvs) or .map/terrain (distance culling). don't bother trying iqm.
Ok
frag.machine wrote:You may avoid some of the problems a huge submodel creates by breaking it in smaller, modular sections (this also opens the possibility of artwork reuse). But the engine will always ressents the lack of PVS data that allows to prevent unseen stuff to be rendered. I believe FTE (like every other Quake engine under the sun) assumes submodels always change position or angle and thus light info must be recreated every frame, killing the performance. The bigger the submodel, the greater the hit.
Well, since I modeled levels as single piece(not great choice, I must say,but, whatever), I guess I'll stick to one piece-one texture advice
Spike wrote:vid_restart tears down everything, including the entire rendering context and window.
vid_reload destroys models+textures+shaders, but doesn't obliterate the rendering context itself. this avoids video mode switches or the desktop appearing, or other programs getting input focus, etc. its much less jarring.
flush flushes the models+sounds, and reloads them when they're next poked. doesn't destroy anything that would require any internal pointers to change - sadly this includes textures.
Wow, very exhaustive explanation, thanks Spike. I guess I'll use vid_reload. Seems to me the best choice
Spike wrote:iirc, my purecsqc mod was lame and just used a cvar to carry info from one map to the next. resetting the entire game was also a simpler way to flush all the ents and static ents/sounds. that said I did initially have a version that just directly swapped the map over (as well as precaching it during intermission for pretty much instant map changes).
frag.machine wrote:EDIT: in SSQC you can use the parm1 to parm15 cvars to store info between maps.
That's EXACTLY what I was looking for! Thanks a lot man! In fact in CSQC Spike used it in SetNewParms() in CSQC_Init() and DecodeLevelParms() in CSQC_WorldLoaded() using a string and infoadd() and infoget(), which it's just what I needed to store stupid things between map changes! :)

I understood everything except this statement:
Spike wrote: dynamiclightadd allocates a dynamic light (or reuses an existing one).
static lights start at 32, which is messy. and you'd need to scan for new ones or something. which is ugly. and override ALL settings, which sucks too. you might also be able to get away with makestatic. not sure if I remembered to code that for static lights in csqc, but its a bit more limited.
You told me yesterday that I can store dynamiclight in a float like float foo = dynamiclightadd(...), but, how can I make lights static? They are not tied to any entity. I cannot do entity e = dynamiclightadd(), so how can I use makestatic with it? Plus, I don't understand what do you mean "static lights start at 32"? 32 what? :) I don't know if it could be useful for your reply, but I could be ok to use just one light per level, if it's simpler

Thanks both for your great advices!
Meadow Fun!! - my first commercial game, made with FTEQW game engine
frag.machine
Posts: 2126
Joined: Sat Nov 25, 2006 1:49 pm

Re: [FTE]Load / refresh level performance question

Post by frag.machine »

I think he means FTE starts with a 32 positions array to store static lights, and this can be expanded later if required, but probably with some performance penalty.

EDIT: regarding cvars you can use between levels, there is also scratch1 to scratch4, and if I am not mistaken FTE allows you to create your own cvars via QC (at least SSQC), although I never used it.
I know FrikaC made a cgi-bin version of the quakec interpreter once and wrote part of his website in QuakeC :) (LordHavoc)
toneddu2000
Posts: 1395
Joined: Tue Feb 24, 2009 4:39 pm
Location: Italy

Re: [FTE]Load / refresh level performance question

Post by toneddu2000 »

frag.machine wrote:I think he means FTE starts with a 32 positions array to store static lights, and this can be expanded later if required, but probably with some performance penalty.
Thanks frag.machine, but still not clear to me, since I don't have much experience in engine programming. Does this position could be accessed via qc?
frag.machine wrote:EDIT: regarding cvars you can use between levels, there is also scratch1 to scratch4, and if I am not mistaken FTE allows you to create your own cvars via QC (at least SSQC), although I never used it.
wow, thanks! I'm starting thinking that parm1,parm2,..etc were just floats set in qc, I always thought that they were like engine-related.. I guess that infoadd/infoget do the real magic
Meadow Fun!! - my first commercial game, made with FTEQW game engine
frag.machine
Posts: 2126
Joined: Sat Nov 25, 2006 1:49 pm

Re: [FTE]Load / refresh level performance question

Post by frag.machine »

toneddu2000 wrote:
frag.machine wrote:I think he means FTE starts with a 32 positions array to store static lights, and this can be expanded later if required, but probably with some performance penalty.
Thanks frag.machine, but still not clear to me, since I don't have much experience in engine programming. Does this position could be accessed via qc?
AFAIK, no. They are handled engine side.
I know FrikaC made a cgi-bin version of the quakec interpreter once and wrote part of his website in QuakeC :) (LordHavoc)
toneddu2000
Posts: 1395
Joined: Tue Feb 24, 2009 4:39 pm
Location: Italy

Re: [FTE]Load / refresh level performance question

Post by toneddu2000 »

ok, thanks anyway frag.machine
Meadow Fun!! - my first commercial game, made with FTEQW game engine
Spike
Posts: 2914
Joined: Fri Nov 05, 2004 3:12 am
Location: UK
Contact:

Re: [FTE]Load / refresh level performance question

Post by Spike »

If I got off my arse and made a proper builtin for this stuff, it'd be equivalent to the following:

Code: Select all

float(vector org, float radius) staticlight_spawn =
{
    for (float l = 32; ; l++) //evil magic number
    {
        if (!(float)dynamiclight_get(l, LFIELD_RADIUS))
        {  //okay, this light seems to be dead, so use this one
            dynamiclight_set(l, LFIELD_RADIUS, radius);
            dynamiclight_set(l, LFIELD_ORIGIN, org);
            dynamiclight_set(l, LFIELD_COLOUR, [1.0, 1.0, 1.0]); //white light.
            dynamiclight_set(l, LFIELD_FLAGS, LFLAG_REALTIMEMODE);
            dynamiclight_set(l, LFIELD_STYLE, -1); //or 0, if you want it to respond to style 0
            dynamiclight_set(l, LFIELD_ANGLES, '0 0 0'); //vectoangles or something. beware inverse pitch.
            dynamiclight_set(l, LFIELD_ROTATION, '0 0 0'); //avelocity
            dynamiclight_set(l, LFIELD_FOV, 0); //0 for omni light, otherwise spotlight
            dynamiclight_set(l, LFIELD_CORONA, 1.0); //you probably want 0 here.
            dynamiclight_set(l, LFIELD_CORONASCALE, 0.25);
            dynamiclight_set(l, LFIELD_CUBEMAPNAME, ""); //for proper paths
            dynamiclight_set(l, LFIELD_AMBIENTSCALE, 0);
            dynamiclight_set(l, LFIELD_DIFFUSESCALE, 1);
            dynamiclight_set(l, LFIELD_SPECULARSCALE, 1);

            dynamiclight_set(l, LFIELD_DIETIME, 0); //don't let it die.
            dynamiclight_set(l, LFIELD_RGBDECAY, '0 0 0'); //don't let it die.
            dynamiclight_set(l, LFIELD_RADIUSDECAY, 0); //don't let it die.

            return l; //so the caller can override stuff.
        }
    }
};
like I said - the static lights start at index 32. dynamic lights have indexes beneath that (required by LFLAG_LIGHTMAP, and thus limited to only those slots).
yes, I should make a proper builtin.
try to avoid gaps too (so avoid hardcoding large indexes), but don't worry about renumbering to do so.

note that by changing the dietime and the decay values, you can spawn semi-static dlights (like the shrinking light spawned from explosions) without needing to remember to update them every single frame.
lights with a negative light value are not guaranteed to work.
cubemaps are cool. use them where feasible.
toneddu2000
Posts: 1395
Joined: Tue Feb 24, 2009 4:39 pm
Location: Italy

Re: [FTE]Load / refresh level performance question

Post by toneddu2000 »

So ****** awesome! :biggrin: HUGE thanks Spike!
I wasn't even aware of all those LFIELD_

Funny thing is that I don't see any difference on performance with just one light. Probably it takes 2 or more lights to see performance diff. And, I must say, I used LOTS of shaders in this project, so, probably it's better to start with a fresh new project for better benchmarks

I tried to change LFIELD_FLAGS from LFLAG_REALTIMEMODE to LFLAG_NORMALMODE and LFLAG_LIGHTMAP but no effect, I guess lightmap is only in "get" mode and not set.

Anyway super cool function Spike, thanks a lot!
Meadow Fun!! - my first commercial game, made with FTEQW game engine
toneddu2000
Posts: 1395
Joined: Tue Feb 24, 2009 4:39 pm
Location: Italy

Re: [FTE]Load / refresh level performance question

Post by toneddu2000 »

frag.machine wrote:
toneddu2000 wrote:
frag.machine wrote:I think he means FTE starts with a 32 positions array to store static lights, and this can be expanded later if required, but probably with some performance penalty.
Thanks frag.machine, but still not clear to me, since I don't have much experience in engine programming. Does this position could be accessed via qc?
AFAIK, no. They are handled engine side.
Well, that's a puzzle, because they're set in pr_cmds.c file, in knowndefs[] array,

Code: Select all

{"parm1, parm2, parm3, parm4, parm5, parm6, parm7, parm8, parm9, parm10, parm11, parm12, parm13, parm14, parm15, parm16", "float", QW|NQ},
But it "seems" that they're not used anywhere so it probably would be the same to use foo1,foo2,etc. I'll try to load same .bsp map again and again passing these vars every time with infoadd()/infoget()and see if game will store/retrieve them.
I'll let you know!
Meadow Fun!! - my first commercial game, made with FTEQW game engine
frag.machine
Posts: 2126
Joined: Sat Nov 25, 2006 1:49 pm

Re: [FTE]Load / refresh level performance question

Post by frag.machine »

parm1...parm9 are used in single player to pass player info across levels (health, ammo, armor, weapons, etc). Check client.qc and you'll see in DecodeLevelParms ().
I know FrikaC made a cgi-bin version of the quakec interpreter once and wrote part of his website in QuakeC :) (LordHavoc)
Post Reply