Improving "ProQuake" Demo Record At Any Time

Post tutorials on how to do certain tasks within game or engine code here.
Baker
Posts: 3666
Joined: Tue Mar 14, 2006 5:15 am

Improving "ProQuake" Demo Record At Any Time

Post by Baker »

ProQuake since at least 2002 has had the ability to record a demo at any time; this feature has filtered through to different engines like JoeQuake --> Qrack and DirectQ.

In 2006 or 2007, Jozsef the author of JoeQuake caught something of critical interest to single player!!!!!!!!!!! The ProQuake "record" command does NOT save all the single player stats and it should.

To illustrate the problem: Load up engine, go to e1m1, kill 3 or 4 monsters, start recording demo, wait a few seconds, stop recording. Play your demo and press TAB to see monsters killed: It is ZERO!!! But you had killed 3 or 4 monsters!

Suspected historical origin of problem: Since WinQuake/GLQuake never allowed recording of a demo except before map load, it wasn't necessary to save the monsters killed or secrets because they would be zero.

In cl_demo.c in CL_Record_f ... the yellow code is a suggested addition which will save monster kill stats and secrets to the demo.
byte *data = net_message.data;
int i, cursize = net_message.cursize;

for (i=0 ; i<2 ; i++)
{
net_message.data = demo_head;
net_message.cursize = demo_head_size;
CL_WriteDemoMessage ();
}

net_message.data = demo_head[2];
SZ_Clear (&net_message);

// current names, colors, and frag counts
for (i=0 ; i<cl.maxclients ; i++)
{
MSG_WriteByte (&net_message, svc_updatename);
MSG_WriteByte (&net_message, i);
MSG_WriteString (&net_message, cl.scores.name);
MSG_WriteByte (&net_message, svc_updatefrags);
MSG_WriteByte (&net_message, i);
MSG_WriteShort (&net_message, cl.scores.frags);
MSG_WriteByte (&net_message, svc_updatecolors);
MSG_WriteByte (&net_message, i);
MSG_WriteByte (&net_message, cl.scores.colors);
}

// send all current light styles
for (i=0 ; i<MAX_LIGHTSTYLES ; i++)
{
MSG_WriteByte (&net_message, svc_lightstyle);
MSG_WriteByte (&net_message, i);
MSG_WriteString (&net_message, cl_lightstyle.map);
}

// send stats - OMG from JoeQuake 0.15 ... Baker: What about the CD track or SVC fog or SVC skybox ... future consideration.
MSG_WriteByte (&net_message, svc_updatestat);
MSG_WriteByte (&net_message, STAT_TOTALSECRETS);
MSG_WriteLong (&net_message, pr_global_struct->total_secrets);

MSG_WriteByte (&net_message, svc_updatestat);
MSG_WriteByte (&net_message, STAT_TOTALMONSTERS);
MSG_WriteLong (&net_message, pr_global_struct->total_monsters);

MSG_WriteByte (&net_message, svc_updatestat);
MSG_WriteByte (&net_message, STAT_SECRETS);
MSG_WriteLong (&net_message, pr_global_struct->found_secrets);

MSG_WriteByte (&net_message, svc_updatestat);
MSG_WriteByte (&net_message, STAT_MONSTERS);
MSG_WriteLong (&net_message, pr_global_struct->killed_monsters);
// This entire yellow block is missing in ProQuake, Qrack, DirectQ

// view entity
MSG_WriteByte (&net_message, svc_setview);
MSG_WriteShort (&net_message, cl.player_point_of_view_entity);

// signon
MSG_WriteByte (&net_message, svc_signonnum);
MSG_WriteByte (&net_message, 3);

CL_WriteDemoMessage ();

// restore net_message
net_message.data = data;
net_message.cursize = cur size;
The night is young. How else can I annoy the world before sunsrise? 8) Inquisitive minds want to know ! And if they don't -- well like that ever has stopped me before ..
qbism
Posts: 1236
Joined: Thu Nov 04, 2004 5:51 am
Contact:

Re: Improving "ProQuake" Demo Record At Any Time

Post by qbism »

I didn't realize this was possible in Proquake. Proquake vs. Vanilla quake demo record- would it be easy to see the changes with a diff of cl_demo.c, or is it more elaborate than that?
Baker
Posts: 3666
Joined: Tue Mar 14, 2006 5:15 am

Re: Improving "ProQuake" Demo Record At Any Time

Post by Baker »

It is pretty straightforward to implement, I'd diff ProQuake 3.50 ( source link ftp://ftp.runequake.com/quake/engines/p ... /3.50/src/ ) versus GLQuake.
The night is young. How else can I annoy the world before sunsrise? 8) Inquisitive minds want to know ! And if they don't -- well like that ever has stopped me before ..
qbism
Posts: 1236
Joined: Thu Nov 04, 2004 5:51 am
Contact:

Re: Improving "ProQuake" Demo Record At Any Time

Post by qbism »

Just got around to trying this, works almost out-of-the-box diffing with Proquake 3.50 cl_demo.c as Baker mentioned.

Now thinking about saving particle info as well. Will have to reverse-engineer color based on particle type (if possible) so it can be saved correctly. In svc_particle message the color number is encoded w/ particle type. Even if successful, svc_particle doesn't send remaining time information (or alpha for some engines). The temptation is to create a new svc_ command that sends an absolute particle snapshot.
r00k
Posts: 1111
Joined: Sat Nov 13, 2004 10:39 pm

Re: Improving "ProQuake" Demo Record At Any Time

Post by r00k »

qBISM UR ICON STILL SCARES ME.;)

Edit: Hmm, i went to look and Qrack has had this since v1.5 maybe sooner, though i do remember talking about this in the past...
Last edited by r00k on Mon Jul 16, 2012 4:58 pm, edited 2 times in total.
mh
Posts: 2292
Joined: Sat Jan 12, 2008 1:38 am

Re: Improving "ProQuake" Demo Record At Any Time

Post by mh »

I smell some copy/paste from Host_Spawn_f...

I'd suggest the following instead:

Code: Select all

		CL_UpdateDemoStat (STAT_TOTALSECRETS);
		CL_UpdateDemoStat (STAT_TOTALMONSTERS);
		CL_UpdateDemoStat (STAT_SECRETS);
		CL_UpdateDemoStat (STAT_MONSTERS);
And:

Code: Select all

void CL_UpdateDemoStat (int stat)
{
	if (stat < 0 || stat >= MAX_CL_STATS) return;

	MSG_WriteByte (&net_message, svc_updatestat);
	MSG_WriteByte (&net_message, stat);
	MSG_WriteLong (&net_message, cl.stats[stat]);
}
Reason why is because the PQ original relies on pr_global_struct stuff, but demo recording is initiated on the client so (if e.g recording a co-op demo) these values might not exist (or might be utter garbage) on the client that is doing the recording. Even in a pure single-player "you have the data locally so you can cheat a little" scenario, the QC may be sending different values to the client than those it stores internally. The mod author may wish to hide the totals, for example, so it's more correct to use the stats as sent to the client.

Everything else in the "initialize the demo file if we're already connected" block uses info from the client, so it seems fairly weird to be crossing the client/server boundary here, especially for data that is also available on the client.

I also hate code duplication... :twisted:

Otherwise what I like about this one is that it just uses standard protocol messages rather than a custom "something else", which means it will work for playback on any engine.
We had the power, we had the space, we had a sense of time and place
We knew the words, we knew the score, we knew what we were fighting for
qbism
Posts: 1236
Joined: Thu Nov 04, 2004 5:51 am
Contact:

Re: Improving "ProQuake" Demo Record At Any Time

Post by qbism »

r00k wrote:qBISM UR ICON STILL SCARES ME.;)
It's a warning, stay outta my swamp! :D Actually a screenshot, DP w/ retexture and high contrast. Now, this is scary-

First attempt at saving a snapshot (EDIT: that is, a snapshot within demo recording) of all particles in their current state. Works perfectly except that it doesn't work. All the particles are saved and appear to be parsed correctly during playback, but after rebuilding the particles they just don't show up. One possibility is an error in 'die' time parsing that snuffs particles immediately. Another is some aspect of demo message composure and/or playback that I'm not understanding.

I created 'svc_particledemo' and added this function to cl_demo.c

Code: Select all

void CL_UpdateDemoParticle (particle_t *p) //qbism
{
    int i;
    MSG_WriteByte (&net_message, svc_particledemo);
    for (i=0 ; i<3 ; i++)
        MSG_WriteCoord (&net_message, p->org[i]);
    for (i=0 ; i<3 ; i++)
        MSG_WriteCoord (&net_message, p->vel[i]);
    MSG_WriteFloat(&net_message, p->color);
    MSG_WriteByte(&net_message, (byte)p->type);
    MSG_WriteFloat(&net_message, p->ramp);
    MSG_WriteFloat(&net_message, p->die - cl.time);  //qbism - add back cl.time when parsing
    MSG_WriteFloat(&net_message, cl.time - p->start_time);
    MSG_WriteFloat(&net_message, p->alpha);
    MSG_WriteFloat(&net_message, p->alphavel);
}
CL_UpdateDemoParticle is called from CL_Record_f via a loop that runs through all the existing particles. The whole function is pasted here, along with ProQuake 'record any time' modifications. Sorry for any resulting scrolling fatigue but the context is needed. The interesting part starts about 3/4 down, at 'for (p=active_particles ; p ; p=p->next)'

Code: Select all

void CL_Record_f (void)
{
    int		c;
    char	name[MAX_OSPATH];
    int		track;
    particle_t		*p, *pnext; //qbism - send particles

    if (cmd_source != src_command)
        return;

    c = Cmd_Argc();
    if (c != 2 && c != 3 && c != 4)
    {
        Con_Printf ("record <demoname> [<map> [cd track]]\n");
        return;
    }

    if (strstr(Cmd_Argv(1), ".."))
    {
        Con_Printf ("Relative pathnames are not allowed.\n");
        return;
    }

    // JPG 3.00 - consecutive demo bug
    if (cls.demorecording)
        CL_Stop_f();

    /* JPG 1.05 - got rid of this because recording after connecting is now supported
    if (c == 2 && cls.state == ca_connected)
    {
        Con_Printf("Can not record - already connected to server\nClient demo recording must be started before connecting\n");
        return;
    }
    */

    // JPG 1.05 - replaced it with this
    if (c == 2 && cls.state == ca_connected && cls.signon < 2)
    {
        Con_Printf("Can't record - try again when connected\n");
        return;
    }

// write the forced cd track number, or -1
    if (c == 4)
    {
        track = atoi(Cmd_Argv(3));
        Con_Printf ("Forcing CD track to %i\n", cls.forcetrack);
    }
    else
        track = -1;

    sprintf (name, "%s/%s", com_gamedir, Cmd_Argv(1));

//
// start the map up
//
    if (c > 2)
        Cmd_ExecuteString ( va("map %s", Cmd_Argv(2)), src_command);

//
// open the demo file
//
    COM_DefaultExtension (name, ".dem");

    Con_Printf ("recording to %s.\n", name);
    cls.demofile = fopen (name, "wb");
    if (!cls.demofile)
    {
        Con_Printf ("ERROR: couldn't open.\n");
        return;
    }

    cls.forcetrack = track;
    fprintf (cls.demofile, "%i\n", cls.forcetrack);

    cls.demorecording = true;

    // JPG 1.05 - initialize the demo file if we're already connected
    if (c == 2 && cls.state == ca_connected)
    {
        byte *data = net_message.data;
        int cursize = net_message.cursize;
        int i;

        for (i = 0 ; i < 2 ; i++)
        {
            net_message.data = demo_head[i];
            net_message.cursize = demo_head_size[i];
            CL_WriteDemoMessage();
        }

        net_message.data = demo_head[2];
        SZ_Clear (&net_message);

        // current names, colors, and frag counts
        for (i=0 ; i < cl.maxclients ; i++)
        {
            MSG_WriteByte (&net_message, svc_updatename);
            MSG_WriteByte (&net_message, i);
            MSG_WriteString (&net_message, cl.scores[i].name);
            MSG_WriteByte (&net_message, svc_updatefrags);
            MSG_WriteByte (&net_message, i);
            MSG_WriteShort (&net_message, cl.scores[i].frags);
            MSG_WriteByte (&net_message, svc_updatecolors);
            MSG_WriteByte (&net_message, i);
            MSG_WriteByte (&net_message, cl.scores[i].colors);
        }

        // send all current light styles
        for (i = 0 ; i < MAX_LIGHTSTYLES ; i++)
        {
            MSG_WriteByte (&net_message, svc_lightstyle);
            MSG_WriteByte (&net_message, i);
            MSG_WriteString (&net_message, cl_lightstyle[i].map);
        }

        CL_UpdateDemoStat (STAT_TOTALSECRETS);
        CL_UpdateDemoStat (STAT_TOTALMONSTERS);
        CL_UpdateDemoStat (STAT_SECRETS);
        CL_UpdateDemoStat (STAT_MONSTERS);

        for (p=active_particles ; p ; p=p->next) //qbism - send particle snapshot
        {
            pnext = p->next;
            if (pnext)
            {
                p->next = pnext->next;
                pnext->next = free_particles;
                free_particles = pnext;
            }
            CL_UpdateDemoParticle(p);
            //qbism - break into several messages if many particles exist
            if ((net_message.cursize + sizeof(particle_t)+2000) > net_message.maxsize)
            {
                CL_WriteDemoMessage();
                net_message.data = demo_head[2];
                SZ_Clear (&net_message);
            }
        }

        // view entity
        MSG_WriteByte (&net_message, svc_setview);
        MSG_WriteShort (&net_message, cl.viewentity);

        // signon
        MSG_WriteByte (&net_message, svc_signonnum);
        MSG_WriteByte (&net_message, 3);

        CL_WriteDemoMessage();

        // restore net_message
        net_message.data = data;
        net_message.cursize = cursize;
    }
}

In CL_ParseServerMessage, added

Code: Select all

        case svc_particledemo: //qbism - snapshot for demos
            R_ParseParticleDemo ();
            break;
Finally, R_ParseParticleDemo added to r_part.c

Code: Select all

/*
===============
R_ParseParticleDemo

qbism - Parse an individual particle snapshot, made for 'any time' demo recording
===============
*/
void R_ParseParticleDemo (void)
{
    int i;
    particle_t	*p;

    if (!free_particles)
        Sys_Error("Demo error: free_particles = NULL");

    p = free_particles;
    free_particles = p->next;
    p->next = active_particles;
    active_particles = p;
    for (i=0 ; i<3 ; i++)
        p->org[i] = MSG_ReadCoord ();
    for (i=0 ; i<3 ; i++)
        p->vel[i] = MSG_ReadCoord ();

    p->color = MSG_ReadFloat();
    p->type = (int)MSG_ReadByte();
    p->ramp = MSG_ReadFloat();
    p->die = MSG_ReadFloat() + cl.time;  //qbism - add back cl.time when parsing
    p->start_time = cl.time - MSG_ReadFloat();
    p->alpha = MSG_ReadFloat();
    p->alphavel = MSG_ReadFloat();
}
Besides visual continuity for players, this would allow more concise demo slices for piecing together videos.
Last edited by qbism on Mon Jul 16, 2012 2:54 am, edited 2 times in total.
Baker
Posts: 3666
Joined: Tue Mar 14, 2006 5:15 am

Re: Improving "ProQuake" Demo Record At Any Time

Post by Baker »

I hate to be a killjoy -- and please do not let this thought stop your experimentation because unconventional thought benefits us all ----

But savegames are supposed to be from the perspective of the server. The server doesn't have particles or temp entities or R_(anything) since R_(anything) is "render" (which isn't something the server ever does).

[Again ... at the same time, wouldn't it be nice if savegames preserved the entire world state, so kudos to your ideas .... I have some terribly unconventional ideas in this department myself ... not the particles department, but things like the server thinking about lighting and knowing lighting levels without the Quake2 hackery where the client sends it back to the server ... solar powered armor, monsters that ignore pitch black players, players entirely in shadows not getting sent to the client, etc. ]
The night is young. How else can I annoy the world before sunsrise? 8) Inquisitive minds want to know ! And if they don't -- well like that ever has stopped me before ..
qbism
Posts: 1236
Joined: Thu Nov 04, 2004 5:51 am
Contact:

Re: Improving "ProQuake" Demo Record At Any Time

Post by qbism »

This is still only for demo recording at this point, rather than savegames. Yeah, savegames would involve another insane layer including clients sending their individual particle states up to the server.
Spike
Posts: 2914
Joined: Fri Nov 05, 2004 3:12 am
Location: UK
Contact:

Re: Improving "ProQuake" Demo Record At Any Time

Post by Spike »

its really not worth it.
1: you loose demo compatibility. different engines can have wildly different particle systems, so don't expect any sane workarounds for that.
2: doesn't help multiplayer at all. different clients still get different particle patterns.
3: 1/8th qu precision probably isn't enough for tight bunches of particles.
4: console tends to cover up everything when starting playback, so you miss the first halfsecond anyway.

if you're trying to capture a good bit of action, it would be best to write the data to memory even when not actually dumping to disk, so when you actually start recording you can rewind a bit and actually record the action that the user wanted to record, instead of only the parts after it.
you can keep whatever particle log active for a little longer.
qbism
Posts: 1236
Joined: Thu Nov 04, 2004 5:51 am
Contact:

Re: Improving "ProQuake" Demo Record At Any Time

Post by qbism »

The
Spike wrote:if you're trying to capture a good bit of action, it would be best to write the data to memory even when not actually dumping to disk, so when you actually start recording you can rewind a bit and actually record the action that the user wanted to record, instead of only the parts after it.
you can keep whatever particle log active for a little longer.
Writing 1 or 2 seconds to memory ahead of record would be a better all-around solution to capture that "action moment" and would pick up other missing bits in addition to particles. But how to do this little time-travel?
taniwha
Posts: 401
Joined: Thu Jan 14, 2010 7:11 am
Contact:

Re: Improving "ProQuake" Demo Record At Any Time

Post by taniwha »

The way it's always done: with a ring-buffer that stores X number of packets. Packets are stuffed into the head of the buffer, and read out of the tail. For this particular case, when not actually recording, when the buffer is full, packets are discarded from the tail.
Leave others their otherness.
http://quakeforge.net/
qbism
Posts: 1236
Joined: Thu Nov 04, 2004 5:51 am
Contact:

Re: Improving "ProQuake" Demo Record At Any Time

Post by qbism »

taniwha wrote:The way it's always done: with a ring-buffer that stores X number of packets. Packets are stuffed into the head of the buffer, and read out of the tail. For this particular case, when not actually recording, when the buffer is full, packets are discarded from the tail.
So, in CL_GetMessage something like

Code: Select all

    if (cls.demorecording)
	        CL_WriteDemoMessage ();
else CL_WriteRingMessage();
where CL_WriteRingMessage is more-or-less a paraphrased writedemomessage that sends to the ring buffer rather than file. Then when it's timed to record, the ring buffer packets just get dumped into the demo file ahead of real-time recording. But is there a catch?
Spike
Posts: 2914
Joined: Fri Nov 05, 2004 3:12 am
Location: UK
Contact:

Re: Improving "ProQuake" Demo Record At Any Time

Post by Spike »

yes, there's a catch. if you have any large state info that can get destroyed/replaced (like stats) you need to provide some way to restore them to what they are at the start of the log, rather than just dumping whatever the client currently has and hope that it hasn't changed.
doing this with quakeworld is much messier with its state-based delta compression.
you can generate a keyframe every second or so if there's a lot of data, though if its just NQ's stats most of it is repeated anyway, memory usage probably allows you to just dump out a copy of that data each frame.
obviously, you'd not be able to rewind over an svc_serverdata. flush the log when you get such a message. perhaps also for static entities/sounds too. baselines shouldn't generate extra data, and shouldn't be changed mid-game so should be safe to rewind over (on account of that baseline not existing and thus not being relevent before).

5 secs is probably enough, 10 secs would be great.
if you've got enough memory, you could even try and record an entire match. a 30-min 4v4 (quakeworld) demo comes out as only 15mb.
taniwha
Posts: 401
Joined: Thu Jan 14, 2010 7:11 am
Contact:

Re: Improving "ProQuake" Demo Record At Any Time

Post by taniwha »

qbism: I meant more

Code: Select all

CL_WriteRingMessage ();  // write message to head of ring buffer
while (CL_RingCount () > max_delay) {
    CL_ReadRingMessage ();  // read message from tail of ring buffer
    if (cl.demorecording)
        CL_WriteDemoMessage ();
}
Some things to watch out for: CL_ReadRingMessage and CL_WriteDemoMessage should be altered to use a message buffer separate from the main message buffer. Also, for speed purposes, CL_ReadRingMessage shouldn't transfer any bytes, but rather update the message buffer pointer.

Also, this is if you want demo recording to always be a little delayed. If you don't want a delay, then yes, your code is correct. When you begin writing the demo, you flush the ring buffer to the demo file, set max_delay to 0, then continue as normal.

Using the above design, you could have it both ways. The only difference would be whether or not you flush the ring buffer and zero max_delay when beginning the recording.
Leave others their otherness.
http://quakeforge.net/
Post Reply