EXT_CSQC_1

Discuss programming in the QuakeC language.
Spike
Posts: 2914
Joined: Fri Nov 05, 2004 3:12 am
Location: UK
Contact:

EXT_CSQC_1

Post by Spike »

I was gonna post this is reply to LordHavoc's reply in the cs_project topic, but its offtopic there, and tbh deserves a topic of its own. Anyway...

As you may know, EXT_CSQC was never really polished. The implementations (both of them) differ from the spec as documented, and it has many features missing that might be useful. It was never properly finished.

The full EXT_CSQC_1 spec is still not finalised fully. And I'd like your input on features and things.
These are the things that come to my mind. I'm sure there are some other issues that can be raised by anyone else.
Many of the items on this list are differences between DP and FTE.
And many of them are to remind me to fix issues in FTE. :)



CSQC_1 features some stat changes based on LordHavoc's recommendation (ie: strings are variable length).

Another thing is the final progs header crc. FTE still accepts any. This was for whatever support could be obtained from the spec when it was properly finalised, and its time to properly finalise it.

Sound events. FTE and DP both do that differently. FTE's is flawed in that it stops triggering if the csqc doesn't know about the entity (phs > pvs, so quite common). DP's is awkward as the sound isn't always linked to the source entity.
Imho, adopt DP's as standard, but add an extra ent parameter, in case csqc is meant to know that entity. This saves a findfloat call.

+showscores is yet another difference. FTE allows/expects the csqc to register this command itself, while DP adds specific handling into the api. FTE also has +showteamscores which DP does not. Personally I prefer FTE's mechanism as standard, as its logically extendable to other similar commands (eg: the +showteamscores command). Although which commands are similar is arguable.
Edit: Apparently nexuiz supports +showscores just fine.

Engine entities/Deltaed entities:
Automatic interpolation is supported.
Fields overwritten each frame are:
entnum, modelindex, skin, (frame1time nyi), (frame2time nyi), frame, frame2, lerpfrac, origin, (velocity qw players only), angles, colormap, scale, alpha.
Update: Callbacks for every single entity every single frame whether it moved or not was resulting in some really noticeable FPS loss. Now, you register a function to be called when the engine would link an engine edict - based on its model name (actually its modelindex). Wildcards should be supported (delayed precaches will need to remember it). if supported (nexuiz players need such selections). Registering by model cuts out a redundant ent-copy and prevents the qc from having to ignore ents it didn't care about. This saves a significant amount of fps for me. It may also allow engine greater code reuse for other engines.

movetypes? Currently its an implement-your-own aproach, and touch events are not easy.
Expanding csqc to do it automatically is possible, but may result in conflicts between current/future mods. It was intended to be a separate extension.

clientside prediction. specifically touching other entities.
there's no touchtriggers function. EnumerateTriggers, pass a function pointer and get told about all the triggers in the box? Extend to solids?
Prediction is fully implemented in my csqctest mod. Moving platforms are a little awkward, but the rest works fine.
Teleporters can be predicted, but finding the triggers is via a linked list in qc.

serverside (custom) prediction handling.
qc is called upon receipt of a movement command. QC must copy entity fields into globals before calling the predict builtin.
QC could just use the ent and predict based on that, but the function is intended to be shared between ss and cs, so the csqc needs to be able to revert the ent to a previous state easily.
I'm having serious doubts over the use of pmove_* in the first place now. input_* is fine still.
csqc ought to read movevars itself. the builtin set are too specific.
The main problem at the mo is triggering touch events. With no parent ent, touches cannot be done via the engine. With no self, the ssqc pmove code can't trigger events either, and tracelines will hit itself.
If self is moved, instead of using globals, this would presumably be more intuitive to qc coders. Touch functions can be called by engine code (and added to qc code). This can work in csqc and ssqc, automatically, no need to worry about touching triggers, as they'd be auto-touched. csqc ents need to be aware that its touches need to be revertable, or might not occur that frame (q3 predicts item grabs).
Thoughts on this?
Last edited by Spike on Wed Apr 15, 2009 1:25 pm, edited 2 times in total.
LordHavoc
Posts: 322
Joined: Fri Nov 05, 2004 3:12 am
Location: western Oregon, USA
Contact:

Post by LordHavoc »

Clearly I need to implement the stat strings support - this was not marked with EXT_CSQC_1 comments in the FTEQW source so I did not find it.
Spike
Posts: 2914
Joined: Fri Nov 05, 2004 3:12 am
Location: UK
Contact:

Post by Spike »

There's another difference, trailparticles parameter order.

While playing around with models, I noticed that DP's gettaginfo builtin does not interpolate like the renderer will.
Additionally, Q3 normalizes each tag axis, while DP (and currently FTE) do not. This can result in some outrageously proportioned player models (see openarena's).
LH's stance appeared to be 'don't fix, let the mod do it if it needs to be done'. Certainly the standard q3 player models all work okay.

Additionally, DP's getinputstate builtin doesn't return any kind of success or failure indicator. Different engines have different input logs, so a meaningful return value lets the mod know when to stop.
LH fixed this one now.
Spike
Posts: 2914
Joined: Fri Nov 05, 2004 3:12 am
Location: UK
Contact:

Post by Spike »

Posts updated. I could really do with some feedback on how you'd like to do things.
Particularly the default-prediction function. Change to ents, or stick with globals?

If I don't get feedback, I'll just end up doing things the easy way then moan when noone follows the 'standard'. Be warned. :P
Chris
Posts: 79
Joined: Sat Aug 05, 2006 5:31 am

Post by Chris »

* full bone controllers, access to all bones / joints / tags in a model and offsetting them / getting and setting data.

* client -> server communication in which I can specify a function to receive the string / float / vector passed in, not have it have to be stuffed into only a couple functions like Gamecommand or ParseClientCommand and deal with a giant switch statement (also server -> client would be nice to have this as well)

* multi-anim support, be able to play multiple animations on the same model by determining which bone in the hierarchy to play below (or above, the exact implementation is irrelevant as long as it is easily exposed to qc and works).

* engine side prediction of origin / angles optional flag for csqc entities

* net priority beyond the "if its distant send less". So I can give priority to things that I feel in qc are more / less important

* post processing effects, whether this be premade glsl shaders exposed to qc or otherwise, just allow for things like motion blur, capture screen and overlay, warping effects, color desaturation.

* sound occlusion abilities / integrate third party sound software for pitch modulation, and premade abilities like reverberation etc.

* properly integrated physics (beyond movetype_bounce etc) such as basic rigid body with constraints control that is handled engine side for optimum speed and exposed to qc thoughtfully with general architecture and attention to use (also known as, well documented)

* built in compression algorithms that automatically compress certain pieces of data and extrapolate that data when sending to csqc.

* flag to expose server side entity to csqc but not have it handle the prediction

* special care for simulated players in csqc in which the prediction is handled by engine however csqc has the information about them when they are relevant (ie, being rendered on screen)

* not needing to change map, ability to do some sort of data streaming for progressive maps so you stream in another map, have qc handle cleaning up it's entities and making pseudo garbage collection and allow not having to load a brand new map.


* render viewport to brush / poly(s)

* make your damn compiler support unique data types / structs

* custom iterators that can be specified in engine to be cataloged for quick access (yes you can make it in qc, but it really shouldn't have to be)

* (.)string to (.)void(...) support in csqc/svqc, so I can set up function pointers read from a text file :P

* seek currently playing ogg/audio file exposed to qc

* true type font or better means than conchars (ew)

* vector graphics, when I run at 640x480 I can't read any text and images are all blurry

* save / load picture from file directly and put this onto a model / poly / brush. (slow bad example but it would work: if I want what I see to be rendered as a half transparent shader over a person)



- These changes might not all be pertinent purely to csqc, but hell I want them.
Spike
Posts: 2914
Joined: Fri Nov 05, 2004 3:12 am
Location: UK
Contact:

Post by Spike »

* full bone controllers, access to all bones / joints / tags in a model and offsetting them / getting and setting data.

Working on that as an extension to csqc. Will not be part of ext_csqc_1 itself, but will work with it.

* client -> server communication in which I can specify a function to receive the string / float / vector passed in, not have it have to be stuffed into only a couple functions like Gamecommand or ParseClientCommand and deal with a giant switch statement (also server -> client would be nice to have this as well)

Hrm, right, extended user inputs. What do you actually see yourself sending anyway?

* multi-anim support, be able to play multiple animations on the same model by determining which bone in the hierarchy to play below (or above, the exact implementation is irrelevant as long as it is easily exposed to qc and works).

Already implemented in FTE (SVN versions, also the qtv versions). Skeletal models are not intended as part of the base spec though. Check for FTE_CSQC_BASEFRAME and then use base* fields for the legs animations (basebone is the separating bone).

* engine side prediction of origin / angles optional flag for csqc entities

Not sure there's much point in interpolation really. If you're sending a rocket in terms of start/velocity, you don't care about interpolating the rocket.
Grenades/gibs would be nice, though. See Movetypes in my first post. I guess this is what you mean by prediction.
The problem is that the engine cannot auto-lerp entities, as such an entity may be added multiple times in a single frame.

* net priority beyond the "if its distant send less". So I can give priority to things that I feel in qc are more / less important

I dread to think of a mod that will actually need to filter like that. Although I can understand that when entering new areas.
I don't feel that this is important enough for the base extension.

* post processing effects, whether this be premade glsl shaders exposed to qc or otherwise, just allow for things like motion blur, capture screen and overlay, warping effects, color desaturation.

for the base extension, it has to be possible to plausably run it in SW, thus post processing effects are not part of the base spec.
Capturing the screen then painting it back via glsl isn't an unreasonable request though.
Would it be for just the viewport though?
Consider that csqc can draw multiple views on the screen at once. Would this effect be per-view, or per frame?
This one doesn't mandate csqc.

* sound occlusion abilities / integrate third party sound software for pitch modulation, and premade abilities like reverberation etc.

Would be nice, but not csqc-specific. Certainly not the base spec.

* properly integrated physics (beyond movetype_bounce etc) such as basic rigid body with constraints control that is handled engine side for optimum speed and exposed to qc thoughtfully with general architecture and attention to use (also known as, well documented)

Yeah. Electro moans about all that too. ragdoll ftw.
The problem is that its such a huge addition.
Not for the base spec. Sorry, if I don't keep focused I'll never 'finish' it.

* built in compression algorithms that automatically compress certain pieces of data and extrapolate that data when sending to csqc.

* flag to expose server side entity to csqc but not have it handle the prediction

Clients do not predict. They only interpolate. It is the csqc that is responsible for prediction.

* special care for simulated players in csqc in which the prediction is handled by engine however csqc has the information about them when they are relevant (ie, being rendered on screen)

Prediction is only performed by the engine when the csqc asks it to do so, based on information the csqc tells it.
Rendering and prediction are not fully linked.

* not needing to change map, ability to do some sort of data streaming for progressive maps so you stream in another map, have qc handle cleaning up it's entities and making pseudo garbage collection and allow not having to load a brand new map.

Now this would be nice...
Although the bulk of this feature would prohibit other engines from implementing the csqc extension (csqc is useless if its not reasonably consistant between engines - I'm scared of anything that is too large or invasive of the rest of the engine).

* render viewport to brush / poly(s)

GL specific. Although not as hard as one might think for SW. The main issue is how to control this. Its fairly trivial to draw a scene on a texture. The catch is doing it without the engine crashing, and only if the relevent textures are visible in the first place.

* make your damn compiler support unique data types / structs

It is meant to support structs already.
But you need pointers in order to use them properly, and that means #pragma TARGET FTE


* custom iterators that can be specified in engine to be cataloged for quick access (yes you can make it in qc, but it really shouldn't have to be)

An example would be nice.
I admit that I don't like .chain.

* (.)string to (.)void(...) support in csqc/svqc, so I can set up function pointers read from a text file :P

FTE_MULTIPROGS can do that.
It can read/write globals/functions out of a progs by name. As well as read/write globals from other progs running in the same VM.

* seek currently playing ogg/audio file exposed to qc

Oggs are a separate extension.
But it would be nice if there was a way to adjust sounds after they have started, yes.

* true type font or better means than conchars (ew)

Talk to bigfoot about that... lol.
DP already has variable width chars. I'm not entirely sure how that all works.

* vector graphics, when I run at 640x480 I can't read any text and images are all blurry

tbh, vid_conwidth not matching vid_width can make certain things really ugly.

* save / load picture from file directly and put this onto a model / poly / brush. (slow bad example but it would work: if I want what I see to be rendered as a half transparent shader over a person)

FTE has a forceshader field which can refer to a shader to force on a model/brush.


- These changes might not all be pertinent purely to csqc, but hell I want them.

Lol :)

I'm trying to solve things. Not find more things to solve! :/
Electro
Posts: 312
Joined: Wed Dec 29, 2004 11:25 pm
Location: Brisbane, Australia
Contact:

Post by Electro »

oi! keep me out of this ;)

All in all looks like good stuff. Just hopefully some of it is documented better from now on and has a kind of CSQC basemod that is made, making use of it all. Otherwise all this will kind of get lost in the wind.. that is traveling ridiculously fast..
Benjamin Darling
http://www.bendarling.net/

Reflex - In development competitive arena fps combining modern tech with the speed, precision and freedom of 90's shooters.
http://www.reflexfps.net/
goldenboy
Posts: 924
Joined: Fri Sep 05, 2008 11:04 pm
Location: Kiel
Contact:

Post by goldenboy »

csqc really needs to be supported by more engines. (Engines that mappers will use, engines that multiplayer types will use)

I have been aware of the things csqc could do for RemakeQuake for quite some time now. But it's not supported well enough. Or documented.

What I'd want: That it works in software, that it is supported by more engines (remaining crossplatform, too), that it is better documented.

As for an example mod ... RemakeQuake could demonstrate a csqc inventory and other stuff pretty well, if it comes to it... I'd estimate that is still some time away though. I would be open toward this, not sure about the others (we have our hands full with other stuff right now). Perhaps some time after qexpo we can start thinking about this. We do eventually want some stuff that would apparently best be done with csqc, though. So we would be a natural ally.

Just thinking out loud :)
Urre
Posts: 1109
Joined: Fri Nov 05, 2004 2:36 am
Location: Moon
Contact:

Post by Urre »

Chris said an important one, csqc->ssqc communication made sane.

The way it works from ssqc->csqc seems more sane and specificly dedicated for the task than all the interpretation and parsing you need to do when sending info from csqc to ssqc.

I currently have the csqc control the clients position completely, and want to send this information to ssqc for proper pvs updates, overriding the current ssqc position of the client. I also send mouse click events, which vary depending on context, and it feels very convoluted and weird right now, sending commands through the console and whatnot...

The area of uses will expand for me, and I bet Chris already has a wider span of tasks where csqc needs to communicate to ssqc.

On a side-note, mostly DP specific and doesn't have much to do with the spec, I really want packetloss handling and DP7 distance-handling for shared ents. Currently none of those are supported, even if they exist for regular serverside entities. I bet FTE would benefit from these features as well.

I'll post more thoughts as they arise, but the communication is a big one.
I was once a Quake modder
mh
Posts: 2292
Joined: Sat Jan 12, 2008 1:38 am

Post by mh »

goldenboy wrote:csqc really needs to be supported by more engines.
Too true. One thing I'd really like to see is a "sample implementation" built on a vanilla Quake source base, so that we could get the diffs and start adding support properly to other engines.
We had the power, we had the space, we had a sense of time and place
We knew the words, we knew the score, we knew what we were fighting for
Spike
Posts: 2914
Joined: Fri Nov 05, 2004 3:12 am
Location: UK
Contact:

Post by Spike »

Urre wrote:Chris said an important one, csqc->ssqc communication made sane.

The way it works from ssqc->csqc seems more sane and specificly dedicated for the task than all the interpretation and parsing you need to do when sending info from csqc to ssqc.
for ssqc->csqc, you're expected to use stats and ents to transfer such data across.
for csqc->ssqc, ents don't make sense.
But stats do.
The added complication is that csqc->ssqc is expected to require prediction on any outgoing commands.
In order to predict, the commands need to be buffered somewhere. In order to avoid large arrays/entity chains, the buffering is done in the engine, where they were generated in the first place. But as more fields added to this input log, you'll see a linear deceleration of the engine.
Now, in your case, you want a vector each frame, and two floats (or maybe a second vector) each time they click.
For the origins, you want it pretty much every frame, as that's logical and ought to be deltaed somehow (ie: should be packed into the input state). But for clicks, it really is possible to click twice within a single frame (ie: needs explicit, and reliable, .notifications to the server).
Regarding inventories and things, it actually makes sense to go via the console (which is why I never really thought about the limitations). Going via the console ensures that you can use the same console commands via aliases as via csqc.

But at the end of the day, if you're cycling through 15 inventory items to get to the one you want, you'd prefer the cycling to be done locally rather than with eg 300ms latency.
I am still considering the idea of predicting stat changes (ie: you can override a stat until the server acks the current outgoing frame, at which point it reverts if the server didn't also change it).

The expectation is that, for example a diabloo-style inventory, you use logical slot-id commands, rather than actual click coords, in order to swap and use things:
localcmd("cmd invswap", ftos(sourceslot), " ", ftos(destslot), "\n");

Personally I don't think its too bad... but you do loose precision.

How about:
sendrequest("invswap", "ff", sourceslot, destslot);
matching to:
void(float sourceslot, float destslot) UserCmd_invswap_ff;
Suggestions?
Would this would be easier to use, and less convoluted?
(strings would be temps, the engine might need 6 temp strings here, ents would read out entnum on the client, but the server may convert to world if they were freed. There's a limit of 6 parameters (can be vectors or infostrings)).

On the other hand, input is extended with:
vector input_customv1;
vector input_customv2;
string input_customs1;
entity input_custome1;

obviously you can store floats in the vectors, and the string is self explainatory. But the ent index that is sent to the server is actually input_custome1.entnum - but if it gets removed or is out of pvs, then the event log's custome can fail and spontaneously become world - at either end.
I'm not entirely sure what fields prydoncursor uses. Prydoncusor's input state probably ought to be included as well, separately. And I'm not sure how many fields there need to be. 8 independent components sounds like a good number to me though.

PVS updates is an additional thing. With FTE you can set self.view2.origin to give you a second pvs window. But that's hijacking specific functionality provided by a different extension, just as using EF_NODEPTHTEST is useful for sending ents without pvs checks.
On a side-note, mostly DP specific and doesn't have much to do with the spec, I really want packetloss handling and DP7 distance-handling for shared ents. Currently none of those are supported, even if they exist for regular serverside entities. I bet FTE would benefit from these features as well.
Actually, this affects FTE much much less. DP caps packet sizes based on your rate, ensuring a consistent packet rate. FTE has fixed size (fixed-max at least) packets and simply drops packets if your rate is too low.
Its rare that any individual packet reaches the cap, so csqc ents don't go missing so much in FTE.
But I can understand the lack of prioritisation being a major issue when the packets are a limited size and only the first 5 are sent (instead of first 20).
Its a server implementation difference rather than a protocol difference.
Personally I feel that its fine so long as its no worse than it would normally be if you were not using csqc. For DP this isn't true, but this is really a DP issue.
One thing I'd really like to see is a "sample implementation" built on a vanilla Quake source base
Started. And trying to do it without breaking (existing) qccx hacks. This might be a challenge. Any such awkwardness will be duely noted. So far, QCLib #includes and links without conflict. Physics is supposedly implemented, delta updates *should* work. Input logs will make the prediction parts of csqc awkward without breaking qccx hacks when csqc is not in use. csqc can resize the screen... But that's about it... Oh, and water makes it crash. And there is a single bugged triangle on one model each frame and its really really annoying that this only happens to a single poly, and only if csqc is in use. grr. Maybe I shoulda stuck with GL. I'll get there in the end, but I may go insane in the process. I'm not gonna do separate nq/qw versions, but I will get nq gl+sw using csqc, eventually. Any preference to net protocol versions? And how to query support? Cos I'm not gonna break networking even with csqc in use.
Chris
Posts: 79
Joined: Sat Aug 05, 2006 5:31 am

Post by Chris »

I always thought of it you target what function you want to send to which'd have an identical function signature. So for example in csqc:


void SendDataToServer(string funcName, string data);

So then you have a function that takes the following parameters after it possibly and then in svqc you have that function which gets filled with those values passed. And converts string to the proper function name and inputs the 'data' parameter as it's only argument.


For more complex things like say you don't want to deal with parsing a string and sending any combination of floats / vectors I don't know, I just dislike having to use only a couple of outlets to send data to, I also don't like how any data being sent to and fro from svqc/csqc with the current method is easily replicatable via the console making it extremely easy to mess with just by chance. Whether the whole system is safe or not something not exposed to a simple console terminal is still miles better.

I don't pretend like I know the best way to implement this, there might be a better design out there, this was more a knee jerk reaction to how I'd feel I'd use it.
Spike
Posts: 2914
Joined: Fri Nov 05, 2004 3:12 am
Location: UK
Contact:

Post by Spike »

Part of the awkwardness of client commands is the fact that you have to tokenise it. so any string with quotes in is automagically broken. And such strings may be able to do bad things to a mod.
I don't really want to restrict mods to ftosing floats and stofing them the other end either.

The reason my example had the types listed is because an engine cannot determin the types that were actually passed. The distinction is important when its a vector vs a string. Strings are very good at crashing quake.

thus:
sendrequest("invswap", "ff", sourceslot, destslot);
on client
matches to
void(float sourceslot, float destslot) UserCmd_invswap_ff;
on server
the two floats could be strings if you used "ss" instead of "ff" or vectors, or ents. but not functions. and not fields. so basically just sfve. And capped to max 6 args due to progs restrictions (two are already in use).
Seeing as you don't have to ftos stuff, there's no loss of precision. But yeah, you might find entities become world.

This would make life much easier, no?
Spike
Posts: 2914
Joined: Fri Nov 05, 2004 3:12 am
Location: UK
Contact:

Post by Spike »

Software renderers...
The csqc spec as it currently stands requires that an engine support scaling, recolouring, and alpha blending. Trivial things for GL or D3D to support, but things that are a major pain for 8bpp paletted sw renderers.

Is it sane to require a paletted renderer to blend palette indexes like that?
Writing the code to do this isn't the issue. Writing the code to do this and still obtain decent frame rates is the real issue.
Urre
Posts: 1109
Joined: Fri Nov 05, 2004 2:36 am
Location: Moon
Contact:

Post by Urre »

I doubt most will care in the end, but if you want to do things right, you would drop that requirement. Especially seeing you've already had software renderers as the main reason not to implement a bunch of graphics-specific features as part of the base spec. I'd say this is especially important if we are to see csqc really happen in engines on other platforms, like various handhelds and whatnot.

Concerning the csqc->ssqc talk, have I got it right if I interpret your example as the first two parameters of the send function being the last two parts of the function name in the recieve function? Cause that seems pretty awesome.
I was once a Quake modder
Post Reply