FTE_MULTIPROGS how to do that?

Discuss CSQC related programming.
Post Reply
toneddu2000
Posts: 1395
Joined: Tue Feb 24, 2009 4:39 pm
Location: Italy

FTE_MULTIPROGS how to do that?

Post by toneddu2000 »

ok, I said to myself: nice and easy. Let's add to csprogs.dat, another hello.dat which has only one stupid function that prints "hello" in csprogs.dat. Why should it be so difficult, huh? Huh? Huh?
Well, it's kinda difficult! :biggrin:

hello.dat

Code: Select all

void CSQC_Init(float apilevel, string enginename, float engineversion)
{
	//FTE_MULTIPROGS init
	init(-1);//-1 consider this as "main" .dat file
	initents();
}

extern void Ext_Hello()
{
	cprint("Hello\n");
}
csprogs.dat

Code: Select all

float MULTIDAT_HELLO;
void CSQC_Init(float apilevel, string enginename, float engineversion)
{
	//FTE_MULTIPROGS load
	MULTIDAT_HELLO= addprogs("hello.dat");
	//..other useless junk
}

void CSQC_UpdateView(float vwidth, float vheight, float notmenu)
{
	//..useless junk before
	externcall(MULTIDAT_HELLO,"ExtUI_Hello");
	//..useless junk after
}
Without calling the externcall, csprogs works but with a warning

Code: Select all

You are trying to load a string-stripped progs as an addon. This behaviour is not supported. Try removing some optimizations.
calling externcall will make, obviously, FTE crash. Engine code says in pr_edict.c that

Code: Select all

//friked progs only allow one file.
??What are friked progs?
And also

Code: Select all

//progs 0 always acts as string stripped.
//partly to avoid some bad/optimised progs.
Tried to change init to 0 and even to -2 but same error

Please Spike, help! :biggrin:
Meadow Fun!! - my first commercial game, made with FTEQW game engine
Spike
Posts: 2914
Joined: Fri Nov 05, 2004 3:12 am
Location: UK
Contact:

Re: FTE_MULTIPROGS how to do that?

Post by Spike »

An addon that hooks itself in

Code: Select all

#praga progs_dat "someaddon.dat"
#pragma target FTE

#define MAINPROGS 0 //sometimes a constant just makes stuff easier.
#define ANYPROGS -1 //checks every progs for whatever simple (uses first symbol found)

extern void() test123; //function auto-imported from another progs.
var void(float vwidth, float vheight, float notmenu) origupdateview;
void(float vwidth, float vheight, float notmenu) wrapperupdateview
{
    print("prewrapper\n");
    origupdateview();
    print("postwrapper\n");
};
float(float arg) calledfrommainprogs =
{
    print("calledfrommainprogs was called\n");
    return arg+1;
};
void() init =
{
    print("Addon init called\n");
    void(float vwidth, float vheight, float notmenu) *ptr = externvalue(MAINPROGS, "&CSQC_UpdateView");
    origupdateview = *ptr;
    *ptr = wrapperupdateview; //zomg the csprogs probably defined it (implicitly) as const.
    test123();
};

Code: Select all

#pragma progs_dat "csprogs.dat"
float somehandle;
void(float apilevel, string enginename, float engineversion) CSQC_Init = 
{
    somehandle = addprogs("someaddon.dat");
    if (somehandle < 0)
        print("it failed, dude\n");
};
var void CSQC_UpdateView(float vwidth, float vheight, float notmenu) =
{ //put what you want in here.
    print("regular updateview\n");

    if (somehandle < 0)
        return;

    if (2 != (float)externcall(somehandle, "calledfrommainprogs", 1)) error("gah"); //a lazy call
    var float(float) foo = externvalue(somehandle, "calledfrommainprogs");  //yay for caching something
    if (2 != foo(1)) error("gah");  //no slower than any qc function call, also supports up to 8 args rather than 6.
};
void() test123 = //some test function for the addon to call
{
    print("testing 1... 2... 3...\n");
};
for lack of a better name, a 'friked' progs is one that has been compiled with string constants stripped from the progs.
with fteqcc, you should never use -O3 for addons. -O2 is the highest optimisation level supported for addons.

so yeah, strings...
in qc, string immediates are stored as an offset from the string table, the first byte of which is always empty, thus the null string is always empty.
side note: in 32bit processes, this string offset thing works well enough to refer to any string in the engine's memory space, and this is why you can't just recompile vanilla quake as 64bit (because pointer->string->pointer offsets get truncated and the pointers end up pointing to the wrong place with a crash when its executed).
anyway, this string table thing is the text blob that you can normally read inside the progs file. all qc strings are expressed relative to that (unless the engine uses some of the high bits to denote special string types from eg strzone or tempstrings or whatever).
this of course is a problem when there are logically multiple string tables, which would make strings in one progs unreadable gibberish while running code in another, so to fix that fteqw loops through some 'defs' table within the progs and biases all the strings that it finds to be relative to the string table within the main progs, allowing entity fields to be read as actual strings, with no special conversions needed for string args between addons, etc.
obviously this only works when the defs table actually still includes all the string immediates - if that data was stripped out then those embedded strings will be gibberish. and that's what fteqw is noticing and complaining about in an attempt to avoid total malfunction. obviously you can still end up with false positives if you don't have any string immediates, but that generally only happens in small test mods...
note that stripped string immediates is fine within the main progs, as there's no need to rebias when they're already set correctly anyway.
toneddu2000
Posts: 1395
Joined: Tue Feb 24, 2009 4:39 pm
Location: Italy

Re: FTE_MULTIPROGS how to do that?

Post by toneddu2000 »

Wow thanks Spike, I didn't know fteqcc did optimisations at all! I'll use O2 from now on for addons

Very interesting the concept of create "hook" functions like Drupal for example does, which it's quite cool for devs that don't want to re-invent the wheel but add only new functionalities. UT mutators-like will be a breeze with this method.

Unfortunately your code throws error at

Code: Select all

void(float vwidth, float vheight, float notmenu) *ptr = externvalue(MAINPROGS, "&CSQC_UpdateView");
csqc.c:23: error: "*" - not a name
I don't understand if your exporting a value or a function (what is ptr? It seems a func name). I'm sorry but pointers litteraly freak me out!
Another thing. You declared test123() func in addons.dat but you initialized it in csprogs.dat. I'd need the opposite: create all the code in addons.dat and jus call the function in csprogs.dat.
Is it possible? There's a simpler way that don't involve pointers? Pleease! :biggrin:
I could be good enough without using hook functions as first experiment, just putting all the code in addons and using it in csprogs
Thanks a bunch again
Meadow Fun!! - my first commercial game, made with FTEQW game engine
frag.machine
Posts: 2126
Joined: Sat Nov 25, 2006 1:49 pm

Re: FTE_MULTIPROGS how to do that?

Post by frag.machine »

toneddu2000 wrote:Another thing. You declared test123() func in addons.dat but you initialized it in csprogs.dat. I'd need the opposite: create all the code in addons.dat and jus call the function in csprogs.dat.
Just invert things ? Unless there is some CSQC specific issue I am not aware, it's a matter of placing in csprogs.dat:

Code: Select all

extern void test123(); // this line just tells csprogs.dat about the existence of test123() somewhere
And in addons.dat:

Code: Select all

void test123() =
{
  dprint ("123"); // or whatever other fancy stuff you want
}
The only problem/weirdness I see is creating a runtime dependency of addons.dat inside csprogs.dat.
Usually this would NOT be desirable (it would be okay if addons.dat does require csprogs.dat though).
Personally, I'd avoid making things more complex than required but hey, it's perfectly viable from the technical point of view.
I know FrikaC made a cgi-bin version of the quakec interpreter once and wrote part of his website in QuakeC :) (LordHavoc)
Spike
Posts: 2914
Joined: Fri Nov 05, 2004 3:12 am
Location: UK
Contact:

Re: FTE_MULTIPROGS how to do that?

Post by Spike »

'extern' only works to import from progs which are already loaded. they won't work the other way around (and you don't really know which dat the symbol was copied from either), so be sure to use it only when its meant to exist in the main progs. also, only use it for functions.

I don't know why you're getting a syntax error from that def.
origupdateview = externvalue(MAINPROGS, "CSQC_UpdateView");
externset(MAINPROGS, wrapperupdateview, "CSQC_UpdateView");
those two lines should work the same, to read the prior value and then clobber it with some replacement/wrapper function (which can call the original function as needed). no pointers needed.

note that the CSQC_UpdateView function I wrote provides two alternative ways to call a function from the main progs. one is by name, the other does a name lookup in one (slow) step up then calls by value (which can be repeated without the need for constant lookups).
however, I'd personally recommend to make 'register' functions for various things that can then be called from both the main progs or your addon's init functions, then you're not quite so limited in what addons can do when you want to extend them.
toneddu2000
Posts: 1395
Joined: Tue Feb 24, 2009 4:39 pm
Location: Italy

Re: FTE_MULTIPROGS how to do that?

Post by toneddu2000 »

frag.machine wrote:The only problem/weirdness I see is creating a runtime dependency of addons.dat inside csprogs.dat.
Usually this would NOT be desirable (it would be okay if addons.dat does require csprogs.dat though).
Personally, I'd avoid making things more complex than required but hey, it's perfectly viable from the technical point of view.
Thanks frag.machine for the suggestion, I only wanted to make things tidy and to release separated addons that devs could use in their main progs without re-inventing the wheel (UI most of all, because it's a painful process), but, if it's not possible to do, I'll stick with original method to release ui source code + a README file where explain where putting files in progs.src and how to call functions. Just for a moment a dreamed about an Open FTE Marketplace like Unreal or Unity's ones.... :lol:
Spike wrote:'extern' only works to import from progs which are already loaded. they won't work the other way around (and you don't really know which dat the symbol was copied from either), so be sure to use it only when its meant to exist in the main progs. also, only use it for functions.
mmm..got it. You think there's chances you could add in the future something like this

Code: Select all

getExternalProgs("someprogs.dat").blah()
or is it impossible to implement it because of quake .dat structure?
I don't know why you're getting a syntax error from that def.
origupdateview = externvalue(MAINPROGS, "CSQC_UpdateView");
externset(MAINPROGS, wrapperupdateview, "CSQC_UpdateView");
those two lines should work the same, to read the prior value and then clobber it with some replacement/wrapper function (which can call the original function as needed). no pointers needed.
No more error, thanks! But now, when map starts, it prints on top screen .com.com.com.com.com.....and so on, for 4 lines! :shock:
I have to comment print function in calledfrommainprogs to make it stop.
The problem is that I don't understand the utility to have a function declaration in main progs and call it in additional ones. For things like UI or Skeleton libraries, this method wouldn't help me.

Anyway the code you posted unfortunately doesn't print anything on screen or in console, so it seems that additional progs is loaded(in fact no error "gah"), but parented functions are lost
Meadow Fun!! - my first commercial game, made with FTEQW game engine
toneddu2000
Posts: 1395
Joined: Tue Feb 24, 2009 4:39 pm
Location: Italy

Re: FTE_MULTIPROGS how to do that?

Post by toneddu2000 »

Anyway the code you posted unfortunately doesn't print anything on screen or in console, so it seems that additional progs is loaded(in fact no error "gah"), but parented functions are lost
Little update to what I wrote: copying exactly your code both in addons and main progs now it prints "dude" and "regular updateview", so it lost "it failed, dude". If I remove comma and space in string("it failed dude"), it prints "ude" and "regular updateview". If I replace string with ftos(somehandle), it prints "ateview" and "regular updateview", but first it printed something like "mmainprogs" (with 2 m, I guess) :?:

Plus, I don't understand this passage

Code: Select all

if (2 != (float)externcall(somehandle, "calledfrommainprogs", 1)) error("gah\n"); //a lazy call
...
if (2 != foo(1)) error("gah\n");
What that "2" stands for? Is the return float from calledfrommainprogs func?

Anyway, this is the code that compiles but doesn't print 1..2..3

addon

Code: Select all

#pragma 	progs_dat "../../someaddon.dat"
#pragma 	target FTE
#include 	"defs.c"
#define 	MAINPROGS 0 //sometimes a constant just makes stuff easier.
#define 	ANYPROGS -1 //checks every progs for whatever simple (uses first symbol found)


extern void() test123; //function auto-imported from another progs.
var void(float vwidth, float vheight, float notmenu) origupdateview;
void(float vwidth, float vheight, float notmenu) wrapperupdateview
{
	print("prewrapper\n");
	origupdateview(800,600,0);
	print("postwrapper\n");
};

float(float arg) calledfrommainprogs =
{
	print("calledfrommainprogs was called\n");
	return arg+1;
};

void(float prevprogs) init =
{
	print("Addon init called\n");
	origupdateview = externvalue(MAINPROGS, "CSQC_UpdateView");
	externset(MAINPROGS, wrapperupdateview, "CSQC_UpdateView");
	test123();
};
csprogs

Code: Select all

#pragma 	progs_dat "../../csprogs.dat"
#pragma 	target FTE
#define 	CSQC
#include	"defs.c"

float somehandle;
void(float apilevel, string enginename, float engineversion) CSQC_Init = 
{
	somehandle = addprogs("someaddon.dat");
	if (somehandle < 0){
		print("it failed, dude");
	}
};

var void CSQC_UpdateView(float vwidth, float vheight, float notmenu) =
{
	//put what you want in here.
	print("regular updateview\n");
	//return if not present addon
	if (somehandle < 0)
		return;
	//external calls
	if (2 != (float)externcall(somehandle, "calledfrommainprogs", 1)) error("gah\n"); //a lazy call
	var float(float) foo = externvalue(somehandle, "calledfrommainprogs");  //yay for caching something
	if (2 != foo(1)) error("gah2\n");  //no slower than any qc function call, also supports up to 8 args rather than 6.
	//color bg or screen won't be usable for print stuff
	drawfill([0,0,-100],[vwidth,vheight,0],[0.1,0.6,0.5],1);
};

void() test123 = //some test function for the addon to call
{
	print("testing 1... 2... 3...\n");
};
Meadow Fun!! - my first commercial game, made with FTEQW game engine
Post Reply