Porting codebases to 64 bit

Discuss programming topics for the various GPL'd game engine sources.
JasonX
Posts: 422
Joined: Tue Apr 21, 2009 2:08 pm

Porting codebases to 64 bit

Post by JasonX »

I started cleaning up and fixing some old engine ports i made, all based on sdlquake, on a 64 bit linux box. I'm not able to run my ports anymore, since i'm getting lot's of segfaults. When loading a new game for example, i get a segfault on the usage of the strcmp inside PR_ExecuteProgram. There are other crashes and i'm pretty sure this has to do with 64 code.

Do you guys have any tips and would like to share any caveats faced when porting your codebases to 64 bit? I don't have lot's of experience on the topic and the tutorials i read after a quick google search did not help a lot (no mentions of std lib caveats, like this strcmp one).
Spike
Posts: 2914
Joined: Fri Nov 05, 2004 3:12 am
Location: UK
Contact:

Re: Porting codebases to 64 bit

Post by Spike »

QC strings are expressed as indexes relative to the stringtable loaded from the progs.dat.
the correct way is to make multiple types of strings - make ones with the high bit set are indexes into a pointer table instead, and ones with it clear are reguar stringtable indexes.
the lazy way is to ensure that your string_t type is signed, and to allocate space for all the strings that you will use on the hunk or zone memory (as these will always be within 2gb of the progs string table).
Currently it'll crash whenever the QC tries to reference the player's netname or refer to a string pointing to a buffer inside ftos for example. There are other extension builtins of course where stuff like reading cvar strings is even more problematic, but for vanilla you should be able to get away with just netnames and ftos.

the issue is not strcmp, its with address truncation on the 'qcstring = nativestring - progs->stringtable' logic. the result of the negation needs more than 32 bits, but you can't change the type of qcstring(ie: string_t).
It means that 'nativestring = (char*)progs->stringtable+qcstring' calculates a different offset for nativestring and then you get a segfault when its read.

there's a 64bit bug in sprites too, iirc
JasonX
Posts: 422
Joined: Tue Apr 21, 2009 2:08 pm

Re: Porting codebases to 64 bit

Post by JasonX »

string_t type? Inside pr_exec?
Spike
Posts: 2914
Joined: Fri Nov 05, 2004 3:12 am
Location: UK
Contact:

Re: Porting codebases to 64 bit

Post by Spike »

typedef int string_t;
see. 32bit.

sv.clients[0].edict->v.netname = "some string thats somewhere in the .cdata section of your binary" - pr_strings;
overflow(OVR_DONTREALISE);
waitfortheqcakaprexectotestselfdotnetname();
c->_float = !strcmp(pr_strings+a->string,pr_strings+b->string);
crash(SIG_SEGFAULT, qcstring);

if you're using msvc, try using the 'Smaller Type Check' setting. You should see these overflows that way (it will still hide things that get cast explicitly though - this shouldn't be one of those).
JasonX
Posts: 422
Joined: Tue Apr 21, 2009 2:08 pm

Re: Porting codebases to 64 bit

Post by JasonX »

I have changed to a signed value, but i keep getting segfaults. Is there any good tool for aiding the port of 32-bit applications to 64-bit on Linux?
Spike
Posts: 2914
Joined: Fri Nov 05, 2004 3:12 am
Location: UK
Contact:

Re: Porting codebases to 64 bit

Post by Spike »

it doesn't matter if its signed or unsigned. if its a 32bit offset into a 64bit address space where the two pointers are greater than 2gb/4gb apart, you will have problems.
the only semi-trivial fix is to use a signed integer and to allocate *all* qc-accessible strings on the quake hunk (player names and temp strings will need to be moved). naturally this will still break many qccx hacks.
address space layout randomisation likes using the WHOLE 47-bit address space available to it - nice big gaps make it harder to guess where the actual code is, avoiding potential exploits.

I don't know about linux, but msvc has a 'smaller type check' setting that will generate some extra code that triggers a fault whenever a variable is stored into a variable that is too small for the value. this should generally be sufficient to catch pointer offset overflows, but does depend upon the result not explicitly being cast to an int. if all goes well, this gives you a fault when the subtraction happens instead of merely where the dereference happens.
linux seems to only have compile-time warnings, and will thus probably contain lots of false positives.
either way, you're actually going to have to use a debugger.
JasonX
Posts: 422
Joined: Tue Apr 21, 2009 2:08 pm

Re: Porting codebases to 64 bit

Post by JasonX »

I'm not sure if i follow. Instead of allocating inside the function heap, should i add the strings to the Quake hunk?
Spike
Posts: 2914
Joined: Fri Nov 05, 2004 3:12 am
Location: UK
Contact:

Re: Porting codebases to 64 bit

Post by Spike »

exactly that. instead of using globals or malloc, you should use Hunk_Alloc (or friends) for qc-accessible strings. This ensures all the strings are within the same 2gb range as the progs.dat's stringtable, and thus prevents the overflows.
its lame, but it works.
jitspoe
Posts: 217
Joined: Mon Jan 17, 2005 5:27 am

Re: Porting codebases to 64 bit

Post by jitspoe »

What's the point in switching to 64bit? Are you planning to address more than 2 gigs of memory?
Baker
Posts: 3666
Joined: Tue Mar 14, 2006 5:15 am

Re: Porting codebases to 64 bit

Post by Baker »

jitspoe wrote:What's the point in switching to 64bit? Are you planning to address more than 2 gigs of memory?
Linux on 64-bit won't run 32-bit binaries.

Windows will. The Mac will. Linux = no. So you have to convert to 64-bit on Linux.
The night is young. How else can I annoy the world before sunsrise? 8) Inquisitive minds want to know ! And if they don't -- well like that ever has stopped me before ..
Spirit
Posts: 1065
Joined: Sat Nov 20, 2004 9:00 pm
Contact:

Re: Porting codebases to 64 bit

Post by Spirit »

You just need to install 32 bit libraries on Linux.
Improve Quaddicted, send me a pull request: https://github.com/SpiritQuaddicted/Quaddicted-reviews
Spike
Posts: 2914
Joined: Fri Nov 05, 2004 3:12 am
Location: UK
Contact:

Re: Porting codebases to 64 bit

Post by Spike »

for debian/ubuntu:
apt-get install package-name:i386
no need to port, but 64bit users are less likely to have 32bit versions of your dependancies installed, and if you only provide 32bit dependancy lists then they'll need to tweak them (and figure out any omissions too).
you probably still want to as a personal pride sort of thing, but its really just a dependancy issue. there's no reason you can't make a (bash) install script that downloads+installs the appropriate dependancies along with your game.

abuse of textures can result in a need for more than 2gb ram. both dp and fte were modified to make use of windows' large-address-aware thing to allow 32bit versions to access 4gb of ram on 64bit windows (or 3gb on 32bit windows), in order to avoid oom-crashing when used with the ultra version of rygel's texture package.
quite impressive for a 20-year-old game tbh.
Baker
Posts: 3666
Joined: Tue Mar 14, 2006 5:15 am

Re: Porting codebases to 64 bit

Post by Baker »

Spirit wrote:You just need to install 32 bit libraries on Linux.
Didn't know that but if the answer is weird enough a pedestrian won't know, it may as well be. :D

For sure a 32-bit binary for Linux is a true 2nd class citizen because it needs bizarre treatment. Almost as bad as using WINE.
Spike wrote:both dp and fte were modified to make use of windows' large-address-aware thing to allow 32bit versions to access 4gb of ram on 64bit windows
Very interesting.
The night is young. How else can I annoy the world before sunsrise? 8) Inquisitive minds want to know ! And if they don't -- well like that ever has stopped me before ..
Spirit
Posts: 1065
Joined: Sat Nov 20, 2004 9:00 pm
Contact:

Re: Porting codebases to 64 bit

Post by Spirit »

Baker wrote:For sure a 32-bit binary for Linux is a true 2nd class citizen because it needs bizarre treatment.
Not if you "do the right thing" and get your software packaged by smart maintainers. I don't want to think about the blood and tears that would require though. PS: Archlinux' AUR rocks and makes maintaining/installing packages trivial ;)
Improve Quaddicted, send me a pull request: https://github.com/SpiritQuaddicted/Quaddicted-reviews
qbism
Posts: 1236
Joined: Thu Nov 04, 2004 5:51 am
Contact:

Re: Porting codebases to 64 bit

Post by qbism »

Baker wrote:For sure a 32-bit binary for Linux is a true 2nd class citizen because it needs bizarre treatment. Almost as bad as using WINE.
Sometimes worse than Wine. :lol: A win32 binary might work out-of-the-box on Linux64 when a Linux32 won't.

Besides FTEQW, Hammer of Thyrion might be a good source to look at. Multi-platform build that is closer to a vanilla quake-like engine. I haven't tried the Linux builds but both 32 and 64-bit cross-compiled builds work well on Windows.
Post Reply