Forum

Porting codebases to 64 bit

Discuss programming topics for the various GPL'd game engine sources.

Moderator: InsideQC Admins

Porting codebases to 64 bit

Postby JasonX » Sat Mar 29, 2014 3:31 pm

I started cleaning up and fixing some old engine ports i made, all based on sdlquake, on a 64 bit linux box. I'm not able to run my ports anymore, since i'm getting lot's of segfaults. When loading a new game for example, i get a segfault on the usage of the strcmp inside PR_ExecuteProgram. There are other crashes and i'm pretty sure this has to do with 64 code.

Do you guys have any tips and would like to share any caveats faced when porting your codebases to 64 bit? I don't have lot's of experience on the topic and the tutorials i read after a quick google search did not help a lot (no mentions of std lib caveats, like this strcmp one).
JasonX
 
Posts: 407
Joined: Tue Apr 21, 2009 2:08 pm

Re: Porting codebases to 64 bit

Postby Spike » Sat Mar 29, 2014 4:44 pm

QC strings are expressed as indexes relative to the stringtable loaded from the progs.dat.
the correct way is to make multiple types of strings - make ones with the high bit set are indexes into a pointer table instead, and ones with it clear are reguar stringtable indexes.
the lazy way is to ensure that your string_t type is signed, and to allocate space for all the strings that you will use on the hunk or zone memory (as these will always be within 2gb of the progs string table).
Currently it'll crash whenever the QC tries to reference the player's netname or refer to a string pointing to a buffer inside ftos for example. There are other extension builtins of course where stuff like reading cvar strings is even more problematic, but for vanilla you should be able to get away with just netnames and ftos.

the issue is not strcmp, its with address truncation on the 'qcstring = nativestring - progs->stringtable' logic. the result of the negation needs more than 32 bits, but you can't change the type of qcstring(ie: string_t).
It means that 'nativestring = (char*)progs->stringtable+qcstring' calculates a different offset for nativestring and then you get a segfault when its read.

there's a 64bit bug in sprites too, iirc
Spike
 
Posts: 2883
Joined: Fri Nov 05, 2004 3:12 am
Location: UK

Re: Porting codebases to 64 bit

Postby JasonX » Sun Mar 30, 2014 1:39 am

string_t type? Inside pr_exec?
JasonX
 
Posts: 407
Joined: Tue Apr 21, 2009 2:08 pm

Re: Porting codebases to 64 bit

Postby Spike » Sun Mar 30, 2014 5:45 am

typedef int string_t;
see. 32bit.

sv.clients[0].edict->v.netname = "some string thats somewhere in the .cdata section of your binary" - pr_strings;
overflow(OVR_DONTREALISE);
waitfortheqcakaprexectotestselfdotnetname();
c->_float = !strcmp(pr_strings+a->string,pr_strings+b->string);
crash(SIG_SEGFAULT, qcstring);

if you're using msvc, try using the 'Smaller Type Check' setting. You should see these overflows that way (it will still hide things that get cast explicitly though - this shouldn't be one of those).
Spike
 
Posts: 2883
Joined: Fri Nov 05, 2004 3:12 am
Location: UK

Re: Porting codebases to 64 bit

Postby JasonX » Thu Sep 11, 2014 3:44 am

I have changed to a signed value, but i keep getting segfaults. Is there any good tool for aiding the port of 32-bit applications to 64-bit on Linux?
JasonX
 
Posts: 407
Joined: Tue Apr 21, 2009 2:08 pm

Re: Porting codebases to 64 bit

Postby Spike » Thu Sep 11, 2014 1:08 pm

it doesn't matter if its signed or unsigned. if its a 32bit offset into a 64bit address space where the two pointers are greater than 2gb/4gb apart, you will have problems.
the only semi-trivial fix is to use a signed integer and to allocate *all* qc-accessible strings on the quake hunk (player names and temp strings will need to be moved). naturally this will still break many qccx hacks.
address space layout randomisation likes using the WHOLE 47-bit address space available to it - nice big gaps make it harder to guess where the actual code is, avoiding potential exploits.

I don't know about linux, but msvc has a 'smaller type check' setting that will generate some extra code that triggers a fault whenever a variable is stored into a variable that is too small for the value. this should generally be sufficient to catch pointer offset overflows, but does depend upon the result not explicitly being cast to an int. if all goes well, this gives you a fault when the subtraction happens instead of merely where the dereference happens.
linux seems to only have compile-time warnings, and will thus probably contain lots of false positives.
either way, you're actually going to have to use a debugger.
Spike
 
Posts: 2883
Joined: Fri Nov 05, 2004 3:12 am
Location: UK

Re: Porting codebases to 64 bit

Postby JasonX » Sat Sep 13, 2014 9:34 pm

I'm not sure if i follow. Instead of allocating inside the function heap, should i add the strings to the Quake hunk?
JasonX
 
Posts: 407
Joined: Tue Apr 21, 2009 2:08 pm

Re: Porting codebases to 64 bit

Postby Spike » Sat Sep 13, 2014 11:41 pm

exactly that. instead of using globals or malloc, you should use Hunk_Alloc (or friends) for qc-accessible strings. This ensures all the strings are within the same 2gb range as the progs.dat's stringtable, and thus prevents the overflows.
its lame, but it works.
Spike
 
Posts: 2883
Joined: Fri Nov 05, 2004 3:12 am
Location: UK

Re: Porting codebases to 64 bit

Postby jitspoe » Fri Feb 27, 2015 1:30 am

What's the point in switching to 64bit? Are you planning to address more than 2 gigs of memory?
jitspoe
 
Posts: 217
Joined: Mon Jan 17, 2005 5:27 am

Re: Porting codebases to 64 bit

Postby Baker » Fri Feb 27, 2015 2:02 am

jitspoe wrote:What's the point in switching to 64bit? Are you planning to address more than 2 gigs of memory?

Linux on 64-bit won't run 32-bit binaries.

Windows will. The Mac will. Linux = no. So you have to convert to 64-bit on Linux.
The night is young. How else can I annoy the world before sunsrise? 8) Inquisitive minds want to know ! And if they don't -- well like that ever has stopped me before ..
User avatar
Baker
 
Posts: 3666
Joined: Tue Mar 14, 2006 5:15 am

Re: Porting codebases to 64 bit

Postby Spirit » Fri Feb 27, 2015 5:34 pm

You just need to install 32 bit libraries on Linux.
Improve Quaddicted, send me a pull request: https://github.com/SpiritQuaddicted/Quaddicted-reviews
Spirit
 
Posts: 1028
Joined: Sat Nov 20, 2004 9:00 pm

Re: Porting codebases to 64 bit

Postby Spike » Fri Feb 27, 2015 6:28 pm

for debian/ubuntu:
apt-get install package-name:i386
no need to port, but 64bit users are less likely to have 32bit versions of your dependancies installed, and if you only provide 32bit dependancy lists then they'll need to tweak them (and figure out any omissions too).
you probably still want to as a personal pride sort of thing, but its really just a dependancy issue. there's no reason you can't make a (bash) install script that downloads+installs the appropriate dependancies along with your game.

abuse of textures can result in a need for more than 2gb ram. both dp and fte were modified to make use of windows' large-address-aware thing to allow 32bit versions to access 4gb of ram on 64bit windows (or 3gb on 32bit windows), in order to avoid oom-crashing when used with the ultra version of rygel's texture package.
quite impressive for a 20-year-old game tbh.
Spike
 
Posts: 2883
Joined: Fri Nov 05, 2004 3:12 am
Location: UK

Re: Porting codebases to 64 bit

Postby Baker » Fri Feb 27, 2015 9:23 pm

Spirit wrote:You just need to install 32 bit libraries on Linux.

Didn't know that but if the answer is weird enough a pedestrian won't know, it may as well be. :D

For sure a 32-bit binary for Linux is a true 2nd class citizen because it needs bizarre treatment. Almost as bad as using WINE.

Spike wrote:both dp and fte were modified to make use of windows' large-address-aware thing to allow 32bit versions to access 4gb of ram on 64bit windows
Very interesting.
The night is young. How else can I annoy the world before sunsrise? 8) Inquisitive minds want to know ! And if they don't -- well like that ever has stopped me before ..
User avatar
Baker
 
Posts: 3666
Joined: Tue Mar 14, 2006 5:15 am

Re: Porting codebases to 64 bit

Postby Spirit » Fri Feb 27, 2015 11:19 pm

Baker wrote:For sure a 32-bit binary for Linux is a true 2nd class citizen because it needs bizarre treatment.


Not if you "do the right thing" and get your software packaged by smart maintainers. I don't want to think about the blood and tears that would require though. PS: Archlinux' AUR rocks and makes maintaining/installing packages trivial ;)
Improve Quaddicted, send me a pull request: https://github.com/SpiritQuaddicted/Quaddicted-reviews
Spirit
 
Posts: 1028
Joined: Sat Nov 20, 2004 9:00 pm

Re: Porting codebases to 64 bit

Postby qbism » Sat Feb 28, 2015 4:32 pm

Baker wrote:For sure a 32-bit binary for Linux is a true 2nd class citizen because it needs bizarre treatment. Almost as bad as using WINE.

Sometimes worse than Wine. :lol: A win32 binary might work out-of-the-box on Linux64 when a Linux32 won't.

Besides FTEQW, Hammer of Thyrion might be a good source to look at. Multi-platform build that is closer to a vanilla quake-like engine. I haven't tried the Linux builds but both 32 and 64-bit cross-compiled builds work well on Windows.
User avatar
qbism
 
Posts: 1236
Joined: Thu Nov 04, 2004 5:51 am

Next

Return to Engine Programming

Who is online

Users browsing this forum: No registered users and 1 guest