Targeting 64-bit

[Baker]

Based on examination of the Quore engine (a modified Qrack Linux engine which has a 64-bit Linux compile) and some looking over Quakespasm, I know for certain that the following files need worked over.

host_cmd.c
pr_cmds.c
pr_edict.c
pr_exec.c
progs.h
sv_main.c

I'm no expert of 64-bit, but my understanding is that the pointers for the progs interpreter use signed int and this is why the progs interpreter is particularly problematic going from 32 bit to 64 bit. (Then again, I could be all wrong).

Either way I've implemented many of the Quore/Quakespasm changes.

I'm burying this info here for future use and/or maybe discussion. I likely won't be acting on any of it at a minimum of a couple of days but more than likely a couple of weeks. But I want to document the changes that are required.

[mh]

The alias model stuff needs working over too as it stores data as offsets from the start of the header. The only real reason for this is to allow the model to be moved from hunk memory to cache memory, and to allow it to be moved around in cache memory if required. It might be simpler to just get rid of cache memory and store pointers directly. The downside is that MDLs would now need to be reloaded and converted to compatible commands each map load, but it's not 1996 any more and that shouldn't pose too much of a problem for a reasonably modern PC.

Rearchitecting cache memory to avoid the crappiness is another option that gets you the best of both but it's obviously a lot more work.

The third viable option is to say "to hell with native 64-bit support". A 64-bit CPU with a 64-bit OS on it should be able to run a 32-bit app. Of course, sooner or later 32-bit is going to go the way of 16-bit and the problem will need to be dealt with then.

[revelator]

for mingw64 you can replace the offending ints by intptr_t not sure if anything similar exists for msvc.

but as mh said atm its not a big concern as pretty much all 64 bit OS loads 32 bit quite fine.

then again its not a bad idea to try and fix 64 bit compilation allready now since some day it will come in handy. if not in quake then in whatever else you will be working on at that time.

[szo]

.... following files need worked over.

host_cmd.c
pr_cmds.c
pr_edict.c
pr_exec.c
progs.h
sv_main.c

Assuming you're targeting windows, you also have to deal with socket handle handling: unlinke unix world, socket() and accept() doesn't return signed int file descriptors but socket handles which are 32 bit in win32 but 64 bit on win64, not to mention that they are unsigned as opposed to bsd socket api. See net_sys.h & co in quakespasm or uhexen2.

There also places where int (or long) used with the assumption of sizeof(int)==sizeof(ptr*) and those you need dealing with too.

The alias model stuff needs working over too as it stores data as offsets from the start of the header. The only real reason for this is to allow the model to be moved from hunk memory to cache memory, and to allow it to be moved around in cache memory if required.

Some pointer to the problematic places in the code would be appreciated. Haven't had a problem with alias models so far (at least haven't noticed.)

The third viable option is to say "to hell with native 64-bit support". A 64-bit CPU with a 64-bit OS on it should be able to run a 32-bit app.

Not all 64 bit cpus support 32 bit mode, y'know..

[Spike]

from what I remember, there are not too many issues blocking it.
mostly its just warnings.
however...

there's a bug in the sprite code, from what I remember.
and then there's the big one.

qc strings are indexes into the string table. logically a 32bit index is just fine. the catch is that quake is EVIL and gives offsets into other bits of memory, namely the sv.client[X].name strings, as well as any builtins that return a temp string.
the 'simple' solution is to just allocate those string buffers in some block on the hunk, after the progs is loaded(names persist, so you probably want to have two copies of it).
However, frik_file's strzone needs some sort of dynamic memory.
To be honest, if frik_file uses the Z_Malloc functions to allocate memory, you end up with a signed pointer to the hunk. negative strings tend to be in the dynamic zone at the start of the hunk (so dynamic<0, 0<=stringtable<tablesize, entlump>=tablesize, tempstrings>=tablesize,netnames>=tablesize).
If those strings are arranged to always only refer to a heap object, and your heap does not exceed 2gb in size, there's no unsolvable issue.
Alternatively you can do it the fte/dp way, and use the upper bits as a string classification ident. more forwards thinking, but more changes up front.

if you're targetting windows only, I wouldn't bother. sure win64 builds are a nice thing to have... but geez, what's the point? noone knows what sort of windows they have. they don't understand 32bit or 64bit systems, they just know they're running windows! If you're lucky, they might know that its called 'XP'...
And even if you do get a 64bit build, msvc doesn't have the full feature set of the 32bit build, namely lacking edit-and-continue, which is basically the sole feature that puts msvc above any other ide, which is basically the only reason I boot windows instead of linux (linux gets higher fps anyway).
Oh, and two binaries build for different archetectures might see different versions of the registry.

You should generally not need to use intptr_t (the win32 equivelent is named DWORD_PTR, by the way). In most situations, a size_t should suffice. intptr_t is really only useful if you want to cast directly from a pointer to an int, which shouldn't really be done anyway.
A size_t should be fine for the alias model code (typically its a uintptr_t anyway), and is sized correctly for any ptr-ptr result anyway.

[mh]

The third viable option is to say "to hell with native 64-bit support". A 64-bit CPU with a 64-bit OS on it should be able to run a 32-bit app.

Not all 64 bit cpus support 32 bit mode, y'know..

Ultimately you can't work on the basis that an int is 32-bits, or even that a byte is 8-bits either - even the language standards don't define sizes for most types. Yet this assumption is all over Quake, so the question becomes one of "where do you draw the line?"

Spike is correct about the VM, by the way. Let's take a reasonably common example:

Code: Select all

G_INT (OFS_RETURN) = pr_string_temp - pr_strings;

This code is not 64-bit safe. pr_string_temp could be at any address in memory, pr_strings could be at any address, and the VM doesn't dictate that they must fall within any boundaries. Unless you set up your allocations so that you have absolutely everything within the same 2 GB region (stack, heap, hunk, the whole lot) you're eventually going to encounter a problem. (It also assumes that a float is the same size as an int.) One consequence of this is that you can't use malloc for anything that's going to interact with pr_strings. Another consequence is that you can't use a global (not a QC "global", a C global).

So code like this:

Code: Select all

#define STRINGTEMP_BUFFERS 16
#define STRINGTEMP_LENGTH 1024
static char pr_string_temp[STRINGTEMP_BUFFERS][STRINGTEMP_LENGTH];
static byte pr_string_tempindex = 0;

It may work fine for a day, a week, a year even. But sometime...

[Baker]

if you're targetting windows only, I wouldn't bother. sure win64 builds are a nice thing to have... but geez, what's the point? noone knows what sort of windows they have. they don't understand 32bit or 64bit systems, they just know they're running windows! If you're lucky, they might know that its called 'XP'...
And even if you do get a 64bit build

I'm not too worried about Windows at the moment, but I installed Ubuntu on a 64-bit machine and couldn't use any 32 bit Linux engine (only the 64-bit version of Quakespasm ran for me, of the various NQ single player type engines out there) so I'm just a bit worried and concerned about particularly multi-platform stuff mostly at the moment.

Still, collecting info on the 64-bit topic seems useful.

The alias model stuff needs working over too as it stores data as offsets from the start of the header.

I'm pretty ignorant about this stuff, but Quakespasm 64-bit seems to run fine on Ubuntu 64 and the source code changes didn't have anything involving that in there. Neither did Quore.
Thanks for info on MSVC not having edit and continue for 64-bit at the moment.

Other comment: I wish the very weird way the progs interpreter references different chunks of memory wasn't so weird and compounding that, macros obfuscate what is going on. I'm sure in time I'll pick up on what is up with that, but it seems a bit unusual.

[Spike]

but I installed Ubuntu on a 64-bit machine and couldn't use any 32 bit Linux engine

install the 32bit versions of the various library packages

the alias-model code is not really an issue. if you want to be 'correct', change those offsets to a size_t. if you don't do anything, well its all on the heap anyway so none of it will break until you get really large models that wouldn't work on 32bit builds anyway.

linux typically uses pbrk to get space for the heap. this generally means that your heap is allocated in a similar region to the bss/data segments. because of this, many a 64bit linux port did not realise that there were any issues at all with the progs strings. this is not true with all malloc implementations, of course, so it'll likely break on many a system - just that ubuntu with default packages will be unlikely to give any noticable problems if string_t is signed.
However, due to malloc replacement libraries, just because a binary works on your ubuntu, it doesn't mean it'll work on all ubuntu's with the same cpu.
If all strings are signed/on the heap, you never have any problems until your heap is larger than 2gb.

in all seriousness, compile it and see. :)

[daemonicky]

What about using portable types from stdint.h http://linux.die.net/man/3/int32_t instead of non-portable int?

[Spike]

because it makes no real difference.
quake already expects an int to be 32bit, and uses longs sparingly if at all. you could change the entire engine to use those instead, but why bother. and on weird non-8bit-byte machines, you have extra problems, and it'll never work on 16bit cpus, so you don't really gain anything that isn't provided by a friendly c compiler anyway.
the only useful type in there is intptr_t/uintptr_t, but as I already mentioned, needing to use intptr is generally considered bad practise in the first place - you should generally use a union instead. You should generally use size_t/ssize_t in most places where intptr would otherwise be used.

stdint may be useful for portability on embedded/non-mainstream platforms where ints are 16bit (read: early versions of wince), or on 64bit machines with slow support for 32bit ints (due to needing to truncate or whatever) where an int is defined to be 64bit... but really, machines like that need special machine-specific ports anyway.
If you do use int32, you really only need it for file formats. you tend to not want to use it elsewhere, as int32 may be slower than using the native int type.

besides, int32 and friends are C99, which means extra work and checks and OHNOESTHEPRECOMPILERISGOINGTOEATME, especially if you're going to retain support for msvc.

if you assume int is 32bit, you'll have support for all the mainstream platforms, even protected-mode dos apps. in many ways, that's more portable than stdint.h - either way, you've no real way to test it. :)

if you don't use longs, and don't cast between pointer and integer, and use size_t for pointer differences, then 64bit is easy (until it comes to the legacy windows API that uses longs all over the place and assumes long is both 32bit and the size of a pointer). 64bit linux is _EASY_! :)

valgrind is an awesome tool, by the way.

[revelator]

valgrind is an awesome tool, by the way.

cheers to that

if only there was a native win32 non cygbiatch version

ill see if i can port it someday hehe.

besides that great info here

[daemonicky]

quake already expects an int to be 32bit, and uses longs sparingly if at all. you could change the entire engine to use those instead, but why bother. and on weird non-8bit-byte machines

if you don't use longs, and don't cast between pointer and integer, and use size_t for pointer differences, then 64bit is easy (until it comes to the legacy windows API that uses longs all over the place and assumes long is both 32bit and the size of a pointer). 64bit linux is _EASY_!

If you do use int32, you really only need it for file formats. you tend to not want to use it elsewhere, as int32 may be slower than using the native int type.

Thanks.

besides, int32 and friends are C99, which means extra work and checks and OHNOESTHEPRECOMPILERISGOINGTOEATME, especially if you're going to retain support for msvc.

msvc does not support C99?

What causes the problem? Or what is the symptom? What does not works? I thought it is that integers dont fit ...

I personally would transfer Quake codebase into Python and what would be slow I would let in C. Otherwise I could use some higher level concepts like maps, filters, objects ... Quake is done in C and in C if you are making for example list (queue, front, stack) of something you are always reimplementing some of list operations (like searching, removing, inserting ...) ... I think it has been already transfered to Python or some other languages ...

[revelator]

msvc does not support C99?

it does but its not the most strict in that regard i guess the ms folks came to the same conclusion in regards to whats needed for things to run and the 100% correct way of doing things.

[Spike]

msvc does not support C99?

it does but its not the most strict in that regard

That's like saying windows 95 runs apps designed for windows 7. 90% of them fail because they depend on some extra library or dll export (missing headers/functions, eg, no snprintf), 5% of them fail outright because they're 64bit (like defining locals mid-block), and the remaining 5% only work because someone went through the thing being careful to check which system they're on and using replacement functions they wrote themselves to do it (#ifdef _MSC_VER all over the place).

you can use c99 features in msvc if you're willing to write them yourself, but even then its only the api/libs, and not the language itself.

anyway, that's all offtopic.
you can do ifdefs before including stdint and using your own typedefs for msvc - have fun when you use libraries that do the same and the duplicate typedefs that ensue.
I still say its kinda pointless. :P

[mh]

Aside from // comments (which are supported anyway) the biggest gain from C99 for me would be being able to declare a variable anywhere (there are still some iffinesses in switch blocks but it's good enough otherwise). That's totally productivity-wise; it's a pain in the butt to need to add a new variable (say for a loop counter) and have to be jumping back and forth between the start of a function and somewhere in the middle.

No, it's not a deal-breaker. Yes, it can be worked around. But yes, it is a total pain in the butt. Let's stop pretending that tedious crap from the 1970s is somehow superior.

That's why I'd prefer the C-like subset of C++ over C any day. C is rubbish for productivity on this count alone. The big things are a nuisance but it's the little things that grate the most.