(Multi-) Threading

Discuss programming topics for the various GPL'd game engine sources.
Baker
Posts: 3666
Joined: Tue Mar 14, 2006 5:15 am

Re: (Multi-) Threading

Post by Baker »

revelator wrote:Ah ok had no idea you where developing it for ARM, but tbb works mostly for ARM also now :) and theres ongoing work on it so it's worth considering.
ARM's opencv port allready uses tbb.
A good example of diversity of interests and points of view in this thread. Spike thinks of browser clients, I think of devices, you think of Doom 3 :D
The night is young. How else can I annoy the world before sunsrise? 8) Inquisitive minds want to know ! And if they don't -- well like that ever has stopped me before ..
Spike
Posts: 2914
Joined: Fri Nov 05, 2004 3:12 am
Location: UK
Contact:

Re: (Multi-) Threading

Post by Spike »

Baker wrote:A good example of diversity of interests and points of view in this thread. Spike thinks of browser clients, I think of devices, you think of Doom 3 :D
I only think of browser clients because those are the ones that will give you the most nightmares. In comparison, an Android port is not only much easier to create, but vastly easier to maintain (remember that I say this as someone who does not even have a real android device to test on).

linux+android+mac+presumably ios provide pthreads. there are also libraries for pthreads on windows. pthreads everywhere! you don't really need third-party apis (just libraries that implement pthreads). the only exception is platforms where threads simply do not exist - like in the browser (webworkers are more like separate processes than threads).
reap the benefit of choosing standards instead of implementations!
Baker
Posts: 3666
Joined: Tue Mar 14, 2006 5:15 am

Re: (Multi-) Threading

Post by Baker »

I'm using pthreads http://sourceforge.net/projects/pthreads4w/

And I'm slightly irritated that using the .lib requires using the .dll too. The dll is rather small. I can't think of any good excuse for that :( I have a rather passionate dislike for having to distribute any .dlls with a binary.

I know I dislike debugging on the Mac, XCode isn't anywhere close to as nice as Visual Studio so using Win32 pthreads would minimize debug time in XCode.

In time this will solve itself.

This may eventually be an alternative: http://locklessinc.com/downloads/winpthreads.h
The night is young. How else can I annoy the world before sunsrise? 8) Inquisitive minds want to know ! And if they don't -- well like that ever has stopped me before ..
revelator
Posts: 2621
Joined: Thu Jan 24, 2008 12:04 pm
Location: inside tha debugger

Re: (Multi-) Threading

Post by revelator »

i do have patches for linking to a static pthread library if you want :) they are mostly intended for windows but with a little work they could probably be modified to work for other arch's to.
btw .dll ? i hope you mean .so as unix archs are not compatible with the windows dll arch normally.
Oh and yes i didnt mean doom3 specifically ;) just pointed at it for some hrm pointers into how it could be done hehe.
Productivity is a state of mind.
Baker
Posts: 3666
Joined: Tue Mar 14, 2006 5:15 am

Re: (Multi-) Threading

Post by Baker »

revelator wrote:i do have patches for linking to a static pthread library if you want :) they are mostly intended for windows
Yes I'd be interested in that, I should only need a pthread library on Windows as a general rule of thumb anyway because pthread is posix and non-Windows systems are almost always going to have that available.
The night is young. How else can I annoy the world before sunsrise? 8) Inquisitive minds want to know ! And if they don't -- well like that ever has stopped me before ..
revelator
Posts: 2621
Joined: Thu Jan 24, 2008 12:04 pm
Location: inside tha debugger

Re: (Multi-) Threading

Post by revelator »

http://sourceforge.net/projects/cbadvan ... z/download

patches for the mingw64 winpthreads library, not sure if they also apply to posix systems but i have the patches for the original pthreads library somewhere on my harddrive so ill upload them as soon as i can find them.

to link statically with pthreads use -DWINPTHREAD_STATIC in cflags and or cxxflags. also rename the libpthreads.dll.a library to libpthreads_s.dll.a to avoid autotools trying to link with the shared version anyway.

same goes for the original pthread library when i dug up the patches.
Productivity is a state of mind.
revelator
Posts: 2621
Joined: Thu Jan 24, 2008 12:04 pm
Location: inside tha debugger

Re: (Multi-) Threading

Post by revelator »

original pthreads just need to have -DPTW32_STATIC_LIB in cflags and or cxxflags, static support is allready working without patches.
Productivity is a state of mind.
Baker
Posts: 3666
Joined: Tue Mar 14, 2006 5:15 am

Re: (Multi-) Threading

Post by Baker »

Hehe, yeah. [I did some more reading of docs after you had posted the static lib thing.]

According to the docs, pthreads for win32 has had this maybe 4 years but it doesn't exactly stand out in the docs.

I guess on Unix systems (which seems likely to include the Mac as well) dynamic linking for pthreads is the norm. On Windows, this is highly undesirable since it isn't like Windows is going to have a pthreads dll sitting around.
The night is young. How else can I annoy the world before sunsrise? 8) Inquisitive minds want to know ! And if they don't -- well like that ever has stopped me before ..
revelator
Posts: 2621
Joined: Thu Jan 24, 2008 12:04 pm
Location: inside tha debugger

Re: (Multi-) Threading

Post by revelator »

depends i would say, allthough i seen a rising tendency of the unix guys to prefer dynamic loading because it simplifies linking quite a bit.
Downside is that you end up with having a dependency on a ton of shared objects :( and if really unlucky dll hell.
Productivity is a state of mind.
Baker
Posts: 3666
Joined: Tue Mar 14, 2006 5:15 am

Re: (Multi-) Threading

Post by Baker »

revelator wrote:depends i would say, allthough i seen a rising tendency of the unix guys to prefer dynamic loading because it simplifies linking quite a bit.
Downside is that you end up with having a dependency on a ton of shared objects :( and if really unlucky dll hell.
If you do happen to have a Visual Studio compatible Win32 static lib sitting around, I'd love to have it. [I have Windows MinGW builds for my stuff like always, but in practice I don't do too much with this except review the ofter superior warnings gcc does and correct.]

Apparently I have to recompile myself, an I made an attempt but won't work with Visual C++ 2008 Express (I can't stomach installing a more recent Visual Studio, I dislike the changes in 2012, 2013 and newer and it will wreck all my file associations and botch other things --- I'm not interested in 2 days of hell dealing with the repercussions) and I made a run at CodeBlocks and a command line attempt and failed both times (the command line build likes some compiler I haven't heard of and looks for some header to a dependency I don't know what is).

I forgot how "not fun" it is compile some of these things.
The night is young. How else can I annoy the world before sunsrise? 8) Inquisitive minds want to know ! And if they don't -- well like that ever has stopped me before ..
Spike
Posts: 2914
Joined: Fri Nov 05, 2004 3:12 am
Location: UK
Contact:

Re: (Multi-) Threading

Post by Spike »

personally I use mingw for public builds as a way to ensure the GPL is honoured correctly. they're built from svn, and thus I can ensure that others can build an equivelent build (really just randomized register allocations that might prevent exact builds).
its not strictly needed, but it does guarentee my obligations under the gpl are followed.
when it comes to visual studio, you tend to find that you need to hack everything in order to get stuff to work (like throwing stdint.h into system header locations etc).

dynamic linking is often favoured on linux because it means the system packaging system can provide updates for your dependancies (so eg a buggy libpng is actually fixed without your intervention), it also helps solve the malloc/free mismatch issue when libc is dynamically linked.

static linking is often favoured on windows as a way to avoid the whole msvc runtime installer nightmare (I get 'is not a valid win32 program' ? YOU GAVE ME A VIRUS!). the 'fix' for this forces the malloc/free mismatch issue, and the easy way to solve that is to then just statically link _everything_.
on the other hand, dynamically linking (c api) libraries which correctly provide destructor functions means that calls from different dlls can use the correct TLS entries. This sort of thing can be important when it comes to threads, which is likely why the dll form is favoured enough to require people to compile their own.


dependancies only suck when they're likely to be missing, otherwise dynamically linking to other people's libraries is only ever a good thing.
revelator
Posts: 2621
Joined: Thu Jan 24, 2008 12:04 pm
Location: inside tha debugger

Re: (Multi-) Threading

Post by revelator »

with gcc i often find this to be an annoyance even though it makes sense.
Problem here is that when an update to the respective libraries finally makes it in they are pretty consistently versioned differently,
so forcing me to recompile every damn package that normally depended on that library's dll to avoid trouble.
Autotool based build systems can actually be coerced to replace the library version with the old version number to get around this,
but it causes problems further down the line because sometimes the exports changed a bit so you cannot guarantee that everything works after.

Newer the less its the right way, but its damn annoying :).

Btw you can sometimes get lucky with a mingw built static library by just renaming the libpthreads.a library tp pthreads.lib you can sometimes link directly to it with msvc.
sometime you need to also link to libgcc.a where you can use the same trick by renaming it to gcc.lib, just make sure its from the gcc version that built it.
Debugging will be rather nasty though as the mingw built libraries will have incompatible debugging information (dwarf / sjlj) as compared to msvc's.
Productivity is a state of mind.
Jay Dolan
Posts: 59
Joined: Tue Jan 22, 2008 7:16 pm
Location: Naples, FL
Contact:

Re: (Multi-) Threading

Post by Jay Dolan »

Hm. I've spent a fair amount of time adding threading to Quetoo (previously Quake2World). Let me offer a couple tips based on what I've found:

1) If you're already using libSDL, then you should use SDL_Thread. It provides portable thread creation, non-busy waiting, mutexes, conditions, etc. Very easy to use, and works across the board. Win.

2) BSP recursion (cmodel.c) can be made thread-safe by adding just a few thread-local variables (__thread keyword for GCC and Clang). Basically, there's a global trace state structure that is common to the various tracing functions that are called throughout the recursion. By making that state local to each thread, you're then free to run multiple threads through CM_BoxTrace in parallel. Be careful if your engine has any optimizations that cache things like "current trace number" or "check count" on BSP nodes or leafs, as these will have to be moved into that shared thread-local state bucket in order to be made safe. But by and large, making cmodel.c thread-safe isn't too hard.

3) Parallelizing the Quake engine to any measurable benefit is actually really difficult. This is why tech3 and tech4 use only a pair of threads, and basically run the game in one thread, and draw everything in another. If you parallelize at a more fine-grained level, any benefit is lost on the overhead of coordinating the threads. So the right way to do, as far as I can tell, is to encapsulate an entire renderable scene into a struct, and flip-flop between two instances of that struct at each frame (think of how GL's front and back buffers work). This way, you run a frame of game logic in one thread, populating the scene, and then render the result of that frame asynchronously in the other. Note that doing this safely will require that you literally copy objects into the scene struct. Holding pointers to things that the main thread will modify would be a recipe for crashes ;)

Quetoo doesn't quite work this way as of yet. Instead, I've tried to parallelize scene population itself. So, for example, all particle thinking and entity culling happens in one thread while the main thread recurses the BSP and prepares it for rendering. So far, I've only had minuscule and inconsistent gains using this method. Which is what leads me to believe that 3) above is the way to go.
revelator
Posts: 2621
Joined: Thu Jan 24, 2008 12:04 pm
Location: inside tha debugger

Re: (Multi-) Threading

Post by revelator »

SDL would definatly be number one for consideration :) also nice to see someone who has actually tested things a bit with threaded optimization.
I agree with the whole thread coordination issue, its damn hard to get it running to a point where it makes a damn,
if done right it could be worth it though but the ammount of work included might defer some from trying :).
Productivity is a state of mind.
Baker
Posts: 3666
Joined: Tue Mar 14, 2006 5:15 am

Re: (Multi-) Threading

Post by Baker »

I've sure SDL threads are awesome.

But I occasionally write console apps and other miscellaneous things and SDL isn't a good fit for that, for me personally because of this.

The Win32 pthreads is awesome if it didn't require a DLL would be perfect, but as it stands that creative .h file I linked to earlier in the thread may assist me in platform neutralizing to pthreads. I'll probably live with the pthreads .dll until I get some nice stuff using it.

SDL is getting to be quite nice, and was always great for sound (who wants to learn the ins and outs of sound mixing? :D )
The night is young. How else can I annoy the world before sunsrise? 8) Inquisitive minds want to know ! And if they don't -- well like that ever has stopped me before ..
Post Reply