Putting "Downloading" In a Thread

[Baker]

Ok, as referenced in a prior post, utilizing libcurl to download within the engine in a function often doesn't send "keepalive" messages with the right frequency to keep the client connected to the server.

I want to split this out to be its own thread.

No big deal.

1. Create a barebones download.exe and use CreateProcess(..., ...) [Linux would probably be execlp (..., ...)
2. Pass the information via the "command line"

I can do that easy. Start the process and have it check whether or not the process is completed each frame.

But what is "right way" to do this?

1. I could, with enough experimentation turn the .exe into a .dll ... is there a way that the .dll could provide feedback information to the engine like percent downloaded while it is running?

2. Is there a way within the main engine source to create a threaded process? Google Chrome, for instance, as far as I understand it creates a new thread for each tab.

[daemonicky]

Google Chrome, for instance, as far as I understand it creates a new thread for each tab.

IIUIC they use processes for tabs http://www.hanselman.com/blog/Microsoft ... reads.aspx . They use threads too, but not for each tab. http://blog.marcchung.com/2008/09/05/ch ... ained.html You can check it with process explorer.

It works so it is a solution. Is it slow that you want to change it? Where is this download going on?
I guess the other option is by linking library and using it directly but your solution is nicer because it does not uses linked library.
Anothe one would be to use some engines function ...

[mh]

If the API used has an asynchronous download you can use that. Must check out the Windows stuff I'm using (sometimes using a linked library directly is nicer because it removes an external dependency - how many engines use different - and incompatible - versions of zlib1?)

So kick of your asynchronous download, go into a loop sending keepalive messages and updating the screen (a Sleep (5) would be a good idea here too), then each iteration sniff for a result and act accordingly.

I originally built a console app as proof of concept for my downloader, and I think I still have the source for that, so I can tinker with it without corrupting my main codebase.

[Spike]

wait, thread or process?

if you just want a new thread in windows, call CreateThread with the right arguments in your main thread, set some bit of memory somewhere when its complete/failed, return from thread's entry function, and call CloseHandle() inside the creating thread.
if you just want a new thread in linux, call pcreate_thread to create it, pthread_join in the parent thread to clean it up, otherwise the same as windows.

general rules when using threads is that you can poke whatever you want, just make sure you know that the other thread isn't modifying whatever it is that you're reading at the time. use mutexes or something for that. you shouldn't need any mutexes for downloads if you prepare all the info as an argument before creation, with just a single var written atomically to say when its done. if its aligned, its fine.


if you want an entire new process, you can just invoke the original app with arguments, then you don't need a download.exe.
with linux you can just popen it to get a readable pipe so you can see when its done. windows is more involved.


you can alternatively poll for data using curl, however, your download will be slower in certain (60fps) situations.

[Baker]

I originally built a console app as proof of concept for my downloader, and I think I still have the source for that, so I can tinker with it without corrupting my main codebase.

If you have that lying around, I'd like to see it.

wait, thread or process?

if you just want a new thread in windows, call CreateThread with the right arguments in your main thread, set some bit of memory somewhere when its complete/failed, return from thread's entry function, and call CloseHandle() inside the creating thread.
if you just want a new thread in linux, call pcreate_thread to create it, pthread_join in the parent thread to clean it up, otherwise the same as windows.

general rules when using threads is that you can poke whatever you want, just make sure you know that the other thread isn't modifying whatever it is that you're reading at the time. use mutexes or something for that. you shouldn't need any mutexes for downloads if you prepare all the info as an argument before creation, with just a single var written atomically to say when its done. if its aligned, its fine.


if you want an entire new process, you can just invoke the original app with arguments, then you don't need a download.exe.
with linux you can just popen it to get a readable pipe so you can see when its done. windows is more involved.


you can alternatively poll for data using curl, however, your download will be slower in certain (60fps) situations.

I meant process and typed thread. Multi-threaded applications are way out of my league.

But that information was an interesting read. I didn't think of the idea or recursively calling the engine with a command line param to re-use it. That certainly is an interesting idea, but I'm not sure if I want to do that. Then again, it kills a file dependency. It is definitely an interesting idea.

[Spike]

threads are imho easier, at least simple ones with a 'go' function and a 'done' flag set somewhere. :)
Use only reenterant functions (ie: don't write to globals) and its fairly easy.

[frag.machine]

wait, thread or process?

if you just want a new thread in windows, call CreateThread with the right arguments in your main thread, set some bit of memory somewhere when its complete/failed, return from thread's entry function, and call CloseHandle() inside the creating thread.
if you just want a new thread in linux, call pcreate_thread to create it, pthread_join in the parent thread to clean it up, otherwise the same as windows.

I'd even suggest going pthreads (Posix Threads, supported in Linux and Win-32) all the way to keep the cross platform spirit up.

[Baker]

threads are imho easier, at least simple ones with a 'go' function and a 'done' flag set somewhere.
Use only reenterant functions (ie: don't write to globals) and its fairly easy.

Hmmmm.

[Spike]

using fte's sys_win.c, you could create a download thread with this:

Code: Select all

struct dlthreadargs
{
    char *url;
    int done;
    void *threadhandle;
};

struct dlthreadargs someglobal;
//called when you need a new file
void downloadbegin(char *url)
{
if (someglobal.url)
    return; //already busy
someglobal.url = strdup(url);
someglobal.done = 0;
someglobal.threadhandle = Sys_CreateThread(downloadthread, &someglobal, 0);

//make it look like an error if it failed
if (!someglobal.threadhandle)
    someglobal.done = 2;

while (someglobal.threadhandle && !someglobal.done)
    waitorsomething();

if (someglobal.threadhandle)
    Sys_WaitOnThread(someglobal.threadhandle);
}
//called once per frame
void dlcheck(void)
{
if (!someglobal.done)
    return; /nothing happened yet
//release memory
if (someglobal.threadhandle)
    Sys_WaitOnThread(someglobal.threadhandle);

if (someglobal.done == 1)
    downloadsucceeded(someglobal.url);
else
    downloadfailed(someglobal.url);
//clean up
free(someglobal.url);
memset(&someglobal, 0, sizeof(someglobal));
}


obviously you'll probably want to structure that a bit better, maybe.
poll on the done field once per frame, or whatever
fix up the indents or whatever

the download thread:

Code: Select all

int downloadthread(void *arg)
{
    struct dlthreadargs *a = arg;
    if (dodownload(a->url))
        a->done = 1; //success
    else
        a->done = 2; //error
}


see, easy... *cough*.


The actual glue inside fte's sys_win.c is for abstraction between win32 and posix, rather than any proper magic.

its up to you how you integrate it.
I just personally find threads easier than figuring out some way to pass arguments as something other than a basic pointer.

obviously, you have to avoid calling Con_Printf inside your download thread, you can use mutexes for that if you need it. alternatively you can write some error message field and have the main thread print it when it sees the child stopped. or create a ring buffer of pending prints. or something. if you were using an entirely separate process, you would have to perform some sort of Inter-Process Communication to get the message back into the main thread, system API functions to see if the child is complete, etc.
Threads are easier, if only because you have direct memory access. But you do have to make sure you don't call functions that are not thread safe (sticking only to functions termed as 'reenterant', which is generally considered good coding practice in the first place).

[Baker]

No kidding? Thanks.

using fte's sys_win.c, you could create a download thread with this:

...

see, easy... *cough*.


The actual glue inside fte's sys_win.c is for abstraction between win32 and posix, rather than any proper magic.

.
.
.

Threads are easier, if only because you have direct memory access. But you do have to make sure you don't call functions that are not thread safe (sticking only to functions termed as 'reenterant', which is generally considered good coding practice in the first place).

Noted.