porting super8

[qbism]

Just a thought... will the windows version run in Wine?

[goldenboy]

Probably, but also likely at a performance cost. Grab some LiveDVD and just try it.

[qbism]

Thanks, all, for taking a shot at it and the suggestions. Since Flash compiles, and Windows compiles w/ mingw, I thought at least Linux would be straightforward.
If it's an overrun that Windows and Flash can absorb, but not other OSes, perhaps it will eventually be exposed.
torrenting that Linux Minty now...

[szo]

If it's an overrun that Windows and Flash can absorb, but not other OSes, [...]

No, it is *not* this or that or OS can absorb it: It is the way that the particular compiler you used allocated the stack buffers and aligned them, i.e. the overrun is still there somewhere. Take sv_main.c::localmodels brokenness of fitzquake-0.85 as an example: fitz himself never detected it because his version of msvc behaved too lax. Compile it using mingw's gcc-3.4.x and it will still behave, but compile it using mingw gcc-4.2 (IIRC) and you will get an immediate segfault under windows too.

[Baker]

Take sv_main.c::localmodels brokenness of fitzquake-0.85 as an example: fitz himself never detected it because his version of msvc behaved too lax. Compile it using mingw's gcc-3.4.x and it will still behave, but compile it using mingw gcc-4.2 (IIRC) and you will get an immediate segfault under windows too.

Hmmm ...

"MAX_MODELS from 256 to 2048"

Code: Select all

char   localmodels[MAX_MODELS][5];         // inline model names for pre cache


Code: Select all

   for (i=0 ; i<MAX_MODELS ; i++)
      sprintf (localmodels[i], "*%i", i);


i > 999 won't be null terminated "*1000" = 5 characters

Is that what you are talking about?

[Baker]

Compile it using mingw's gcc-3.4.x and it will still behave, but compile it using mingw gcc-4.2 (IIRC) and you will get an immediate segfault under windows too.

Thanks for the high quality explanation for the behavior differences. I was trying to think through why Windows and Flash would work and OS X wouldn't even with the same problem.

Thanks, all, for taking a shot at it and the suggestions. Since Flash compiles, and Windows compiles w/ mingw, I thought at least Linux would be straightforward.
If it's an overrun that Windows and Flash can absorb, but not other OSes, perhaps it will eventually be exposed.
torrenting that Linux Minty now...

I guess szo is just saying that it is "random luck" how a compiler organizes the binary and writing out-of-bounds to buffer in one scenario, the way the compiler organized the binary the memory that gets overwritten may be something non-critical but with another compiler or even another compiler option, with the exactly same code that same relative area of memory if overwritten might be instantly crashy crashy.

(In ProQuake maybe 3 years ago, I made a version with a severely stupid string handling mistake and it didn't crash. A couple of players randomly had oddities with how the console was displayed. I didn't do too much in that particular revision, used WinMerge and checked the 4 changed files. Nothing. Nothing. Nothing. WTH? Relooked carefully, nothing, nothing ... "oh shit".)

[szo]

"MAX_MODELS from 256 to 2048"

Code: Select all

char   localmodels[MAX_MODELS][5];         // inline model names for pre cache


Code: Select all

   for (i=0 ; i<MAX_MODELS ; i++)
      sprintf (localmodels[i], "*%i", i);


i > 999 won't be null terminated "*1000" = 5 characters

Is that what you are talking about?

Yes, exactly.

Compile it using mingw's gcc-3.4.x and it will still behave, but compile it using mingw gcc-4.2 (IIRC) and you will get an immediate segfault under windows too.

Thanks for the high quality explanation for the behavior differences. I was trying to think through why Windows and Flash would work and OS X wouldn't even with the same problem.

My pleasure

Thanks, all, for taking a shot at it and the suggestions. Since Flash compiles, and Windows compiles w/ mingw, I thought at least Linux would be straightforward.
If it's an overrun that Windows and Flash can absorb, but not other OSes, perhaps it will eventually be exposed.
torrenting that Linux Minty now...

I guess szo is just saying that it is "random luck" how a compiler organizes the binary and writing out-of-bounds to buffer in one scenario, the way the compiler organized the binary the memory that gets overwritten may be something non-critical but with another compiler or even another compiler option, with the exactly same code that same relative area of memory if overwritten might be instantly crashy crashy.

Exactly

(In ProQuake maybe 3 years ago, I made a version with a severely stupid string handling mistake and it didn't crash. A couple of players randomly had oddities with how the console was displayed. I didn't do too much in that particular revision, used WinMerge and checked the 4 changed files. Nothing. Nothing. Nothing. WTH? Relooked carefully, nothing, nothing ... "oh shit".)

Oh, having lived through many similars of them I know such stories all too well

[mh]

Take sv_main.c::localmodels brokenness of fitzquake-0.85 as an example: fitz himself never detected it because his version of msvc behaved too lax. Compile it using mingw's gcc-3.4.x and it will still behave, but compile it using mingw gcc-4.2 (IIRC) and you will get an immediate segfault under windows too.

Hmmm ...

"MAX_MODELS from 256 to 2048"

Code: Select all

char   localmodels[MAX_MODELS][5];         // inline model names for pre cache


Code: Select all

   for (i=0 ; i<MAX_MODELS ; i++)
      sprintf (localmodels[i], "*%i", i);


i > 999 won't be null terminated "*1000" = 5 characters

Is that what you are talking about?

It's not that it won't be NULL termed, it's that sprintf will try to NULL term it and will overflow the buffer in the process of doing so. This array should actually exist as one contiguous memory block so the overflow will write into the next slot each time until we get to the last one, which scribbles over just about anything. Depending on what that anything is, the consequences may or may not be fatal.

Of course, once the engine survives this (assuming that it does) it probably won't even be noticed 99% of the time as maps with more than 1000 models are few enough. Once you do get such a map however you'll likely explode.

[szo]

[...] It's not that it won't be NULL termed, it's that sprintf will try to NULL term it and will overflow the buffer in the process of doing so. This array should actually exist as one contiguous memory block so the overflow will write into the next slot each time until we get to the last one, which scribbles over just about anything. Depending on what that anything is, the consequences may or may not be fatal.

Of course, once the engine survives this (assuming that it does) it probably won't even be noticed 99% of the time as maps with more than 1000 models are few enough. Once you do get such a map however you'll likely explode.

The problem with sprintf() writing a nul terminator is most destructive for the very last [MAX_MODELS-1] member of the array, because if the compiler really makes each member 5 bytes, then the last write of 0 will step on who knows what other data and it segfaults immediately without any need to load a map at all.

[mh]

[...] It's not that it won't be NULL termed, it's that sprintf will try to NULL term it and will overflow the buffer in the process of doing so. This array should actually exist as one contiguous memory block so the overflow will write into the next slot each time until we get to the last one, which scribbles over just about anything. Depending on what that anything is, the consequences may or may not be fatal.

Of course, once the engine survives this (assuming that it does) it probably won't even be noticed 99% of the time as maps with more than 1000 models are few enough. Once you do get such a map however you'll likely explode.

The problem with sprintf() writing a nul terminator is most destructive for the very last [MAX_MODELS-1] member of the array, because if the compiler really makes each member 5 bytes, then the last write of 0 will step on who knows what other data and it segfaults immediately without any need to load a map at all.

Things could get really interesting with a read of localmodels[1000] or higher too, even in cases where that last write is somehow survived (as it is in Fitz) as each string in the array won't be terminated.

[qbism]

Compile it using mingw's gcc-3.4.x and it will still behave, but compile it using mingw gcc-4.2 (IIRC) and you will get an immediate segfault under windows too.

I used 32-bit mingw gcc 4.6.1 for the Windows build and cygwin llvm-gcc 4.0.1 for Flash.
I can understand the idea of chance affecting overrun behavior... once I discovered an overrun that would appear with certain compiler options but not others. Eventually found the code mistake, and after that the options were not a problem. So I'd guess compiler options can affect how the "cards" are stacked as well as compiler version, etc.

I'm looking forward to installing Minty, haven't run Linux in a while. Also might learn some things by cracking open QuakeForge 0.5.99 Beta 1 source... glad to hear it's compiled with mingw! And FYI, software renderer with SDL.

[Baker]

Code: Select all

char   localmodels[MAX_MODELS][5];         // inline model names for pre cache


Code: Select all

   for (i=0 ; i<MAX_MODELS ; i++)
      sprintf (localmodels[i], "*%i", i);


i > 999 won't be null terminated "*1000" = 5 characters

Is that what you are talking about?

It's not that it won't be NULL termed, it's that sprintf will try to NULL term it and will overflow the buffer in the process of doing so.

Actually, it works out that effectively the strings for > 999 won't be null terminated (except for the very last one). Correct me if I am wrong.

sub model 1000: Writes 6 characters to slot 1000. String is null terminated in location 0 for string 1001 since only 5 bytes exist for 6 byte sprintf.
sub model 1001: Writes 6 characters to slot 1001. This wipes away the null terminator of string 1000.

So you end up with a giant sub model string like "*1000*1001*1002* .... *1199" and then a null terminator. So all the sub model strings >=1000 except the last one being abnormally large strings.

I know what I said and posted wasn't very obvious, but technically that is what effectively occurs. I know my post actually looked "wrong" and yet that is really what is going on. I just didn't provide enough info so someone (maybe other than szo) could see what I was thinking.


----- Nevermind, I didn't read your above post before posting this. I ended up needlessly typing in this explanation, but it already got discussed.

[mh]

Actually, it works out that effectively the strings for > 999 won't be null terminated (except for the very last one). Correct me if I am wrong.

sub model 1000: Writes 6 characters to slot 1000. String is null terminated in location 0 for string 1001 since only 5 bytes exist for 6 byte sprintf.
sub model 1001: Writes 6 characters to slot 1001. This wipes away the null terminator of string 1000.

So you end up with a giant sub model string like "*1000*1001*1002* .... *1199" and then a null terminator. So all the sub model strings >=1000 except the last one being abnormally large strings.

I know what I said and posted wasn't very obvious, but technically that is what effectively occurs. I know my post actually looked "wrong" and yet that is really what is going on. I just didn't provide enough info so someone (maybe other than szo) could see what I was thinking.


----- Nevermind, I didn't read your above post before posting this. I ended up needlessly typing in this explanation, but it already got discussed.

That's cool, you've explained it pretty well.

So there are actually two problems here. The first is the one szo pointed out where it may or may not segfault on writing the last model, depending on compiler and compiler options. That affects every map (it actually happens during startup), irrespective of how many models are in the map, and is critical to fix.

The second occurs when trying to read back the name of a model with an index of 1000 or higher, where you get the really really big string. Assuming that the engine survives the first one, that is. And that really really big string isn't actually NULL terminated either, because the '\0' for the last model isn't in the array - it was originally written to somewhere outside of the array bounds. You might get lucky and there might still be a '\0' in whatever memory is at the end of the array, but more likely you're going to read beyond it until you do eventually encounter a '\0' somewhere (if you get really lucky you'll crash and actually be able to find the bug).

[qbism]

Thanks for explanations of the sub model seg fault. Will correct that in next build. And maybe other extended limit engines could use a check..

I've started trying to compile under Linux, trying to adapt TyrQuake sys_linux.c, etc., as a starting point. This is the most current SW Quake Linux engine I could find. Ran into a snag with X DGA libraries... lots of comments in headers stating "depreciated" and compile errors like "Pixmap undefined" within the DGA library headers themselves. Anyone know what are the latest-and-greatest libraries to use?

Quake-based engine (Hexen II) including software engine (even transparent water and entities) and lots of ports: http://uhexen2.sourceforge.net/ might be a source of knowledge.

[szo]

Thanks for explanations of the sub model seg fault. Will correct that in next build. And maybe other extended limit engines could use a check..

I've started trying to compile under Linux, trying to adapt TyrQuake sys_linux.c, etc., as a starting point. This is the most current SW Quake Linux engine I could find. Ran into a snag with X DGA libraries... lots of comments in headers stating "depreciated" and compile errors like "Pixmap undefined" within the DGA library headers themselves. Anyone know what are the latest-and-greatest libraries to use?

Here's some base for you as an easy start: http://uhexen2.sourceforge.net/tmp/qbis ... sdl.tar.gz
This I quickly made using your 12-04-11 source plus a few changes (see files with Dec.12 date). There is no udp networking or sound support in it, either. Runs fine for me using pak0.pak+pak1.pak only. Better yet, I can't make it to segfault under water (maybe I correctly adapted your changes? who knows.) Anyways, good luck.

Quake-based engine (Hexen II) including software engine (even transparent water and entities) and lots of ports: http://uhexen2.sourceforge.net/ might be a source of knowledge.

That's my project , I hope you find something useful for you in there.