Porting Quake 2's MASM R_PolysetCalcGradients to GAS

Discuss programming topics for the various GPL'd game engine sources.
Post Reply
Posts: 8
Joined: Sun May 31, 2015 9:37 am

Porting Quake 2's MASM R_PolysetCalcGradients to GAS

Post by Maraakate »

I am porting Quake 2's inline assembly code for MSVC to MASM then finally to GAS (for use with MinGW). The specific code in question is for the Skin drawing (R_PolysetCalcGradients for those who want to look it up). The code almost "works" what happens is the skin seems to stretched over the model incorrectly.


A few interesting things I noticed is when I run objdump -dwrC r_polysa.obj > r_polysa.masmand the same for the GAS version the code is mostly similar except for the fact fsubp and fsubrp have been swapped in MASM. Please note not the operands (I already know about this issue in GAS).
In the picture the left side is the GAS version, the right side is the MASM version. The original MASM code (and therefore what I have in GAS) is what should be on the left side. I am unsure why MASM is apparently swapping this or if objdump is reporting it incorrectly. However, if I swap the two this does not fix the issue. It just gets inverted in another wrong way.

I mention this FSUBP/FSUBRP swap because this was a problem in porting the Particle blending inline ASM code. This had one call to use an FSUBRP in the MASM version, objdump reported it now being FSUBP and I had to change it to FSUBP in the GAS version for it to work! I don't understand why this is happening?

In any case, I am new to assembly, but understand some basics and have been doing some reading. Obviously the math here is not quite right, but it seems as though it should be. I don't know how or what to do next. How do I fix and debug this problem?

The code repository to what I am working on is at: https://bitbucket.org/neozeed/q2dos/com ... /win32_asm (specifically the Win32_ASM branch). The files I am working with are gas\r_polysa.s and ref_soft\r_polysa.asm.
Posts: 2621
Joined: Thu Jan 24, 2008 12:04 pm
Location: inside tha debugger

Re: Porting Quake 2's MASM R_PolysetCalcGradients to GAS

Post by revelator »

Try having a look at the gas assembler code for quake1, you might find some answers there since quake 1 assembler was in gas format from the start and actually required conversion by the gas2masm tool for msvc. Some preliminary code for quake2 was also in the original quake 1 code.
Productivity is a state of mind.
Post Reply