I am porting Quake 2's inline assembly code for MSVC to MASM then finally to GAS (for use with MinGW). The specific code in question is for the Skin drawing (R_PolysetCalcGradients for those who want to look it up). The code almost "works" what happens is the skin seems to stretched over the model incorrectly.
A few interesting things I noticed is when I run objdump -dwrC r_polysa.obj > r_polysa.masm
and the same for the GAS version the code is mostly similar except for the fact fsubp
have been swapped in MASM. Please note not the operands (I already know about this issue in GAS).
In the picture the left side is the GAS version, the right side is the MASM version. The original MASM code (and therefore what I have in GAS) is what should be on the left side. I am unsure why MASM is apparently swapping this or if objdump is reporting it incorrectly. However, if I swap the two this does not fix the issue. It just gets inverted in another wrong way.
I mention this FSUBP/FSUBRP swap because this was a problem in porting the Particle blending inline ASM code. This had one call to use an FSUBRP in the MASM version, objdump reported it now being FSUBP and I had to change it to FSUBP in the GAS version for it to work! I don't understand why this is happening?
In any case, I am new to assembly, but understand some basics and have been doing some reading. Obviously the math here is not quite right, but it seems as though it should be. I don't know how or what to do next. How do I fix and debug this problem?
The code repository to what I am working on is at: https://bitbucket.org/neozeed/q2dos/com ... /win32_asm
(specifically the Win32_ASM branch). The files I am working with are gas\r_polysa.s