Doom 3 engine release and game code

[revelator]

Took a little digging to get the correct register transforms but heres the correct version.

Code: Select all

void R_LocalPointToGlobal (const float modelMatrix[16], const idVec3 &in, idVec3 &out)
{
   __m128 m0, m1, m2, m3;
   __m128 in0, in1, in2;
   float i0, i1, i2;

   i0 = in[0];
   i1 = in[1];
   i2 = in[2];

   m0 = _mm_loadu_ps (&modelMatrix[0]);
   m1 = _mm_loadu_ps (&modelMatrix[4]);
   m2 = _mm_loadu_ps (&modelMatrix[8]);
   m3 = _mm_loadu_ps (&modelMatrix[12]);

   in0 = _mm_load1_ps (&i0);
   in1 = _mm_load1_ps (&i1);
   in2 = _mm_load1_ps (&i2);

   m0 = _mm_mul_ps (m0, in0);
   m1 = _mm_mul_ps (m1, in1);
   m2 = _mm_mul_ps (m2, in2);

   m0 = _mm_add_ps (m0, m1);
   m0 = _mm_add_ps (m0, m2);
   m0 = _mm_add_ps (m0, m3);

   _mm_store_ss (&out[0], m0);
   m1 = _mm_cvtepi32_ps(_mm_shuffle_epi32(_mm_cvttps_epi32(m0), 0x55));
   _mm_store_ss (&out[1], m1);
   m2 = _mm_movehl_ps (m2, m0);
   _mm_store_ss (&out[2], m2);
}

Tested and works like a charm minor speedup even about 5 fps.

[revelator]

Cleaned up the source even more and removed resources for the editors.
Prepared the source for using GLEW but the wgl extension code needs a heavy rewrite to work with it.
Working on an update for the outdated jpeg and openal code (possibly removing eax).

[revelator]

jpeg8 code done now for openal.

[revelator]

Maybe someone else might like to update it so im going to show parts of the process here.

in image_files.cpp at the top replace the old bunch by this.

Code: Select all

#define JPEG_INTERNALS
extern "C" {
#include "jpeg-8c/jpeglib.h"

    // hooks from jpeg lib to our system
   static void jpg_Error(j_common_ptr cinfo)
   {
      char buffer[JMSG_LENGTH_MAX];

      (*cinfo->err->format_message) (cinfo, buffer);

      /* Let the memory manager delete any temp files before we die */
      jpeg_destroy(cinfo);

      common->FatalError("%s", buffer);
   }

   static void jpg_Printf(j_common_ptr cinfo)
   {
      char buffer[JMSG_LENGTH_MAX];

      /* Create the message */
      (*cinfo->err->format_message) (cinfo, buffer);

      /* Send it to stderr, adding a newline */
      common->Printf("%s\n", buffer);
   }
}


and replace LoadJPG with this

Code: Select all

static void LoadJPG(const char *filename, unsigned char **pic, int *width, int *height, ID_TIME_T *timestamp)
{
    /* This struct contains the JPEG decompression parameters and pointers to
     * working space (which is allocated as needed by the JPEG library).
     */
    struct jpeg_decompress_struct cinfo = {NULL};
    /* We use our private extension JPEG error handler.
     * Note that this struct must live as long as the main JPEG parameter
     * struct, to avoid dangling-pointer problems.
     */
    /* This struct represents a JPEG error handler.  It is declared separately
     * because applications often want to supply a specialized error handler
     * (see the second half of this file for an example).  But here we just
     * take the easy way out and use the standard error handler, which will
     * print a message on stderr and call exit() if compression fails.
     * Note that this struct must live as long as the main JPEG parameter
     * struct, to avoid dangling-pointer problems.
     */
    struct jpeg_error_mgr jerr;

    /* More stuff */
    JSAMPARRAY      buffer;      /* Output row buffer */
    unsigned int   row_stride;   /* physical row width in output buffer */
    unsigned int   pixelcount, memcount;
    unsigned int   sindex, dindex;
    byte         *out;
    int            len;
    union
    {
        byte *b;
        void *v;
    } fbuffer;
    byte         *buf;

    if (!pic)
    {
        fileSystem->ReadFile(filename, NULL, timestamp);
        return;   // just getting timestamp
    }
    *pic = NULL;

    /* In this example we want to open the input file before doing anything else,
     * so that the setjmp() error recovery below can assume the file is open.
     * VERY IMPORTANT: use "b" option to fopen() if you are on a machine that
     * requires it in order to read binary files.
     */
    len = fileSystem->ReadFile (filename, &fbuffer.v, timestamp);
    if (!fbuffer.b || len < 0)
    {
        return;
    }

    /* Step 1: allocate and initialize JPEG decompression object */

    /* We have to set up the error handler first, in case the initialization
     * step fails.  (Unlikely, but it could happen if you are out of memory.)
     * This routine fills in the contents of struct jerr, and returns jerr's
     * address which we place into the link field in cinfo.
     */
    cinfo.err = jpeg_std_error(&jerr);
    cinfo.err->error_exit = jpg_Error;
    cinfo.err->output_message = jpg_Printf;

    /* Now we can initialize the JPEG decompression object. */
    jpeg_create_decompress(&cinfo);

    /* Step 2: specify data source (eg, a file) */
    jpeg_mem_src(&cinfo, fbuffer.b, len);

    /* Step 3: read file parameters with jpeg_read_header() */
    (void) jpeg_read_header(&cinfo, TRUE);
    /* We can ignore the return value from jpeg_read_header since
     *   (a) suspension is not possible with the stdio data source, and
     *   (b) we passed TRUE to reject a tables-only JPEG file as an error.
     * See libjpeg.doc for more info.
     */

    /* Step 4: set parameters for decompression */
    /*
     * Make sure it always converts images to RGB color space. This will
     * automatically convert 8-bit greyscale images to RGB as well.
     */
    cinfo.out_color_space = JCS_RGB;

    /* Step 5: Start decompressor */
    (void) jpeg_start_decompress(&cinfo);
    /* We can ignore the return value since suspension is not possible
     * with the stdio data source.
     */

    /* We may need to do some setup of our own at this point before reading
     * the data.  After jpeg_start_decompress() we have the correct scaled
     * output image dimensions available, as well as the output colormap
     * if we asked for color quantization.
     * In this example, we need to make an output work buffer of the right size.
     */
    /* JSAMPLEs per row in output buffer */
    pixelcount = cinfo.output_width * cinfo.output_height;

    if(!cinfo.output_width || !cinfo.output_height || ((pixelcount * 4) / cinfo.output_width) / 4 != cinfo.output_height || pixelcount > 0x1FFFFFFF || cinfo.output_components != 4)
    {
        // Free the memory to make sure we don't leak memory
        fileSystem->FreeFile (fbuffer.v);

        jpeg_destroy_decompress(&cinfo);

        common->FatalError("LoadJPG: %s has an invalid image format: %dx%d*4=%d, components: %d", filename, cinfo.output_width, cinfo.output_height, pixelcount * 4, cinfo.output_components);
    }
    memcount = pixelcount * 4;
    row_stride = cinfo.output_width * cinfo.output_components;

    out = (byte *) R_StaticAlloc(memcount);

    *width = cinfo.output_width;
    *height = cinfo.output_height;

    /* Step 6: while (scan lines remain to be read) */
    /*           jpeg_read_scanlines(...); */

    /* Here we use the library's state variable cinfo.output_scanline as the
     * loop counter, so that we don't have to keep track ourselves.
     */
    while (cinfo.output_scanline < cinfo.output_height)
    {
        /* jpeg_read_scanlines expects an array of pointers to scanlines.
         * Here the array is only one element long, but you could ask for
         * more than one scanline at a time if that's more convenient.
         */
        buf = ((out + (row_stride * cinfo.output_scanline)));
        buffer = &buf;
        (void) jpeg_read_scanlines(&cinfo, buffer, 1);
    }
    buf = out;

    // Expand from RGB to RGBA
    sindex = pixelcount * cinfo.output_components;
    dindex = memcount;

    do
    {
        buf[--dindex] = 255;
        buf[--dindex] = buf[--sindex];
        buf[--dindex] = buf[--sindex];
        buf[--dindex] = buf[--sindex];
    } while(sindex);

    *pic = out;

    /* Step 7: Finish decompression */
    jpeg_finish_decompress(&cinfo);
    /* We can ignore the return value since suspension is not possible
     * with the stdio data source.
     */

    /* Step 8: Release JPEG decompression object */
    /* This is an important step since it will release a good deal of memory. */
    jpeg_destroy_decompress(&cinfo);

    /* After finish_decompress, we can close the input file.
     * Here we postpone it until after no more JPEG errors are possible,
     * so as to simplify the setjmp error logic above.  (Actually, I don't
     * think that jpeg_destroy can do an error exit, but why assume anything...)
     */
    fileSystem->FreeFile (fbuffer.v);

    /* At this point you may want to check to see whether any corrupt-data
     * warnings occurred (test whether jerr.pub.num_warnings is nonzero).
     */

    /* And we're done! */
}

the old function used a hack to get around jpegs old bug with alpha chan pics, the new one can actually do them correctly.

ok so much for the image parts.
Jpeg is also used in the cinematic renderer and that code needs a few modifications to work with version 8.

again at the top replace the old bunch with this.

Code: Select all

#define JPEG_INTERNALS
extern "C" {
#include "jpeg-8c/jpeglib.h"
}

now everywhere in cinematic.cpp where you find a METHODDEF boolean or some other type replace that with
METHODDEF(boolean or whatever was there).

example

METHODDEF(boolean) fill_input_buffer(j_decompress_ptr cinfo)

old one looked like this

METHODDEF boolean fill_input_buffer(j_decompress_ptr cinfo)

same method with GLOBAL boolean void or whatever use
GLOBAL(boolean)

example

GLOBAL(void) jpeg_memory_src(j_decompress_ptr cinfo, byte *infile, int size)

old one looked like this

GLOBAL void jpeg_memory_src(j_decompress_ptr cinfo, byte *infile, int size)

and so on

thats it for Doom3 itself, to make it a bit easier on yourself get ioquake3 and snatch the jpeg-8c folder from it.
Place it in the same folder the old jpeg6 folder was in and add it to the project (remember to remove the old jpeg6 folder from the project).
Delete the old jpeg6 folder completely.

Now compile and if all is well you wont get any warnings or errors try converting some noticable textures to jpg and load a map where you know they will show.

[revelator]

Mo mojo for the image loaders

yank this in above _TargaHeader

#define TGA_MAXCOLORS 16384

/* Definitions for image types. */
#define TGA_Null 0 /* no image data */
#define TGA_Map 1 /* Uncompressed, color-mapped images. */
#define TGA_RGB 2 /* Uncompressed, RGB images. */
#define TGA_Mono 3 /* Uncompressed, black and white images. */
#define TGA_RLEMap 9 /* Runlength encoded color-mapped images. */
#define TGA_RLERGB 10 /* Runlength encoded RGB images. */
#define TGA_RLEMono 11 /* Compressed, black and white images. */
#define TGA_CompMap 32 /* Compressed color-mapped data, using Huffman, Delta, and runlength encoding. */
#define TGA_CompMap4 33 /* Compressed color-mapped data, using Huffman, Delta, and runlength encoding. 4-pass quadtree-type process. */

/* Definitions for interleave flag. */
#define TGA_IL_None 0 /* non-interleaved. */
#define TGA_IL_Two 1 /* two-way (even/odd) interleaving */
#define TGA_IL_Four 2 /* four way interleaving */
#define TGA_IL_Reserved 3 /* reserved */

/* Definitions for origin flag */
#define TGA_O_UPPER 0 /* Origin in lower left-hand corner. */
#define TGA_O_LOWER 1 /* Origin in upper left-hand corner. */

now for the cool stuff

TGA2 version for Doom3.

Code: Select all

/*
=============
LoadTGA
=============
*/
static void LoadTGA(const char *name, byte **pic, int *width, int *height, ID_TIME_T *timestamp)
{
   int         w, h, x, y, realrow, truerow, baserow, i, temp1, temp2, fileSize, pixel_size, map_idx;
   int         RLE_count, RLE_flag, size, interleave, origin;
   bool      mapped, rlencoded;
   byte      *data, *dst, r, g, b, a, j, k, l, *ColorMap;
   byte      *buf_p;
   byte      *buffer;
   TargaHeader   header;

   if (!pic)
   {
      fileSystem->ReadFile(name, NULL, timestamp);
      return;   // just getting timestamp
   }
   *pic = NULL;

   //
   // load the file
   //
   fileSize = fileSystem->ReadFile(name, (void **)&buffer, timestamp);

   if (!buffer)
   {
      return;
   }
   buf_p = buffer;

   header.id_length = *buf_p++;
   header.colormap_type = *buf_p++;
   header.image_type = *buf_p++;

   header.colormap_index = LittleShort(*(short *)buf_p);
   buf_p += 2;
   header.colormap_length = LittleShort(*(short *)buf_p);
   buf_p += 2;
   header.colormap_size = *buf_p++;
   header.x_origin = LittleShort(*(short *)buf_p);
   buf_p += 2;
   header.y_origin = LittleShort(*(short *)buf_p);
   buf_p += 2;
   header.width = LittleShort(*(short *)buf_p);
   buf_p += 2;
   header.height = LittleShort(*(short *)buf_p);
   buf_p += 2;
   header.pixel_size = *buf_p++;
   header.attributes = *buf_p++;

   if (header.id_length != 0)
      buf_p += header.id_length;

   /* validate TGA type */
   switch (header.image_type)
   {
   case TGA_Map:
   case TGA_RGB:
   case TGA_Mono:
   case TGA_RLEMap:
   case TGA_RLERGB:
   case TGA_RLEMono:
      break;

   default:
      common->Printf ("%s : Only type 1 (map), 2 (RGB), 3 (mono), 9 (RLEmap), 10 (RLERGB), 11 (RLEmono) TGA images supported\n", name);
      return;
   }

   /* validate color depth */
   switch (header.pixel_size)
   {
   case 8:
   case 15:
   case 16:
   case 24:
   case 32:
      break;

   default:
      common->Printf ("%s : Only 8, 15, 16, 24 or 32 bit images (with colormaps) supported\n", name);
      return;
   }
   r = g = b = a = l = 0;

   /* if required, read the color map information. */
   ColorMap = NULL;
   mapped = (header.image_type == TGA_Map || header.image_type == TGA_RLEMap) && header.colormap_type == 1;

   if (mapped)
   {
      /* validate colormap size */
      switch (header.colormap_size)
      {
      case 8:
      case 15:
      case 16:
      case 32:
      case 24:
         break;

      default:
         common->Printf ("%s : Only 8, 15, 16, 24 or 32 bit colormaps supported\n", name);
         return;
      }
      temp1 = header.colormap_index;
      temp2 = header.colormap_length;

      if ((temp1 + temp2 + 1) >= TGA_MAXCOLORS)
      {
         return;
      }
      ColorMap = (byte *) R_StaticAlloc (TGA_MAXCOLORS * 4);

      map_idx = 0;

      for (i = temp1 ; i < temp1 + temp2 ; ++i, map_idx += 4)
      {
         /* read appropriate number of bytes, break into rgb & put in map. */
         switch (header.colormap_size)
         {
         case 8:   /* grey scale, read and triplicate. */
            r = g = b = *buf_p++;
            a = 255;
            break;

         case 15:   /* 5 bits each of red green and blue. */
            /* watch byte order. */
            j = *buf_p++;
            k = *buf_p++;
            l = ((unsigned int) k << 8) + j;
            r = (byte) ( ((k & 0x7C) >> 2) << 3 );
            g = (byte) ( (((k & 0x03) << 3) + ((j & 0xE0) >> 5)) << 3 );
            b = (byte) ( (j & 0x1F) << 3 );
            a = 255;
            break;

         case 16:   /* 5 bits each of red green and blue, 1 alpha bit. */
            /* watch byte order. */
            j = *buf_p++;
            k = *buf_p++;
            l = ((unsigned int) k << 8) + j;
            r = (byte) ( ((k & 0x7C) >> 2) << 3 );
            g = (byte) ( (((k & 0x03) << 3) + ((j & 0xE0) >> 5)) << 3 );
            b = (byte) ( (j & 0x1F) << 3 );
            a = (k & 0x80) ? 255 : 0;
            break;

         case 24:   /* 8 bits each of blue, green and red. */
            b = *buf_p++;
            g = *buf_p++;
            r = *buf_p++;
            a = 255;
            l = 0;
            break;

         case 32:   /* 8 bits each of blue, green, red and alpha. */
            b = *buf_p++;
            g = *buf_p++;
            r = *buf_p++;
            a = *buf_p++;
            l = 0;
            break;
         }
         ColorMap[map_idx+0] = r;
         ColorMap[map_idx+1] = g;
         ColorMap[map_idx+2] = b;
         ColorMap[map_idx+3] = a;
      }
   }

   /* check run-length encoding. */
   rlencoded = (header.image_type == TGA_RLEMap || header.image_type == TGA_RLERGB || header.image_type == TGA_RLEMono);
   RLE_count = RLE_flag = 0;

   w = header.width;
   h = header.height;

   if (width)
   {
      *width = w;
   }

   if (height)
   {
      *height = h;
   }
   size = w * h * 4;
   data = (byte *) R_StaticAlloc (size);
   *pic = data;

   /* read the Targa file body and convert to portable format. */
   pixel_size = header.pixel_size;
   origin = (header.attributes & 0x20) >> 5;
   interleave = (header.attributes & 0xC0) >> 6;
   truerow = 0;
   baserow = 0;

   for (y=0 ; y<h ; y++)
   {
      realrow = truerow;

      if (origin == TGA_O_UPPER)
      {
         realrow = h - realrow - 1;
      }
      dst = data + realrow * w * 4;

      for (x=0 ; x<w ; x++)
      {
         /* check if run length encoded. */
         if (rlencoded)
         {
            if (!RLE_count)
            {
               /* have to restart run. */
               i = *buf_p++;
               RLE_flag = (i & 0x80);

               if (!RLE_flag)   // stream of unencoded pixels
               {
                  RLE_count = i + 1;
               }
               else      // single pixel replicated
               {
                  RLE_count = i - 127;
               }

               /* decrement count & get pixel. */
               --RLE_count;
            }
            else
            {
               /* have already read count & (at least) first pixel. */
               --RLE_count;

               if (RLE_flag)
               {
                  /* replicated pixels. */
                  goto PixEncode;
               }
            }
         }

         /* read appropriate number of bytes, break into RGB. */
         switch (pixel_size)
         {
         case 8:   /* grey scale, read and triplicate. */
            r = g = b = l = *buf_p++;
            a = 255;
            break;

         case 15:   /* 5 bits each of red green and blue. */
            /* watch byte order. */
            j = *buf_p++;
            k = *buf_p++;
            l = ((unsigned int) k << 8) + j;
            r = (byte) ( ((k & 0x7C) >> 2) << 3 );
            g = (byte) ( (((k & 0x03) << 3) + ((j & 0xE0) >> 5)) << 3 );
            b = (byte) ( (j & 0x1F) << 3 );
            a = 255;
            break;

         case 16:   /* 5 bits each of red green and blue, 1 alpha bit. */
            /* watch byte order. */
            j = *buf_p++;
            k = *buf_p++;
            l = ((unsigned int) k << 8) + j;
            r = (byte) ( ((k & 0x7C) >> 2) << 3 );
            g = (byte) ( (((k & 0x03) << 3) + ((j & 0xE0) >> 5)) << 3 );
            b = (byte) ( (j & 0x1F) << 3 );
            a = (k & 0x80) ? 255 : 0;
            break;

         case 24:   /* 8 bits each of blue, green and red. */
            b = *buf_p++;
            g = *buf_p++;
            r = *buf_p++;
            a = 255;
            l = 0;
            break;

         case 32:   /* 8 bits each of blue, green, red and alpha. */
            b = *buf_p++;
            g = *buf_p++;
            r = *buf_p++;
            a = *buf_p++;
            l = 0;
            break;

         default:
            common->Printf ("%s : Illegal pixel_size '%d'\n", name, pixel_size);
            R_StaticFree (data);
            if (mapped)
            {
               R_StaticFree (ColorMap);
            }
            return;
         }

PixEncode:
         if (mapped)
         {
            map_idx = l * 4;
            *dst++ = ColorMap[map_idx+0];
            *dst++ = ColorMap[map_idx+1];
            *dst++ = ColorMap[map_idx+2];
            *dst++ = ColorMap[map_idx+3];
         }
         else
         {
            *dst++ = r;
            *dst++ = g;
            *dst++ = b;
            *dst++ = a;
         }
      }

      if (interleave == TGA_IL_Four)
      {
         truerow += 4;
      }
      else if (interleave == TGA_IL_Two)
      {
         truerow += 2;
      }
      else
      {
         truerow++;
      }

      if (truerow >= h)
      {
         truerow = ++baserow;
      }
   }

   if (mapped)
   {
      R_StaticFree (ColorMap);
   }
    fileSystem->FreeFile(buffer);
}

It works brilliantly as far as i could test it the image flip code at the end is unnessesary now as the TGA2 code handles that allready.
Just replace the old function allready

[revelator]

check this out

[revelator]

Mh did you have a version with GLEW ? i got most of it working but theres a few places where i need some help.
And have anyone with an ATI card tried out the latest version yet ? im starting to wonder what the hell is causing this as even the old ARB2 backend version now runs like crap on some ATI cards.
Im starting to think its compiler related (yeah sounds far fetched i know but hear me out) i have a really weird feeling msvc is pulling some optimizations that are native to my arch but
dont work on others. I cheked against dhewm3 which besides a few differences in thread handling and SDL is allmost identical to my current code and dhewm3 works fine on ATI if i compile
it via mingw.

[revelator]

With a little help from mh i have replaced the old Qgl interface with GLEW and its fully working.

OpenAL / image handlers / and a few functions have been updated also.

Source is ready as a base for engine modders, if its of interrest ill upload it.

[revelator]

http://realm.googlecode.com/files/doom3-cleansrc-29-10-2012.7z

Here ya go

[revelator]

Before i end this engine completely ... is someone able to write a glsl parralax shader compatible with mh's glsl backend ? i need one for trying out some stuff in the backend but im a nutjob with shaders
took me 2 days to fix 2 lines last time i had to do one

[nbohr1more]

Sorry if this is a self-indulgent request but here goes...

Here is The Dark Mod 1.08 source code:

http://www.thedarkmod.com/sources/theda ... .08.src.7z

Feel free to look at changes to cache and BSP branching optimizations (etc).

If I may request that some of Mh's fixes (VBO, memcopy, etc) be merged in so we can evaluate the improvements, I would be extremely grateful.

This would also give you a better metric to test these improvements, just sayin'

[revelator]

Sure though atm im only using mh's memcpy for matrix operations (does a way better job there than the SIMD or standard version).
Not home for a couple of days, helping a friend move but as soon as i get back ill give it a go.

[nbohr1more]

Thanks!

[revelator]

Not yet done also trying out a patch that adds multiprocessor support to doom3 via openmp. Its said to give a 15 fps + boost so might be nice

[nbohr1more]

Thanks again. I look forward to the results.