Doom 3 engine release and game code

[mh]

ES2 is not the same as regular GL. There is some feature overlap, but there are some things in ES2 that are not in desktop GL, and a veritable boatload in desktop GL that are not in ES2.

Been doing some playing around with the vertexcache.cpp code recently; there is a major performance bottleneck in here because it uses the original GL_ARB_vertex_buffer_object extension, which is absolutely useless on it's own when it comes to dynamic data. The main problem is that it causes constant CPU/GPU synchronization, which breaks parallelism and means that each processor needs to constantly go idle while waiting for the other to catch up. Added some basic GL_ARB_map_buffer_range support which is a huge performance boost (this change alone got my experimental Q2 engine going twice as fast), but I fear that a full exploitation of the potential would need a complete gutting of this code.

[motorsep]

Do you have it as a patch for vanilla Doom 3 to boost performance? What does it break?

[mh]

The current codebase I have is unusable with vanilla - I'm using GLEW for extensions so the "qgl" stuff is all gone, I've removed fallback paths, etc. I could throw my vertexcache.cpp up on pastebin, but someone else would need to do the grunt work of adapting it to vanilla; I no longer have any interest in mucking around with the pain and suffering caused by retaining downlevel support. It's also heavily untested, by the way - most I've done is a 10 min jog through the first part of Delta 2.

[motorsep]

Could you make a patch to speed up vanilla Doom 3 with ARB path?

[revelator]

i Guess it depends on opengl ES support if even possible.
I know glsl is newer than the ARB path does not mean though that all development stopped on it the day they got glsl its still being updated and newer features still going in.

[revelator]

hmm had a look at mh's code for the vertexcache only thing i noticed was

Code: Select all

   
    // use this as a pointer to the actual object ? mh im looking at you.
    if ( this->dynamicAllocThisFrame + size > this->frameBytes )
    {
        // if we don't have enough room in the temp block, allocate a static block,
        // but immediately free it so it will get freed at the next frame
        this->tempOverflow = true;
        this->Alloc( data, size, &block );


The comment above it was directed at mh

i did digged in a little myself and changed the freeing of the vbo in the #if 0 #endif block to the more correct

Code: Select all

      if (block->vbo)
      {
         // do it as it should be done delete the vertex buffer object.
         // no fancy tricks needed and it works fine.
         qglDeleteBuffersARB(1, &block->vbo);
      }
      else if ( block->virtMem )
        {
            Mem_Free( block->virtMem );
            block->virtMem = NULL;
        }


As stated it works just fine though uncommented the previous code seemed to do no harm either but the comment was that it was unnessesary since it would be reused pretty fast anyway.
But trying to remove the whole block from if (block->vbo) { to } and changing the elseif to if crashes doom3 :/ so instead i came up with the above to free the vbo correctly.

[mh]

Not really what I meant. Here's one of the changes:

Code: Select all

   if (block->vbo)
   {
      glBindBuffer (GL_ARRAY_BUFFER, block->vbo);

      if (GLEW_ARB_map_buffer_range)
      {
         void *dst = NULL;

         if ((dst = glMapBufferRange (GL_ARRAY_BUFFER, block->offset, (GLsizeiptr) size, BUFFER_MAP_BITS)) != NULL)
         {
            SIMDProcessor->Memcpy ((byte *) dst, data, size);
            glUnmapBuffer (GL_ARRAY_BUFFER);

            return block;
         }
      }

      glBufferSubData (GL_ARRAY_BUFFER, block->offset, (GLsizeiptr) size, data);
      return block;
   }

   SIMDProcessor->Memcpy ((byte *) block->virtMem + block->offset, data, size);

   return block;

The big problem with Doom 3's VBO code is that allocating and updating/respecifying or destroying VBOs on a regular basis at runtime is one of the worst possible things you can do for performance. A second problem is that the standard ARB_vertex_buffer_object calls for updating buffers are a collective crock of shit, and represent another of the worst possible things you can do for performance. It also generates thousands of VBOs so it's spending a huge amount of time in glBindBuffer calls (which are quite slow).

The Q2 case where I doubled performance by using glMapBufferRange instead of glBufferSubData is an optimal best-case - everything is stored in a single large buffer and the usage pattern is the equivalent of D3D's discard/no-overwrite pattern - that's been proven to work and work fast for well over a decade. Until GL_ARB_map_buffer_range was made available it was just not possible to use this pattern in GL. But you're not going to get anywhere near a doubling of performance in Doom 3 with it because of other huge problems in Doom 3's VBOs code and because of other bottlenecks elsewhere in it's renderer.

All of this is why naive ports to VBOs normally end up running slower.

id can hardly be blamed for this because they just used the best they had at the time, but the fact that this is no longer the best (and is in fact a long way from it) should be acknowledged.

So the summary is that a real patch of the VBO code to get real increased performance requires a total gut-and-rewrite. The MapBufferRange code I gave an example of above resolves one of the problems, but there are still two others that it doesn't resolve, and because it hasn't been extensively tested there may well be cases where it breaks.

So no - I won't release any kind of patch for vanilla including this code, and there's no point in even bothering to ask.

[revelator]

Was more out of curiosity but thanks for explaning.

[mh]

Improved FPS Counter.

This is the FPS counter I wrote for DirectQ; it's much more accurate than the one built into Doom 3 and has been verified as consistent with FRAPS. If you're going to measure performance you may as well do it right

Code: Select all

float SCR_DrawFPS (float y)
{
   static int frames = 0;
   static int starttime = 0;
   static bool first = true;
   static float fps = 0.0f;

   int t = Sys_Milliseconds ();

   if (first)
   {
      starttime = t;
      first = false;
      return y + BIGCHAR_HEIGHT + 4;
   }

   frames++;

   if (t - starttime > 100 && frames > 10)
   {
      fps = (float) (frames * 1000) / (float) (t - starttime);
      starttime = t;
      frames = 0;
   }

   char *s = va ("%ifps", (int) (fps + 0.5f));
   int w = strlen (s) * BIGCHAR_WIDTH;

   renderSystem->DrawBigStringExt (635 - w, idMath::FtoiFast (y) + 2, s, colorWhite, true, localConsole.charSetShader);

   return y + BIGCHAR_HEIGHT + 4;
}

[revelator]

small bug warning.

in idRenderSystemLocal::Init

the call to set the backend renderer screws up selecting the right backend here as the engine does not have all the data yet and if you opted to remove the old backends it will crash doom3.

heres the new function.

Code: Select all

void idRenderSystemLocal::Init( void )
{
    // clear all our internal state
    viewCount = 1;      // so cleared structures never match viewCount
    // we used to memset tr, but now that it is a class, we can't, so
    // there may be other state we need to reset

    ambientLightVector[0] = 0.5f;
    ambientLightVector[1] = 0.5f - 0.385f;
    ambientLightVector[2] = 0.8925f;
    ambientLightVector[3] = 1.0f;

    memset( &backEnd, 0, sizeof( backEnd ) );

    R_InitCvars();

    R_InitCommands();

    guiModel = new idGuiModel;
    guiModel->Clear();

    demoGuiModel = new idGuiModel;
    demoGuiModel->Clear();

    R_InitTriSurfData();

    globalImages->Init();

    idCinematic::InitCinematic( );

    // build brightness translation tables
    R_SetColorMappings();

    R_InitMaterials();

    renderModelManager->Init();

    // set the identity space
    identitySpace.modelMatrix[0*4+0] = 1.0f;
    identitySpace.modelMatrix[1*4+1] = 1.0f;
    identitySpace.modelMatrix[2*4+2] = 1.0f;
}


Now works fine with just the ARB/glsl backend from MH.
This one is also in dhewm3 port.

[mh]

So it was just the calls to common->Printf that caused it?

[revelator]

Hmm no it was this part

// determine which back end we will use
// ??? this is invalid here as there is not enough information to set it up correctly
SetBackEndRenderer();

ID even marked it as faulty but left it in anyway xD its called in the right place somewhere else also though,
but removing the old deprecated arb 1.0 backend causes it to get confused because the nessesary data for telling the engine if the backend was enabled was not yet parsed,
so we end up with an undefined situation.

Major bummer newer leave things you know dont work even if you fixed the problem elsewhere hehe.

[revelator]

Original code to compare

Code: Select all

void idRenderSystemLocal::Init (void)
{

   common->Printf ("------- Initializing renderSystem --------\n");

   // clear all our internal state
   viewCount = 1;      // so cleared structures never match viewCount
   // we used to memset tr, but now that it is a class, we can't, so
   // there may be other state we need to reset

   ambientLightVector[0] = 0.5f;
   ambientLightVector[1] = 0.5f - 0.385f;
   ambientLightVector[2] = 0.8925f;
   ambientLightVector[3] = 1.0f;

   memset (&backEnd, 0, sizeof (backEnd));

   R_InitCvars ();

   R_InitCommands ();

   guiModel = new idGuiModel;
   guiModel->Clear ();

   demoGuiModel = new idGuiModel;
   demoGuiModel->Clear ();

   R_InitTriSurfData ();

   globalImages->Init ();

   idCinematic::InitCinematic ();

   // build brightness translation tables
   R_SetColorMappings ();

   R_InitMaterials ();

   renderModelManager->Init ();

   // set the identity space
   identitySpace.modelMatrix[0 * 4 + 0] = 1.0f;
   identitySpace.modelMatrix[1 * 4 + 1] = 1.0f;
   identitySpace.modelMatrix[2 * 4 + 2] = 1.0f;

   // determine which back end we will use
   // ??? this is invalid here as there is not enough information to set it up correctly
   SetBackEndRenderer ();

   common->Printf ("renderSystem initialized.\n");
   common->Printf ("--------------------------------------\n");
}

[mh]

Ah, OK. I'd already killed that myself anyway (as well as some of the variables it sets - like overbright level), but it's still useful to know.

[revelator]

Indeed Also a few other things that need a lookover according to PVS-Studio (fixed most allready but theres still a few im not certain of like the multitude of calls that are newer tried against nullptr.
Ive ditched parralax shading since it looks dreadfull with highres textures on non flat surfaces instead im going to look into tesselation.
Parralax shaders could be used though for raising certain textures but it would need to be changed to a content shader with a big fat warning sign to mappers that its not intended for non flat surfaces.