Stencil shadow deformation based on lighting direction

[Jay Dolan]

I posted this on SO, but this might actually be the best crowd to run this by. I've calculated an average light direction for my mesh entities, and I'd like to deform their stencil shadows away from the light source. Here's the issue I posted on StackOverflow:

http://stackoverflow.com/questions/2018 ... ght-source

Any insight you guys can share would be much appreciated!

[Spike]

I assume you're after projected shadows and not shadow volumes?

something like http://www.opengl.org/archives/resource ... sld021.htm ? you'll still need to combine that with a model and projection matrix, probably by multiplying that matrix into the (model)view matrix at some point.

z-pass shadow volumes need just the sides, which requires that only some verts are extruded based upon the light position. z-fail needs front+back as well, but if you're already generating sides then its simpler to keep the front+back caps as part of the same mesh/batch. aside from you calling it 'stencil shadows', your matrix stuff seems unrelated...

[Irritant]

Jay, CRX used to have exactly the code you're looking for, where the stencil shadows were stretched/deformed based on light direction, for a very fast, simple method of shadowing. I think that code was in the engine in SVN around 2006-2010, it was pretty efficient and fast, which is what I assume you're going for. Only issue was of course that it didn't project on walls, or get clipped on the edges of brushes, etc, but that was the tradeoff for speed vs true shadow volumes.

[Knightmare]

Only issue was of course that it didn't project on walls, or get clipped on the edges of brushes, etc, but that was the tradeoff for speed vs true shadow volumes.

This sounds like a variation on Quake2's original planar shadows, not stencil shadows.

[jitspoe]

What many engines refer to as "stencil shadows" are the planar shadows, but rendered to the stencil buffer to avoid the ugly zfighting the original shadows had.

[Knightmare]

What you want to reuse is not the actual vertex arrays that are rendered, but the lerped vertices of the model that was just previously rendered. You take the triangles facing the light source from that array, project them an adequate distance away from the light source, forming the volume's sides, and cap the volume with the same triangles again at the end (assuming you're doing more than just a simple z-pass render).

Drawing the backside polys of a model as the endcap instead of a copy of the front faces will result in artifacting on some models from an unclosed volume.

To apply a point or direction vector from an unaltered, game-world space to the rendering matrix space for the model and have it appear the same as it would be in the game world, you have to perform the reverse of what the matrix transformations do. The below code calculates light direction, and reverses rotate and scale for the projection vector. (Note: uses C++ vector class objects instead of Quake's vec3_t float arrays)

Code: Select all

void GL_DrawAliasVolumeShadow (entity_t *ent, dmdl_t *phdr, CVector bbox[8])
{
	CVector		lightVec, temp, vecAdd;
	CVector		shadowVec, endBbox[8], volumeMins, volumeMaxs;
	int			i, lnum;
	float		dist, highest, lowest, alpha, zScale, projectedDistance = 1;
	float		angle, cosp, sinp, cosy, siny, cosr, sinr, ix, iy, iz;
	qboolean	zFail = (gl_shadow_zfail->value != 0);
	qboolean	inVolume = false;
	dlight_t	*dl;

	if (!currentmodel->tri_edges)	// check for missing edge data
		return;

	dl = r_newrefdef.dlights;
	
	vecAdd.Set(680, 320, 1024);	// set base vector

	for (i=0, lnum = 0; i<r_newrefdef.num_dlights; i++, dl++)
	{
		if (dl->origin == ent->origin)
			continue;

		temp = dl->origin - ent->origin;
		dist = dl->intensity - temp.Length();
		if (dist <= 0)
			continue;

		lnum++;
		// Factor in the intensity of a dlight
		temp = temp * (dist * 0.25);
		vecAdd += temp;
	}
	vecAdd.Normalize();
	vecAdd = vecAdd * 1024;

	// get projection distance from lightspot height
	zScale = ent->render_scale.z;
	if (zScale == 0)	zScale = 1.0f;
	highest = lowest = bbox[0].z;
	for (i=0; i<8; i++) {
		if ((bbox[i].z) > highest)	highest = bbox[i].z;
		if ((bbox[i].z) < lowest)	lowest = bbox[i].z;
	}
	projectedDistance = (1.25f * fabs(highest - lightspot.z)/* + fabs(highest-lowest)*/) / fabs(vecAdd.z);
	
	// (omitted z-pass vieworg-in-bbox check)
	.
	.
	.

	projectedDistance /= zScale;

	lightVec = vecAdd;

	// reverse-rotate light vector based on angles
	angle = -ent->angles.pitch / 180 * M_PI;
	cosp = cos(angle); sinp = sin(angle);
	angle = -ent->angles.yaw / 180 * M_PI;
	cosy = cos(angle); siny = sin(angle);
	angle = ent->angles.roll / 180 * M_PI;
	cosr = cos(angle); sinr = sin(angle);

	// rotate for yaw (z axis)
	ix = lightVec.x; iy = lightVec.y;
	lightVec.x = cosy * ix - siny * iy + 0;
	lightVec.y = siny * ix + cosy * iy + 0;

	// rotate for pitch (y axis)
	ix = lightVec.x; iz = lightVec.z;
	lightVec.x = cosp * ix + 0 + sinp * iz;
	lightVec.z = -sinp * ix + 0 + cosp * iz;

	// rotate for roll (x axis)
	iy = lightVec.y; iz = lightVec.z;
	lightVec.y = 0 + cosr * iy - sinr * iz;
	lightVec.z = 0 + sinr * iy + cosr * iz;

As for distorting/scaling the end of the volume, I haven't tried that myself, mainly because there are no singular light sources that can be projected from in the Q2 engine. Thus I end up projecting everything with the same vector, but which is altered by nearby dynamic lights. The results are more than good enough:
http://www.markshan.com/knightmare/pics ... adows8.jpg

[Barnes]

I think that code was in the engine in SVN around 2006-2010

Yes-yes, I even know where it came from Dynamic planar shadow was a good idea at that date... But i think shadow volumes is better

[Knightmare]

It originally came from Quake2Max, right?

BTW, KMQ2 does all 3 variants: static planar, dynamic planar, and volume projection shadows.

[Barnes]

It originally came from Quake2Max, right?

Yess We've improved it thought up Psycho, like static shadow angles and bsp light sources

[leileilol]

I think even Quake3 does this with cg_shadows 2.

[Barnes]

cg_shadows 1 - planar blob shadow
cg_shadows 2 - projective shadow volumes (zpass) shadows hit through the walls
cg_shadows 3 - opacity projective planar shadow

[Jay Dolan]

Hi guys,

Thanks for all of the replies. I finally got this working (the projections, anyway). I'm sorting out a few more bugs, but it looks like this will work out pretty well. I'm using LordHavoc's matrix lib, so here's what the projection looks like:

Code: Select all

/*
 * @brief Projects the model view matrix for the given entity onto the shadow
 * plane. A perspective shear is then applied using the standard planar shadow
 * deformation from SGI's cookbook, adjusted for Quake's negative planes:
 *
 * ftp://ftp.sgi.com/opengl/contrib/blythe/advanced99/notes/node192.html
 */
static void R_RotateForMeshShadow_default(const r_entity_t *e, const r_illumination_t *il) {
	vec4_t pos, normal;
	matrix4x4_t proj, shear;
	vec_t dot;

	if (!e) {
		glPopMatrix();
		return;
	}

	const c_bsp_plane_t *p = &il->shadow.plane;

	// project the entity onto the shadow plane
	vec3_t vx, vy, vz, t;
	Matrix4x4_ToVectors(&e->matrix, vx, vy, vz, t);

	dot = DotProduct(vx, p->normal);
	VectorMA(vx, -dot, p->normal, vx);

	dot = DotProduct(vy, p->normal);
	VectorMA(vy, -dot, p->normal, vy);

	dot = DotProduct(vz, p->normal);
	VectorMA(vz, -dot, p->normal, vz);

	dot = DotProduct(t, p->normal) - p->dist;
	VectorMA(t, -dot, p->normal, t);

	Matrix4x4_FromVectors(&proj, vx, vy, vz, t);

	glPushMatrix();

	glMultMatrixf((GLfloat *) proj.m);

	// transform the light position and shadow plane into model space
	Matrix4x4_Transform(&e->inverse_matrix, il->pos, pos);
	pos[3] = 1.0;

	const vec_t *n = p->normal;
	Matrix4x4_TransformPositivePlane(&e->inverse_matrix, n[0], n[1], n[2], p->dist, normal);

	// calculate shearing, accounting for Quake's positive plane equation
	normal[3] = -normal[3];
	dot = DotProduct(pos, normal) + pos[3] * normal[3];

	shear.m[0][0] = dot - pos[0] * normal[0];
	shear.m[1][0] = 0.0 - pos[0] * normal[1];
	shear.m[2][0] = 0.0 - pos[0] * normal[2];
	shear.m[3][0] = 0.0 - pos[0] * normal[3];
	shear.m[0][1] = 0.0 - pos[1] * normal[0];
	shear.m[1][1] = dot - pos[1] * normal[1];
	shear.m[2][1] = 0.0 - pos[1] * normal[2];
	shear.m[3][1] = 0.0 - pos[1] * normal[3];
	shear.m[0][2] = 0.0 - pos[2] * normal[0];
	shear.m[1][2] = 0.0 - pos[2] * normal[1];
	shear.m[2][2] = dot - pos[2] * normal[2];
	shear.m[3][2] = 0.0 - pos[2] * normal[3];
	shear.m[0][3] = 0.0 - pos[3] * normal[0];
	shear.m[1][3] = 0.0 - pos[3] * normal[1];
	shear.m[2][3] = 0.0 - pos[3] * normal[2];
	shear.m[3][3] = dot - pos[3] * normal[3];

	glMultMatrixf((GLfloat *) shear.m);
}

I'm using a stencil test to clip the shadows to world geometry. So when drawing the world, for each opaque surface, I use write unique(-ish) stencil value based on the surface plane using the stencil test with GL_ALWAYS. Then, when rendering shadows, I enable the stencil test again, with GL_EQUAL. The result us that shadows are clipped nicely to the plane they intersect. The challenge is to gather all of the planes that a shadow hits.. I'll probably make some attempt at this soon.

As for light sources, I've been incorporating BSP light sources into my lighting model for a while now, so that's not anything new. But it's cool to now have perspective-correct shadows for each light source that hits a mesh. Here are some shots (WIP):

[Barnes]

Very nice, but i'm don't understand one thing - it's not shadow volumes, right?

[Jay Dolan]

That's right. Just planar projections, clipped to surfaces. It works like this:

When drawing opaque world surfaces, GL_STENCIL_TEST is enabled with GL_ALWAYS and the following stencil function applied for each surface:

Code: Select all

glStencilFunc(GL_ALWAYS, R_STENCIL_REF(surf->plane), ~0);

Where R_STENCIL_REF is as follows:

Code: Select all

/*
 * @brief Yields a unique-enough stencil value for surfaces in order to perform
 * a meaningful stencil buffer write. This value can be used in a stencil test
 * to clip decals and shadows to a surface's geometry.
 */
#define R_STENCIL_REF(plane) (((plane)->type + (uint32_t) (plane)->dist) % 0xff)

Then, when drawing shadows, GL_STENCIL_TEST is re-enabled with GL_EQUAL, and the same stencil function:

Code: Select all

glStencilFunc(GL_EQUAL, R_STENCIL_REF(&illumination->shadow.plane), ~0);

So that shadows only show up on surfaces sharing a plane with the trace that yielded the shadow. The use of GL_POLYGON_OFFSET_FILL lets the shadows show over the world geometry while still honoring the stencil test.

There are a couple big limitations with this approach:

1) No self-shadowing.
2) Meshes don't shadow other meshes.
3) Gathering all of the planes that a shadow should hit is infeasible for corner cases (pun intended).

But, for a Quake2 engine, they don't look half bad

On a related note, is it possible to acquire a 12 or 16 bit stencil buffer? The limits of an 8 bit stencil buffer and my R_STENCIL_REF macro mean that it's possible for my shadows to appear where they should not. Rare, but possible. A 16 bit stencil buffer would solve this, but I don't think that's a thing. Oh well.

[Spike]

gl3+ method (according to docs. seems to fail on nvidia):
glGetFramebufferAttachmentParameteriv(GL_FRAMEBUFFER, GL_STENCIL, GL_FRAMEBUFFER_ATTACHMENT_STENCIL_SIZE, &someinteger);
deprecated method (which fails on a core context):
glGetIntegerv(GL_STENCIL_BITS, &someinteger);

with d3d10+, microsoft document:
DXGI_FORMAT_D32_FLOAT_S8X24_UINT
DXGI_FORMAT_D32_FLOAT
DXGI_FORMAT_D24_UNORM_S8_UINT
DXGI_FORMAT_D16_UNORM
which means they don't consider that hardware could support any sorts of depthstencil configuration with stencil bits other than 0 or 8bit.
of course, there's nothing stopping you from asking for more than 8 bits from gl. you probably won't get it though.