.lit2 format - request for feedback

[Spike]

aye, it does need something more reliable than merely the surface count.
checksums of the entire header break when a map gets re-vised or has different ents embedded into it, so I don't think that's an ideal solution, but I don't know what to suggest instead.

[mh]

Just to clarify: I don't like the QuakeSpasm method because it's dependent on a human being to not screw up.

For example, say a mapper releases a buggy V1 of a map then a few weeks later a bugfixed V2. They have the same name and go in the same gamedir. Which one does a LIT file belong to?

Maybe rather than just using the surface count, how about embedding some other counts from the original source .bsp into the header? A few other lumps interact with the faces lump, like edges, surfedges, vertexes. Stuff that can survive the hypothetical re-vis or entity embedding. Construct a "magic" from their counts; at the very least the possibility of multiple lumps having the same counts together with the same name should be so low as to be safely ignored.

There may be edge cases I've missed with this, so I'm not religiously welded to the proposal: I just want something more robust is all; how it's achieved isn't too important so long as it's sane.

[ericw]

IMHO, the chance of the filename and numsurfs matching, but the .lit2 still being built against a incompatible (different geometry) version of the map, is small enough to ignore.

I'm just thinking that any geometry change, beyond moving a wall a couple of units, will cause the bsp's numsurfs to change and the .lit2 validation to fail.

That reminds, me I still have to implement that validation in my QuakeSpasm implementation.

[Baker]

I think the biggest risk for that scenario is someone working on a map and recompiling and sharing a map in development possibly forgetting to include the lit2 and someone has the previous lit2 in the right place so the "wrong content protection" doesn't help.

A sanity check like if (bsp->numsurfs == lit2->numsurfs) would prevent that most of the time and the wrong map scenario.

I don't have an opinion on any of this ... except I hope to not see a heavy-weight solution in the event an elegant solution cannot be found.

(For a .lit the lightdata is 3x the size of what is in the bsp, wouldn't this the data be 3 x 4 = 12 times the size.)

[Spike]

you could always validate the extents. if they make no sense (+/- 1) then you know the bsp is wrong. yeah, that's not entirely a serious suggestion, but it would work. reliable, but a hassle.

lit2 has no direct correlation between internal and replacement, on account of surfaces potentially having different counts of valid styles per face. even different surface scales will result in different extents due to rounding (not *4). you can't make trivial assumptions about the size of the contained lighting data withough verifying these things first.

[Baker]

Good to know. numsurfaces it is!! I was with that idea all along, I swear!

Still can't believe ericw did fence texture shadows.

[mankrip]

The 1:1 texel/lexel (light pixel ) ratio will help to speed up software rendering.

Just remember to add an option to create lightmaps for water surfaces. For this, their TEX_SPECIAL flag must be ignored by the compilers (but still included for compatibility with engines that doesn't support lightmapped liquids). This will make big water surfaces to be subdivided, but it's no problem.

[ericw]

mankrip, yes I can add that to the todo list. Do any engines support that, or I guess you're waiting for a light tool to generate the lightmaps
I think It would be independent of .lit2, though.

I've been continuing work polishing the tyrutils and quakespasm patches.

For Quakespasm, I initially forgot to change the types of the coordinates in glRect_t from unsigned char to int. If you increase BLOCK_WIDTH/BLOCK_HEIGHT to 256, glRect_t needs to go to int.

[ericw]

Updated the first post with tyrutils and quakespasm binaries.

[mh]

OK, I've implemented this in a research engine. Some comments.

I'm still strongly in favour of an array-of-structs for the per-surface overrides, and this experience just reinforces it. It wasn't painful to implement, but it was annoying. Don't let hypothetical future modification that may never even happen get in the way of a clean implementation now.

It needs to be documented that the offsets in the .lit2 need to be multiplied by 3, and that values of -1 still need to be caught. This didn't affect the QS sample implementation, but I implemented it slightly differently and was caught by it; so might others.

It's necessary to peek ahead at the surfaces lump and calculate the number of surfaces in the .bsp in advance of loading the .lit2 in order to sanity-check. Again, more annoying than painful.

The .lit2 size for an ID1 map is typically about 12mb (I ran with -lmscale 0.25), so be sure to crank up the -heapsize and MAX_LIGHTMAPS (a relighting of e4m4 used 68 lightmaps at 256x256).

Extents in the .lit2 file are just confusing. I don't get why they can't have the same values (but calculated at the higher precision) as those calculated by CalcSurfaceExtents. I'd suggest that if you're going to include extents at all you should also include texturemins (which is dependent on the same calculation).

The visual quality was roughly comparable to realtime lighting with good shadowmap filtering, and I was quite pleased to see that.

QuakeSpasm obviously hasn't completed the implementation so far as R_LightPoint is concerned. That's OK for a sample implementation but you might get bug reports. I wasn't using LordHavoc's interpolation code here and it looks like you're going to have fun when you get round to doing it.

I got good performance with frametimes ranging from 0.2 to 0.4 milliseconds on an Intel HD 4000. This was with GPU lightmaps however, so I wasn't doing any dynamic light uploads. I've no idea how well (or otherwise) they would run with a more traditional model.

There are a lot of surfaces with little light variation across them, and it would be neat if the light tool could detect this and scale down the lightmap in cases where it happens. I'm not sure how complex that would be.

Crisper edges on shadowed areas did tend to show up stair-steps a bit more, even with -extra4. I'd semi-expected that.

[mh]

Just adding that when the format is finalized, I'll probably hack up coloured .lit2s for the ID1 maps.

[mankrip]

mankrip, yes I can add that to the todo list. Do any engines support that, or I guess you're waiting for a light tool to generate the lightmaps
I think It would be independent of .lit2, though.

Yes, it can be implemented regardless of .lit2 .

I'm not sure if other engines supports that (iIrc there's a thread where Spike mentioned that FTEQW does), but I've partially coded support for it already, and can finish it in less than a couple hours.

[ericw]

@mankrip, cool - I heard hmap2 and darkplaces support lit water as well.

@MH, wow, awesome!

Ok, I agree an array of structs for the "extra face info" would be cleaner.

Pasting what you posted earlier:

Code: Select all

typedef struct litsurf_s
{
    uint offset;
    ushort extents[2];
    byte styles[4];
    byte shift;
} litsurf_t;

One thing, currently it's 13 bytes. We could either pad it to be 16 bytes, or keep it as 13 bytes but do a little extra work when reading/writing to avoid doing unaligned memory access (probably just keep a single litsurf_t on the stack and memcpy to/from it.) Not really sure which I prefer.

Regarding extents, Spike's idea was, if they're included in the lit, the engine can trust the lit values without worrying about floating-point precision.
Alternatively the engine can do its CalcSurfaceExtents calculation, and if the result is off by one from what's in the lit, it can use the lit value.
For reference, this describes the test case where CalcSurfaceExtents will produce an off-by-one result if it's done at 32-bit rather than 80-bit: https://sourceforge.net/p/quakespasm/patches/15/
Thinking about it a bit more... while including the extents in the lit2 is a more proper fix for that problem, I think all engines that do SSE builds should add the casts to double or long double there, if only so they load mfx's map correctly. I'm tempted to just take out extents, to be honest, and rely on engines that need to tweak CalcSurfaceExtents doing so. The light tool could also detect "weird" texture axes which are going to trigger precision problems if the engine does CalcSurfaceExtents with 32bit SSE, and light.exe could print a warning to the mapper.

I tested jam2_scampie.bsp relit with "-lmscale 0.25" on my mid-end laptop (core 2 duo, integrated nvidia 9400m). Performance in Quakespasm was pretty decent, r_speeds frame times were in the range of 0-4ms when no dynamic lights were active, and peaking at 10ms in a room with a styled light + me firing rockets. The worst I got was 50ms when noclipping so the whole map is visible.

I guess maps that use more styled lights could suffer worse performance, but on the other hand, I'm counting on either 1) implementing the light tool optimization that automatically scales down faces with little detail. or 2) mappers just applying it manually to only the faces that need higher resolution. In jam2_scampie.bsp I feel like only a small percent of faces really benefited from the extra resolution.

[Baker]

Dumb question.

What are benefits of this vs. take an upscaling algorithm like Hqx (source code - LGPL) and upsample existing lightmaps in the engine?

[ericw]

Not dumb, but if you take something like this screenshot: http://i.imgur.com/zwuQv4j.jpg
You can imagine a grid of points on the ground at 16 unit intervals; that would be the vanilla lightmap resolution, and the points would be widely spaced enough that they'll miss the fine details of the bars. Regardless of what kind of upscaling you use, you'll never be able to restore the original detail, at best you'll just get a blur.

Some folks in the tyrutils thread on func have been playing with this: http://www.celephais.net/board/view_thr ... &start=377
I do have a bit of a fear that this is a solution in search of a problem, or that it might only be useful for casting shadows of fences, lol.

edit: Also, hqx is a special scaler for upscaling pixel art, my guess is it wouldn't look so great for lightmaps, would be liable to generate strange looking vectorized shapes in the lightmap.