sizeof Mystery

[Baker]

typedef struct _TargaHeader {
unsigned char id_length, colormap_type, image_type;
unsigned short colormap_index, colormap_length;
unsigned char colormap_size;
unsigned short x_origin, y_origin, width, height;
unsigned char pixel_size, attributes;
} TargaHeader;

I was expecting sizeof(TargaHeader) to be 18.

I get the result 20.

char = 1 byte = (1) length, (2) colormap, (3) imagetype + (4) colormap_size + (5) pixel_size + (6) attributes
short = 2 bytes = (1) colormap_index, (2) colormap_length, (3) x_origin, (4) y_origin, (5) width, (6) height

char 1 times 6 = 6 bytes
short 2 times 6 = 12 bytes

6 + 12 = 18

But I'm getting 20. I seem to missing something. Memory allocations are rounded up to blocks of 4, I assume?

[Baker]

When you apply the sizeof operator to a structure or union type name, or to an identifier of structure or union type, the result is the number of bytes in the structure or union, including internal and trailing padding. This size may include internal and trailing padding used to align the members of the structure or union on memory boundaries. Thus, the result may not correspond to the size calculated by adding up the storage requirements of the individual members.

Sheesh. So many hidden surprises ... no wonder one file loaders load things byte by byte with hardcoded constants (I do know what one cannot rely on the sizeof to be consistent across compilers.)

[taniwha]

Yeah, not only is sizeof (somestruct) not necessarily consistent between compilers, it's not necessarily consistent with the same compiler on different hardware (or even operating systems). However, many compilers do provide a way to handle it without resorting to constants.

gcc: __attribute__((packed)) (either on all the fields, or on the whole struct)
borland and msc: #pragma pack(1)

As you can imagine, you get a portability mess. The following is from QF's tga.h (__attribute__ is taken care of with some #define magic elsewhere)

Code: Select all

#ifndef __GNUC__
# if defined (__BORLANDC__) || defined (_MSC_VER)
#  if (defined(__BORLANDC__) && (__BORLANDC__ < 0x550))
#   pragma option -a1
#  else
#   pragma pack(push, tgainclude)
#   pragma pack(1)
#  endif
# else
#  error do some data packing magic here (#pragma pack?)
# endif
#endif

typedef struct _TargaHeader {
    unsigned char id_length; // __attribute__((packed));
    unsigned char colormap_type; // __attribute__((packed));
    unsigned char image_type; // __attribute__((packed));
    unsigned short colormap_index __attribute__((packed));
    unsigned short colormap_length __attribute__((packed));
    unsigned char colormap_size; // __attribute__((packed));
    unsigned short x_origin __attribute__((packed));
    unsigned short y_origin __attribute__((packed));
    unsigned short width __attribute__((packed));
    unsigned short height __attribute__((packed));
    unsigned char pixel_size; // __attribute__((packed));
    unsigned char attributes; // __attribute__((packed));
} TargaHeader;

#if defined (__BORLANDC__) || defined (_MSC_VER)
# if (defined(__BORLANDC__) && (__BORLANDC__ < 0x550))
#  pragma option -a4
# else
#  pragma pack(pop, tgainclude)
# endif
#endif

[Spike]

endian is another reason to load byte-by-byte.
also, bitfields are far far worse.

consider switching the shorts to something like unsigned char colormap_index_lo,colormap_index_hi; and you'll get the verbosity from decent names, as well as a fix for endian issues. If everything in the struct is a char then it'll not be padded to anything bigger than a byte.

[WickedShell]

Generally struct's should be organized from largest to smallest size of member elements to minimize the amount of trailing/wrapping.

Quake2World gets a ~5 fps boost for 64 bit clients (from 85 to 90 on my machine) by rearranging the order of elements in r_bsp_surface_t to save about 8 bytes total (I forget the before and after sizes, that infos on a different machine), But 32 bit clients see no difference. The FPS gain comes almost completely from reducing cache miss as far as I can tell. (Saved about 500KB of memory total on the worst case map I had on hand.) If someone has a better explanation for why saving 8 bytes created such a difference I'd love to hear it

Caveat: Be careful about rearranging all of your structs to minimize memory used. You can do dumb stuff and mess up the copying of common members between structs (node and leaf in the Q2 setting for example) Ofc if that screws up your code that badly it might be an indication that you have some really messy code that would benefit from some more love

@Spike: Any dislike for going all the way to uint32_t rather then just unsigned int for example? I've been going and using all the stdint.h types since it was pointed out to me that there is only a minimum size associated with unsigned int's and that implementation/platform/compiler specific they could easily be 64 or 128 bit's long on me. (Which would be very surprising in file formats ) Alternatively anyone played with the fast types and seen any noticeable differences? IE uint_fast32_t

[taniwha]

WickedShell: More than cache, overall memory bandwidth might be the explanation. IIRC, memory controllers always read whole cache lines, so wasted space in a struct will always be read from memory. No wasted space = no wasted reads = faster reading overall.

(u)int32_t etc: I've done a lot of such conversion in QF. Particularly, any new file-format stuff uses it (eg, I modified the iqm structs to use uint32_t etc when I implemented iqm in qf). I haven't played with the fast versions, and don't see much point: int is supposed to be the fastest size for integers, so if speed is needed, use (unsigned) int.

[Spike]

the fast stdint types are for machines which do not have 8-bit bytes etc. you won't see a difference on x86+amd64+arm.
there are also machines that only support 32bit bus access (alpha chips), in which case using a 32bit type and masking it to 16bits will often be faster.

stdint.h is a c99 feature, that does not exist on (freshly-installed) msvc, so you'd be making custom typedefs there anyway.
its a useful feature, but personally I'm generally not keen on adding dependancies purely for karma reasons.

tbh, the biggest win with int32_t is that its just a single letter to make it unsigned, instead of an entire extra word. otherwise its just 4 extra letters of verbosity.
I'll probably fix up fte's int specifiers when I try to port it to a machine where it makes a difference, otherwise I'm just gonna stay consistant with what its already using.