OpenCubes file formats

[JATothrim]

There are few file formats around. It is however bit unclear what these formats actually are for.
This issue is for discussion of the file formats used project wide + their documentation.

• What is the current export file format?
  I.e. the format all implementations are expected to be able write that contains the polycubes of N=?
• What are the currently used (other) file formats of the implementations?

@bertie2 @datdenkikniet @nsch0e @NailLegProcessorDivide

Rust and python versions seem to implement an format that follows #17 (comment) so I think this is the export file format.

[JATothrim]

The C++ cubes outputs its data currently only in following format as cache-files:

struct Header {
    uint32_t magic = MAGIC;  // should be "PCUB" = 0x42554350
    uint32_t N;              // we will never need 32bit but it is nicely aligned
    uint32_t numShapes;      // defines length of the shapeTable
    uint64_t numPolycubes;   // total number of polycubes
}
header;
struct ShapeEntry {
    uint8_t dim0;      // offset by -1
    uint8_t dim1;      // offset by -1
    uint8_t dim2;      // offset by -1
    uint8_t reserved;  // for alignment, set to zero and ignored currently.
    uint64_t offset;   // from beginning of file
    uint64_t size;     // in bytes should be multiple of XYZ_SIZE that is currently 3
}
shapeTable[header.numShapes];
struct XYZ {
    int8_t x,y,z;
}
polycubes[header.N * header.numPolycubes];

Additional notes:

• structures described here are laid out at begin of an cache-file in above order.
• shapeTable[].offset is not valid if shapeTable[].size is zero.
• shapeTable[].size == 0 entries are used as place holder shapes and don't have any XYZ data.
• the cache-files contains either all polycubes of N or are split shape.
• The polycubes are unordered.

Then question(s):
C++ cache-file format is different than what python and rust seem write.
Should C++ version export the data in the .pcube format?

I have plans for using ShapeEntry::reserved as extensions to the cache-file format.
I would like use it as format selector:

• 0x00: The current C++ cache-file data format as written by @bertie2 originally.
  Contiguous array of XYZ structs. The polycube size is N from the header.
• 0x01: For packed XYZ data. XYZ struct is packed into 16-bits with each X,Y,Z value having 5-bits each.
  Little-endian uint16_t has bits assigned as x:[0...5], y:[6...10], and z:[11...15].
  uint16_t value of 0xFFFF is reserved.
  Negative coordinate values are handled by adding constant +1 and then taking lower 5-bits. The +1 is subtracted when unpacking the coordinate value.
  This XYZ format limits stored X,Y,Z values between [-1 ... 30] and reduces the file/memory size by 33%.
• 0x02: Compressed polycubes. The compression scheme (WIP) is described by @nsch0e in C++ implementation #7 (comment) This means the shape table [offset, offset + size] contains compressed data in this format and each polycube is variable lenght in bytes. This still allows decoding the polycubes invidually, but the polycubes can only be read sequentially.