Expand description
Chunk-based file storage implementation. This is a building block for a DHT or something similar.
The API supports file insertion and retrieval. There is intentionally no
remove support. File removal should be handled externally, and then it
is only required to run garbage_collect() to clean things up.
The filesystem hierarchy stores two directories: files and chunks.
chunks store MAX_CHUNK_SIZE files, where the filename is a BLAKE3
hash of the chunk’s contents.
files store metadata about a full file, which can be retrieved by
concatenating the chunks in order. The filename of a file in files
is the BLAKE3 hash of hashed chunks in the correct order.
It might look like the following:
/files/7d4c0d5539057c8f9b60d32b423964beb38ecd8ea1ab203c0207990cbf0cad22
/files/...
/chunks/9d7abc2efa52b8be63ff82b756edb6822e09aa40fc587aba977185a5bb449c19
/chunks/fc432e087d16d8788e87640511e627be34a4a50533f1e5ed3e1370645a0266b8
/chunks/...In the above example, contents of 7d4c0d5539057c8f9b60d32b423964beb38ecd8ea1ab203c0207990cbf0cad22
may be:
9d7abc2efa52b8be63ff82b756edb6822e09aa40fc587aba977185a5bb449c19
fc432e087d16d8788e87640511e627be34a4a50533f1e5ed3e1370645a0266b8This means, in order to retrieve 7d4c0d5539057c8f9b60d32b423964beb38ecd8ea1ab203c0207990cbf0cad22,
we need to concatenate the files under /chunks whose filenames are the
hashes found above. The contents of the files in /chunks are arbitrary
data, and by concatenating them we can retrieve the original file.
It is important to note that multiple files can use the same chunks. This is some kind of naive deduplication, so we actually don’t consider chunks to be specific to a single file and therefore when we do garbage collection, we keep chunks and files independent of each other.
Structs§
ChunkedFileis a representation of a file we’re trying to retrieve fromGeode.- Chunk-based file storage interface.
Constants§
- Path prefix where file chunks are stored
- Path prefix where file metadata is stored
- Defined maximum size of a stored chunk (256 KiB)