Module darkfi::geode

Chunk-based file storage implementation. This is a building block for a DHT or something similar.

The API supports file insertion and retrieval. There is intentionally no remove support. File removal should be handled externally, and then it is only required to run garbage_collect() to clean things up.

The filesystem hierarchy stores two directories: files and chunks. chunks store MAX_CHUNK_SIZE files, where the filename is a BLAKE3 hash of the chunk’s contents. files store metadata about a full file, which can be retrieved by concatenating the chunks in order. The filename of a file in files is the BLAKE3 hash of hashed chunks in the correct order.

It might look like the following:

/files/7d4c0d5539057c8f9b60d32b423964beb38ecd8ea1ab203c0207990cbf0cad22
/files/...
/chunks/9d7abc2efa52b8be63ff82b756edb6822e09aa40fc587aba977185a5bb449c19
/chunks/fc432e087d16d8788e87640511e627be34a4a50533f1e5ed3e1370645a0266b8
/chunks/...

In the above example, contents of 7d4c0d5539057c8f9b60d32b423964beb38ecd8ea1ab203c0207990cbf0cad22 may be:

9d7abc2efa52b8be63ff82b756edb6822e09aa40fc587aba977185a5bb449c19
fc432e087d16d8788e87640511e627be34a4a50533f1e5ed3e1370645a0266b8

This means, in order to retrieve 7d4c0d5539057c8f9b60d32b423964beb38ecd8ea1ab203c0207990cbf0cad22, we need to concatenate the files under /chunks whose filenames are the hashes found above. The contents of the files in /chunks are arbitrary data, and by concatenating them we can retrieve the original file.

It is important to note that multiple files can use the same chunks. This is some kind of naive deduplication, so we actually don’t consider chunks to be specific to a single file and therefore when we do garbage collection, we keep chunks and files independent of each other.

Structs

• ChunkedFile
  ChunkedFile is a representation of a file we’re trying to retrieve from Geode. The tuple contains blake3::Hash of the file’s chunks and an optional PathBuf which points to the filesystem where the chunk can be found. If None, it is to be assumed that the chunk is not available locally.
• Geode
  Chunk-based file storage interface.

Constants

• CHUNKS_PATH 🔒
  Path prefix where file chunks are stored
• FILES_PATH 🔒
  Path prefix where file metadata is stored
• MAX_CHUNK_SIZE
  Defined maximum size of a stored chunk (256 KiB)