Module darkfi::geode

source ·
Expand description

Chunk-based file storage implementation. This is a building block for a DHT or something similar.

The API supports file insertion and retrieval. There is intentionally no remove support. File removal should be handled externally, and then it is only required to run garbage_collect() to clean things up.

The filesystem hierarchy stores two directories: files and chunks. chunks store MAX_CHUNK_SIZE files, where the filename is a BLAKE3 hash of the chunk’s contents. files store metadata about a full file, which can be retrieved by concatenating the chunks in order. The filename of a file in files is the BLAKE3 hash of hashed chunks in the correct order.

It might look like the following:


In the above example, contents of 7d4c0d5539057c8f9b60d32b423964beb38ecd8ea1ab203c0207990cbf0cad22 may be:


This means, in order to retrieve 7d4c0d5539057c8f9b60d32b423964beb38ecd8ea1ab203c0207990cbf0cad22, we need to concatenate the files under /chunks whose filenames are the hashes found above. The contents of the files in /chunks are arbitrary data, and by concatenating them we can retrieve the original file.

It is important to note that multiple files can use the same chunks. This is some kind of naive deduplication, so we actually don’t consider chunks to be specific to a single file and therefore when we do garbage collection, we keep chunks and files independent of each other.


  • ChunkedFile is a representation of a file we’re trying to retrieve from Geode. The tuple contains blake3::Hash of the file’s chunks and an optional PathBuf which points to the filesystem where the chunk can be found. If None, it is to be assumed that the chunk is not available locally.
  • Chunk-based file storage interface.


  • Path prefix where file chunks are stored
    Path prefix where file metadata is stored
  • Defined maximum size of a stored chunk (256 KiB)