Pre-1.0 hoardDB is pre-1.0. Expect breaking changes.

Example: large file stores

Keep large files (video, audio, archives, images) on storage you already run, such as a shared NFS mount, a SAN volume, or an object store’s FUSE mount. Keep one small document per file in hoardDB saying where the file is and what it should look like. Your webservers look the document up, then read and serve the file themselves. The bytes never pass through hoardDB.

This is a usage pattern, not a store type. It’s an ordinary hash bucket, and it works today with nothing to configure.

When to use it

• Many webservers or workers read the same files, and they can all mount the same storage.
• The files are large enough that you want the webserver’s own sendfile or range handling, rather than the bytes travelling through a database.
• You already have backups, retention and permissions for that storage and want to keep them.

If you want hoardDB to own the bytes (write them, checksum them on every read, and remove them when the key is deleted), use a blob bucket instead, for files up to the blob cap below.

The blob cap is why this pattern exists

A blob bucket refuses a blob over the node’s cap: 16 MiB by default, and at most 31.9 MiB (33,488,896 bytes) when an operator raises it with HOARDDB_MAX_BLOB_BYTES. The limit comes from the wire: hoardDB moves a blob in one 32 MiB frame with no streaming, and the frame also carries the request’s envelope and the copy sent to a replica. See Configuration for the setting and Limitations for the frame.

A single document has a limit too, 24 MiB by default and at most 30 MiB; see Limitations.

A file over the cap, such as a video or an archive, stays on your storage, and hoardDB holds the pointer document described here. The refusal says so:

Blob "videos/intro.mp4" is 5.0 GiB; this server's limit is 16 MiB
(storage.max_blob_bytes, maximum 31.9 MiB). For larger files, keep the file on
your own storage and store a pointer document — see "Large file stores" in the
docs.

Who owns what

hoardDB treats path as a string like any other field. It doesn’t check that the file exists. A dump carries the pointer documents but not the files, so back up the file storage separately.

Recording a file

Compute the size and checksum where the file is written:

$ stat -c %s /mnt/media/videos/intro.mp4
5000000
$ sha256sum /mnt/media/videos/intro.mp4 | cut -d' ' -f1
2e05c61949591b87293a6574c6d894249d0fabedffe756d24a1bd46438fdd875

Then create a hash bucket for the pointers and insert one document per file. Use the name your application asks for as the key, so a lookup is a single get:

[h]oardDB> create database media
OK  created database media
[h]oardDB> use media
Switched to database 'media'
[h]oardDB [media]> create bucket files { type: hash }
OK  created bucket media.files (type hash)
[h]oardDB [media]> db.files.insert({_id_: "videos/intro.mp4", path: "/mnt/media/videos/intro.mp4", size: 5000000, sha256: "2e05c61949591b87293a6574c6d894249d0fabedffe756d24a1bd46438fdd875", content_type: "video/mp4"})
OK  id=videos/intro.mp4

Looking a file up

[h]oardDB [media]> db.files.get("videos/intro.mp4")
{"_id_": "videos/intro.mp4", "content_type": "video/mp4", "path": "/mnt/media/videos/intro.mp4", "sha256": "2e05c61949591b87293a6574c6d894249d0fabedffe756d24a1bd46438fdd875", "size": 5000000}

You can filter on any other field too. This query scans the bucket:

[h]oardDB [media]> db.files.find({content_type: "video/mp4"})
{"_id_": "videos/intro.mp4", "content_type": "video/mp4", "path": "/mnt/media/videos/intro.mp4", "sha256": "2e05c61949591b87293a6574c6d894249d0fabedffe756d24a1bd46438fdd875", "size": 5000000}

To serve a request, a webserver:

1. gets the pointer by key. NOT FOUND means a 404.
2. Opens path on its own mount and serves it with content_type.
3. Uses sha256 as the ETag, and size as the expected length.

hoardDB doesn’t ship a client library yet. Until it does, a webserver talks to hoardDB over its wire protocol.

Removing a file

[h]oardDB [media]> db.files.delete("videos/intro.mp4")
OK  deleted "videos/intro.mp4"
[h]oardDB [media]> db.files.get("videos/intro.mp4")
NOT FOUND

The file on /mnt/media is still there. Removing it is your job.

Keeping the two in step

hoardDB can’t keep the pointers and the files consistent for you. Order the steps so that a failure halfway through leaves only a harmless leftover:

• Adding: write the file completely (to a temporary name, then rename it into place), then insert the pointer. A crash in between leaves a file with no pointer, which nobody can request.
• Replacing: write the new content under a new path, update the pointer, then remove the old file after a delay longer than your slowest download. Overwriting a file in place changes it under readers that are already streaming it, and it no longer matches its sha256.
• Removing: delete the pointer first, then remove the file after the same delay. A crash in between leaves a file with no pointer, never a pointer to nothing.
• Checking: run a periodic job that compares the bucket against the directory to find both kinds of leftover. Re-running sha256sum and comparing against the stored sha256 finds files that changed or rotted on disk.

Choices to make up front

• Store paths, not URLs. If your webservers mount the storage in different places, store paths relative to an agreed mount point: media/videos/intro.mp4 under a per-host root moves more easily than an absolute path baked into every document.
• Add any fields your application needs, such as owner, duration, width and height. They’re ordinary document fields.

Source: docs/user/large-files.md in the repository.