Pre-1.0 hoardDB is pre-1.0. Expect breaking changes.

Limitations

hoardDB is pre-1.0. This page is the honest list of what it does not do or does not do the way you might expect. Read it before you design around it. The sizes and counts that hoardDB enforces, with their defaults and how to change them, are on one page: Limits.

Ring mode, not available yet

Ring mode (hoarddb -ring, Getting started) is the engine this documentation leads with. A bucket declares fields, and a field can be a store of any of the six types, one per document. This section is what ring mode does not do yet. help all in a ring session lists the statements that run, and under “Not available yet in ring mode” the ones that parse and are refused. The rest of this page has not been rewritten for ring mode: where it describes a bucket with a type, a replication factor per bucket or a hash ring, ring mode does not have it.

A heap store cannot be pushed to, and a blob store cannot be put to. You can declare heapStore and blobStore fields. A heap store reads with peek, list, count and pop; push and take are refused with “not available yet”. A blob store answers count, and put is refused. A chunked blob upload, which sends a large value in pieces under one upload id, is planned. The other four stores (fifoStore, lifoStore, hashStore, btreeStore) run every verb in help types.
There is no timeline read yet. One read that fetches the newest elements of many documents’ stores together, such as a merged feed, is planned. Today it is one request per store. A single request can already carry up to 255 writes (see the next item).
Batching is on the wire, not in the CLI. A batch carries up to 255 writes in one request, to any buckets. It is not a transaction: each write is applied or refused on its own, a refused write does not stop the others, and writes to one document apply in the order sent. After a lost answer, resend the whole request with the same request id and the writes that were applied answer deduped, for as long as they are still inside replication.dedupe_window. There is no batch statement in ring mode; the Go ring client in the server module has NewBatch, and no client library has shipped.
There are no indexes in ring mode. A list with a filter reads every document of the bucket. Indexes declared when a bucket is created are planned. Indexes describes indexes as they were before ring mode.
Dropping asks for no confirmation. drop bucket runs at once, with no prompt and no --yes. drop database is not available in ring mode yet. When it arrives, drop database and drop bucket are decided to require the name to be typed to confirm, for every client; that is not built.
alter database is not available yet. A database is created with create database app (one copy) or create database app { replication_factor: n }, and its factor cannot be changed afterwards.
A scalar field’s declared type is not checked on write. A field declared int holds a string if you write one. The server refuses a field name the bucket does not declare, and nothing else about the value (Data types).
A store’s type never changes. alter bucket adds and drops fields. To change a store’s type, add a new field, copy, and drop the old one.
Encryption at rest does not cover ring mode. With encryption.mode on or require, the node does not serve ring mode and logs why: the ring store is not sealed yet. A node started with both cluster.enabled and encryption is refused. Encryption at rest for the ring store is planned.
dump, restore, ingest and batch do not see ring mode. They have not been built for ring mode yet.

Cluster in ring mode

join works on an empty cluster. Start the new node with the first node in its seed list, pin its key on every member, then run join <node>@<host:port> on any member. The join gives the node a vote while the cluster has a free voter slot, and promote <node> gives one later to a node that joined without one (Configuration has the steps). Nothing frees a slot yet, because leave and remove-node are not available. Cluster membership and database definitions are agreed through one leader (Raft), which the cluster elects. This is not the operator-only membership of the earlier engine.
join is refused once any database exists. A joining node cannot yet be seeded with the data it would own. Form the cluster first, then create data.
Replication, seeding and databases with more than one copy are not usable end to end. create database app { replication_factor: 2 } is refused on a cluster that cannot hold it, and a cluster that can is not yet a promise that every copy is written and readable. alter database, show replication, show placement, status seeding, seeding retry, seeding forget, leave, remove-node and replace-node parse and are refused. The sections of this page and of Replication about copies, seeding, rebalancing and removing a node were written before ring mode and do not describe it.

User management and authorization

Role changes apply at the next AUTH. Authorization is decided per connection from the identity established at AUTH. A grant, revoke, or alter user ... roles does not change what an already authenticated connection may do and does not terminate it; the change takes effect when that client authenticates again. If you need the change to bite immediately, change the password or disable the user (below).
Disable, drop and a new password close that user’s sessions on every node. These are the deliberate exception: alter user <name> disable, drop user <name> and a password change close every session that user has open, on the node serving the change at once and on every other node when the change reaches it, normally within a second. A node cut off from the cluster master keeps an open session until the change arrives, though it refuses new logins once it is past auth_config_max_staleness_seconds. grant and revoke do not close sessions.
User management is unavailable during a membership transition. While a node is being added to or removed from the cluster, a user mutation is refused with a retryable error. Queuing it would mean accepting a write the cluster cannot yet place. On a large cluster a rebalance is not a brief window.
A follower that falls behind refuses logins and refreshes. In a cluster, every node holds a full copy of the user store. A node that has not synced with the master for longer than auth_config_max_staleness_seconds (default 300, HOARDDB_AUTH_CONFIG_MAX_STALENESS) refuses new password AUTHs and token refreshes until it syncs. This is the deliberate trade-off: refusing logins is an outage, accepting a deleted or disabled credential is a security failure. Set the bound to what your recovery time allows.
A new cluster node with no user store blocks AUTH until it receives one. A node that has a users.json serves from it while it reconciles; a node with none will not accept logins from an empty user list.
Revocation is per user, not per token. Changing a password, disabling a user, or a role change bumps that user’s epoch and refuses all their outstanding tokens at their next refresh. There is no way to revoke one individual token without affecting the rest.
users.json contains every password hash in the cluster. Decommissioning a node means deleting its users.json, exactly as it means wiping its cluster token. A member removed from the ring keeps its copy until you delete it.
The user store is sealed under a key your backups must keep. users.json is encrypted under the node master key when encryption at rest is on, and under the node’s auth signing key when it is off. A node that starts without that key material refuses to start rather than reconciling, and --reset-password cannot recover it — it reads the same file. Losing the key loses every user, role and epoch in the store, not just the ability to verify stored hashes. This is a wider failure than a lost password pepper used to be.

Bucket engines

A document’s stores are declared on the bucket. A field can be a store of any of the six types, one per document: Friends hashStore gives every user their own hash. Declare it at create bucket, or add one later with alter bucket <name> { … }. Adding a store field is allowed; removing one is refused, because it destroys every document’s elements with no preview. HQL: store fields has the syntax and the output.
Only hash and btree buckets declare store fields. A fifo, lifo, heap or blob bucket’s entries are not documents with keys of their own, and the declaration is refused with the hash bucket to use instead.
A store field has no index, and find and search do not read it. A document’s stores are reached through the document’s key, with get, peek, pop or list. Index and search the bucket’s own fields.
A store field is at-most-once and has no dead list on a bucket that has a type. (Ring mode runs claim and ack on a store field.) claim, ack, release, extend and the dead verbs are refused on a store field: its pop is the only consuming verb, so an answer that does not reach you loses the element, and an element a handler keeps failing is not moved to a dead list. Use a separately named fifo, lifo or heap bucket when you need leases or dead entries. (D-15.)
There is no operator lease verb before v1.1. An entry held by a consumer that keeps extending its lease is released only by that consumer, or when it stops. No command lets an operator ack, release or steal another consumer’s lease. (D-13.)
A dump does not carry blob field payloads, as it does not carry blob bucket payloads (Backup and restore). The elements of every other kind of store field are dumped and restored in order.
Field types on scalar fields are not enforced. Age int and CreatedDate datetime are accepted and noted; only store fields and unique take effect. A scalar inside a store field’s braces is noted the same way, named as <store>.<field>; a unique modifier there is refused, because a unique index covers only a bucket’s own fields. Declare the field on the bucket, or give the elements their own bucket with a unique field on it.
You cannot change a bucket’s own engine after you create it. The engine is fixed at create bucket, and indexes are declared there too. There is no alter bucket <name> { type: ... } in v1.0, so a wrong choice means dropping the bucket and creating it again. Changing the bucket’s engine is planned for v1.1, and it will land in the client libraries as well as the server: alter bucket <name> { type: <type> } converts the bucket’s engine in place, and the conversion is expensive by nature, because it walks the source engine and rewrites every entry.

Changing a bucket’s replication factor

A raise copies one bucket at a time, and a stalled copy stalls only itself. The members a new factor places copy the bucket by themselves. A copy that cannot land (the member is down, or its disk is full) waits and retries; the bucket keeps accepting writes at the copies that hold it, and show buckets on the bucket’s owner reads below N/N until it lands.
A raise onto a full member waits. The alter is accepted, the member says why in its log once an hour, and the copy lands when you free space or run migrate <database> <bucket> on it. With auto_rebalance off the member only lists the buckets waiting.
An alter is refused while a node is joining and while the owner is recording a membership change. Both refusals say to retry; nothing was changed.
One change at a time per bucket. A second raise that keeps a member whose copy from the first has not landed is refused. The way back from a raise that cannot land is to lower the factor so that it no longer places the stuck member.
A lower factor deletes the copies it removes, once the remaining copies have shown that they hold everything the removed copies hold; their space is reclaimed in the background after the delete. With fewer members than the factor no copy is deleted. A lower the remaining copies cannot yet cover is refused, changing nothing; retry it.
No statement returns a bucket to inheriting the node’s factor. An alter to the value a bucket inherited makes it declare that value.

Writes and batches

A damaged data file can stop writes on the node. If a table file of the node store has a damaged block and that table is still in the newest layer, the background merge that would clear it fails every time and leaves it in place, so that layer keeps growing. At its limit every write to every bucket on the node is refused with a message that names the damaged file and says to restore the node, instead of waiting; the node reports it as not ready. Reads of undamaged data keep working, and the node can still be stopped: the shutdown completes and reports the damaged table. Restore the node from a backup (or remove it and add it again if its buckets have replicas on other nodes) as soon as the damaged-read counter or that message appears. Once the node has seen a damaged block it remembers that until it is restarted on restored data, so while that layer is at its limit, writes are refused even if the damage was in an older layer and only the newest layer is slow.
One atomic commit is about 20 MB and about 210,000 entries at the shipped tuning. That is 15% of the store’s memtable (storage.memtable_mb, 128 MiB by default); at its 32 MiB minimum it is about 5 MB and 52,000 entries. Every document adds one entry plus one for each index entry it produces, so many small documents reach the entry count before the byte count. A value of 1 MiB or more counts as almost nothing towards either bound.
A batch above that bound is refused whole, and nothing of it is written. The reply is an error with its own code, BATCH_TOO_LARGE, and (on a standalone node) write_outcome: not_applied. Its message says how many documents and how many bytes the batch was, states both bounds, and says to send fewer or smaller documents per batch (at the default, about 150 documents of 100 KiB, or 2,400 of 8 KiB, fit). Split the batch on the code and send the halves. hoarddb ingest halves a refused batch and retries, so a file of any size loads.
The bound depends on the store’s tuning and on the indexes a bucket declares; read it from the message rather than hard-coding it.
A document does not keep the field order it was written with. Every field comes back with the same value and the same BSON type, at every depth, but the fields may be in a different order from the one you sent, and two documents written identically may come back in different orders. Read fields by name. Do not hash or compare the raw bytes of a stored document; compare it field by field.
pop is at-most-once: an entry whose answer does not reach you is gone. A pop removes up to N entries (pop({limit: N}), one when you give no limit) and returns them in its answer, and the answer is the only copy. If the connection drops or the client gives up before the answer arrives, or a pop forwarded to the bucket’s owner loses its answer between nodes (UNAVAILABLE with write_outcome: unknown), up to N entries may be removed and you never receive them; do not pop again to look for them, because a pop takes the next entries. Run peek to see what is left. Read with peek first when you cannot afford to lose one, and pop fewer at a time when your program can lose an answer. For a fifo, lifo or heap bucket, use claim and ack instead: the entries stay in the store under a lease, and a lost answer becomes a redelivery rather than a loss. A store field is pop-only, so a lost pop of a field’s element is still a loss. Another request failing on the same connection does not lose a pop’s answer: the server answers every request already in flight before it closes a connection after an internal error. A pop still running HOARDDB_TCP_FRAME_TIMEOUT (30 seconds) after that error is cut off, and its entry may be lost.
On a single node, a pop whose disk write fails may serve its entry twice. A node that is alone in its ring syncs every write to disk before it answers, and a write whose sync fails (a full or failing disk) is answered with an error. A pop is the exception: it is answered with its entry, write_outcome: applied, and a message that says the removal is not yet on disk, because the answer is the only copy of the entry. The entry may then come back to the queue after a restart or a power loss, if the removal never reaches the disk, and be popped again. Check the disk when you see that message. On a cluster, a pop whose copies fall short also returns its entry (Replication).
A BATCH request (HQL batch { ... }) is not a transaction. Its items are not applied all-or-nothing, not isolated from other clients, and not ordered across buckets: a client that sends 21 items can see 18 applied, 2 not applied and 1 unknown, and another client can read item 5’s effect before item 2’s. Each item’s answer says what happened to it. Items naming one bucket apply in batch order, and after one that is not applied the later items of that bucket are not run; resend from the item the answer names. A batch whose answer is lost (a timeout, a dropped connection) may have applied any of its items: read back before resending a push or an $inc. A batch has no idempotency key in this release.
A follower can see a feed copy before the post it copies. A batch that stores a post in its author’s bucket and pushes a copy into each follower’s feed writes to different buckets, which have no order between them: a reader of a follower’s feed may meet the copy before the author’s Posts holds the post. Treat a feed entry as a copy that stands on its own.
An update by filter is refused on a spread bucket. A hash bucket recorded spread (name placement: spread; a hash bucket created with no placement is whole in this build) refuses a request that updates by filter (multi) on it, with the fix in the message; nothing is written. Update by _id_, find the documents and update each id, or create the bucket with placement: whole (create bucket User in app { type: hash, placement: whole }). A keyed update on a spread bucket, and any update on a whole one, work as before.
A batch carries store-field writes and reads and keyed updates only. A pop, a list, an update with a filter and the bucket verbs (insert, a bucket push, …) are sent on their own; documents are loaded in bulk with hoarddb ingest (Ingest).

Encryption keys per database and per bucket (planned for v1.1)

These describe a feature that is planned for v1.1 and not in v1.0. They are here now so that you do not design around a promise the feature does not make.

Every node in the cluster must encrypt its own storage first. An encrypted database or bucket needs encryption.keystore (keyfile, command or systemd-creds) and encryption.mode: on (or require) on every node in the cluster, not only on the nodes that hold it today, because a rebalance can move any bucket to any node. A create is refused, naming the node, if any node runs with encryption.mode: off. Once the cluster holds an encrypted database or bucket, a node with encryption.mode: off cannot join it. A node’s mode is chosen when its data directory is created, so turning it on for an existing node means removing the node, starting it again on an empty data directory, and adding it back. The reason is that the nodes hand these keys to each other using the cluster’s shared credential, which is stored in every node’s data directory. On a node whose storage is encrypted, a copy of that directory yields the credential but nothing the fetched keys can open.
A database or bucket key encrypts stored values, not keys or indexes. create database users { encryption: true } and create bucket post { encryption: true } will seal every value you write under a key that belongs to that database or bucket. They will not hide what is stored in keys. The database or bucket key does not encrypt any of these:
- every document key, and every key of a btree bucket (btree keys have to stay in order, so they cannot be encrypted this way);
- every secondary-index entry, which contains the value of the indexed field exactly as you wrote it — index an email field and the email addresses are stored as you wrote them;
- every heap priority, queue position and blob id;
- database, bucket and index names;
- how large each value is, to within a few bytes.
On disk, the node’s own storage encryption covers them, and every node must run it (above). They stay readable to anyone who holds a node’s master key or its escrowed copy, to the nodes themselves, and in every dump. The two layers stack: the node’s encryption covers everything on disk, and the database or bucket key adds a key you can destroy on its own.
What a database or bucket key does give you is a key you can destroy on its own, and separation between tenants’ values. What a drop destroys depends on where the key lives:
- Dropping a bucket that has its own key, or dropping a whole encrypted database, destroys that key on every node that holds it. The values become unreadable wherever copies of them are: the store, the write-ahead log, and every replica.
- Dropping one bucket inside an encrypted database destroys no key. The database key stays, because the database’s other buckets use it. The dropped bucket’s own key is stored, protected by the database key, in the write-ahead log of every node that holds the bucket, until write-ahead-log retention removes the entry that created the bucket. That depends on retention, not on a fixed time, and nothing is removed from the log until retention has a position to work from after a restart. It is also in the storage files until compaction. describe database shows, for each node, whether its log still holds entries from before the drop. If a dropped bucket’s values must become unreadable at once, give that bucket its own key instead of creating it in an encrypted database.
Destroying the key is not instant on disk. On each node, the deleted key’s record stays in the storage files until storage compaction removes it, and until then it can be recovered by someone who holds that node’s master key. A backup or snapshot of a data directory taken before the drop still holds the key. A dump is always plaintext, and destroying a key does not reach dumps you have already taken.
Encryption is chosen when a database or bucket is created. It cannot be turned on or off later. To encrypt existing data, create an encrypted bucket and copy the data into it with dump and restore.
A sealed blob file is a little larger than its content. An encrypted node writes blob content of 64 KiB and more as 24 bytes of header plus 28 bytes for every 64 KiB, so a 16 MiB blob takes 7,192 bytes more on disk than it holds. A bucket’s size in stats counts its content; the node’s disk gauges measure the files, so they include that overhead.

Queries

A scan reads one node, not the whole cluster. find() with no key, and find({filter}), are answered by one node that holds the bucket: the node that receives the request if it holds a copy, otherwise the ring owner, to which the request is forwarded. A scan is an ordinary read, so on a replica that is behind, the count reflects that replica; consistent get remains the verb for a read-your-writes guarantee. A bucket recorded spread is the exception: its find reads every part, as described next.
A find on a spread bucket reads every part. The bucket’s router asks each part for its share and merges the answers, so the result is the same page, in the same order, as a whole bucket holding the same documents returns. count, distinct, search, describe and dump cover every part for the same reason.
A deep page on a spread bucket can be refused where a whole bucket answers it. Each part is asked for its first skip + limit documents, not only its first limit, because one merged order must be possible; a find with a large skip therefore reads that many documents from every part. A part whose share passes its reply budget refuses the whole read with RESULT_TOO_LARGE, and nothing is returned. Ask for a smaller page, or create the bucket placement: whole.
The parts one node holds share one reply budget. A fan-out sends one request to each node that holds parts, and the combined answers must fit that request’s budget: the smaller of query.max_result_bytes and 16 MiB. A page of large documents that a whole bucket returns can therefore be refused on a spread bucket even though each part’s share fits, because a node holds many parts. Ask for fewer or smaller documents in the page.
A spread bucket with a part whose only copy is gone is unreadable. At replication factor 1, if the node holding one of the bucket’s parts is lost, find, count, distinct, search, describe and dump are refused with a message naming the collection and the node, and nothing is returned. There is no forget-lost: the collection stays unreadable until that node returns, or a dump taken before the loss is restored into a new bucket.
consistent get guarantees writes acknowledged at majority or all, and is sometimes refused. That is the default concern. It is refused while the bucket’s owner cannot be read and the other copies do not all agree, and while a new owner that was a replica takes the bucket over; retry. A write acknowledged at one, whose owner is then down, may not be returned.
After a node’s log is lost or restored from an older copy, do not rely on consistent get for a key of its buckets until that key has been written again. Until then the key compares as before: the read can return an older value, and its repair can write that older value over the owner’s acknowledged one.
consistent get is refused for a bucket whose holders are not all encrypted at rest alike. An encrypted node never sends its data to an unencrypted one, so neither copy can be trusted to be complete. A bucket held only by encrypted nodes still answers in a cluster that also has unencrypted ones. Finish encrypting every node (Configuration).
A write the bucket’s owner refused as too large for its batch, and that a replica with a larger memtable kept, is removed from that replica by the next consistent get of its key after the owner’s next write to the bucket. Give every member the same memtable size.
A find on a cluster can miss a write that was just acknowledged. A write is acknowledged once the bucket’s owner and enough other copies have it (majority by default), and a node that holds a copy answers a find from its own copy without asking the owner. If that copy has not applied the write yet, the find returns no match, although the document is not lost and shows up once the copy catches up. It is rare, and it can happen on the node that took the write. To read back a document you just wrote, use consistent get with its key, which reads every holder; for a find, retry it. A consistent find is planned for v1.1.
show buckets answers in one piece and counts every bucket. It returns every bucket of the database, with its entry count, in a single reply: there is no paging and no name filter, and its cost grows with the number of buckets. Keep a database to the thousands of buckets per node that Limits plans for.
Paging is skip/limit over a live bucket, not a snapshot — and sort does not fix it. There is no server-side cursor: each page is a fresh scan, and a sort orders the whole matched set again before skip and limit are applied. A concurrent write between two pages can move a document across the page boundary, so a document can be seen twice (an insert ahead of the boundary) or missed entirely (a delete ahead of it). Sorting makes this worse rather than better, because a paged listing ordered by something like created_at looks reliable and is not. Stop writes before exporting a bucket that must be consistent, or read it in a single page with a limit above the match count. A find with no limit returns the server’s default page of 100 documents; a search with no limit returns every match.
No “any element” matching. A filter key is a path that names one value: {"Lines[0].Sku": "A-1"} reads element 0. {"Lines.Sku": "A-1"} names a member of Lines, and Lines is an array, so it matches nothing. To match a value in any element, store it in a top-level field, or read the documents and test the elements in your client.
Blob buckets do not scan. A blob’s value is binary content rather than a document; find() on a blob bucket refuses and points at info/get for a single key.
Date-range queries on non-key fields are full bucket scans. A btree bucket orders only its key, not its fields. A query like {ts: {$gt: ISODate("...")}} on a date field that is not the key walks every document in the bucket and filters by type and value, with no index help. Declare the date as the btree key if range queries on it are performance-critical.

Indexes and expiry

No index on a nested value. An index reads a top-level field; one declared on a path such as Address.City is refused when the bucket is created. A query on the path still works, by scanning the bucket. To index the value, keep a copy of it in a top-level field and index that. See Indexes.
In this version an index can be declared only on a hash bucket, when it is created; it cannot be added, dropped or changed afterwards, and the other store types refuse one.
An indexed field can hold about 65,000 bytes, less for an index on several fields. An index entry’s key contains the field’s value, and the store caps a key at 65,000 bytes, a little of which the entry’s own prefix and the document key use. A document whose indexed value would push the entry past that is refused whole, with the field and the document named and “nothing was written”. Keep long text out of indexes: store it in an unindexed field, or index a hash or a prefix of it.
Unique means unique within one bucket — never across buckets or databases.
Unique needs placement: whole. A unique field other than _id_ is refused on a bucket recorded spread (name placement: spread; a hash bucket created with no placement is whole in this build), because no part of a spread bucket can check the others. Create the bucket with { placement: whole }; the refusal prints the line. Placement cannot be altered: for a whole bucket, dump it, drop it, create it with the placement you want and restore it. A spread bucket can be dropped (the drop removes every part, and if a part’s node does not answer it is refused partway and finishes when you run it again), but its placement and declaration are fixed at create, so decide them before loading data.
Uniqueness survives a crash in the sense that what is on disk is consistent — no two documents hold one unique value — not in the sense that every acknowledged write survives: writes are not synced one by one, so a crash can lose the last ones, unique or not.
An index serves equality on a prefix of its fields. Other predicates of a search are checked document by document, and a filter no index fits is a full scan, which the response’s index_used reports.
Expiry is background work. The TTL sweeper runs every HOARDDB_TTL_SWEEP_SECONDS (default 60), so an expired document can still be read until the next sweep removes it. A bucket expiring documents faster than the sweeper deletes them grows: one node deleted about 50,000 expired documents a second in our measurement, and the owner of a three-node cluster about 23,000.
A document whose store fields hold too many elements cannot be deleted, and never expires. Deleting a document deletes its store fields in the same atomic commit, and a commit has the entry bound given under Writes and batches. A document whose stores need more than that is left whole: delete answers that its store fields hold too many elements and names the field and the verb that empties it, and the TTL sweeper skips it on every pass, logs one warning, and counts it under hoarddb_ttl_sweep_documents_total{outcome="skipped"}. Empty the store field (for example pop({limit: 100}) until it answers EMPTY), and the next sweep or a delete removes the document. If a TTL is a retention commitment, watch that metric.

Transport and sessions

Requests on one connection are pipelined, up to a window. A client may send up to 4 requests (and 1 MiB of them) before it reads an answer (HOARDDB_TCP_PIPELINE_FRAMES, HOARDDB_TCP_PIPELINE_BYTES); past that the server stops reading until an answer is written, and TCP slows the client. The requests in the window run concurrently, queries among them included, so answers can come back in a different order from the requests: each answer carries its request’s ID, and the client matches them by it. The server advertises this as query_pipelining when a client logs in.
Queries share a node-wide number of slots. At most twice the CPU count of SCAN, SEARCH, BTREE_RANGE, COUNT and DISTINCT run at once on a node (HOARDDB_QUERY_CONCURRENCY); a query beyond that waits for a slot and is never refused for it. Point reads and writes do not take a slot.
A query’s result has a ceiling. What one query may hold in memory is bounded (HOARDDB_QUERY_MAX_RESULT_BYTES, 16 MiB by default). A query over it is answered RESULT_TOO_LARGE, and the connection and the requests in flight on it carry on. Page with skip and limit, or page by the sort field.
A single document can be larger than the result ceiling. RESULT_TOO_LARGE carries retained_bytes, the bytes the query had already kept. It is absent when that is 0, which is the case when the first document alone is over the ceiling: no limit then fits, so project fewer fields or raise HOARDDB_QUERY_MAX_RESULT_BYTES instead of paging.
A query stops when its client goes away. Closing the connection is the only way to cancel; there is no per-request cancel and no per-query deadline. A client that half-closes its connection (CloseWrite, or a TLS close_notify) and then waits for answers loses the answers to queries still running. A write the server has logged is applied and replicated whether or not its client is still there; the client just does not hear how it went.
A connection is closed for idleness only with nothing in flight. The idle timeout (HOARDDB_TCP_IDLE_TIMEOUT, 300 seconds) counts from the last answer, so a query that runs longer than it is still answered. A client that stops reading its answers is disconnected once an answer cannot be written within HOARDDB_TCP_FRAME_TIMEOUT, and nothing more it sent is run.
A frame is capped at 32 MiB (transport.MaxFrameLength). You will meet the two limits below first: a document is capped at 24 MiB by default (30 MiB at most) and a blob at 16 MiB by default (31.9 MiB at most), each to leave room in the frame for its envelope and the copy sent to a replica. A request over the cap is refused with an error that says so and names the fix (split the batch, or store a large value as a blob), and the connection stays open. Anything larger must be split or kept on your own storage; chunked streaming is not implemented yet.
A document is capped at 24 MiB by default. The limit is on one document’s own encoded bytes, and it applies to a put or an insert into any bucket but a blob bucket, a batch’s documents, a push to a queue or heap, a store field’s element, a restored document, and the result of an update: an update that would grow a document past it is refused and leaves the document unchanged. An operator can lower the limit to 4 KiB or raise it to at most 30 MiB with HOARDDB_MAX_DOCUMENT_BYTES (storage.max_document_bytes); 30 MiB is the most that fits a 32 MiB frame with its envelope, its copy for a replica and a dump. A larger document is refused with a message that names the limit. A list that grows belongs in a nested store field, whose elements are stored separately and do not count towards the parent document; a large value belongs in a blob bucket, or on storage you run behind a pointer document (Large file stores). Terms, once: a hoardDB bucket is the counterpart of what many document databases call a collection, and a document is a document. hoardDB sets no limit on a bucket’s size and no limit on how many documents it holds; the disk is the bound. See Configuration for the setting.
A blob is capped at 16 MiB by default. An operator can raise the cap with HOARDDB_MAX_BLOB_BYTES (storage.max_blob_bytes) to at most 33,488,896 bytes (31.9 MiB), not 32 MiB: the frame is 32 MiB including its envelope and the copy sent to a replica, and the ceiling leaves 64 KiB for both. A larger blob is refused with a message that names the limit. The cap is the only size limit on a blob bucket: a document put or inserted into one is stored as the blob’s content and is held to storage.max_blob_bytes, not to the document limit, whether the cap is set above or below it. Keep larger files on storage you run and store a pointer document, as described in Large file stores; see Configuration for the setting.
After you lower the document limit, a document already over it stays readable, and you can shrink it in steps. An update to such a document is accepted when its result is no larger than the stored document, even if the result is still over storage.max_document_bytes; for example, an $unset that takes a 1.9 MiB document to 1.5 MiB under a 1 MiB limit is accepted. An update that makes it larger is refused and leaves it unchanged: Document "big" would be 1.9 MiB after this update; this server's limit is 1 MiB per document. A delete is always accepted.

Cluster

A brand-new cluster serves nothing until every founding node is up. At its first start each node takes its member list from its seed settings and answers database and bucket requests with “not added yet” until every node in that list has answered it and agrees. One founder that never starts keeps the others waiting; status shows Membership: WAITING FOR FOUNDERS with the missing nodes on each node until then. Start the missing node, or, if it is not coming back, run remove-node <node-id> for it on a running node: the others then confirm within seconds. This applies to the first start only: a node that has been confirmed once remembers it across restarts.
Whether the cluster is in sync is read from metrics and the node log, not from a CLI command. Each node reports how many members its streams to are stalled and how far behind the slowest one is (Is the cluster in sync?). There is no lag per bucket: a series per bucket would grow with the bucket count until the metrics endpoint is too large to scrape.
Creating and dropping a database through different nodes is not ordered for you. A create database, drop database or first create bucket is applied where it lands and reaches the other nodes on its own; two such requests sent through two nodes are not ordered against each other. Issue a database’s DDL, and the first buckets in it, through one node, or wait until every node lists it; a database created through two nodes at once and then dropped can keep some of its buckets, so drop it again. A drop database that keeps buckets says so in its answer. Every node agrees on the result either way.
Durability is a number, not a refusal. majority and all do not fail when replicas are out of sync; they report copies_confirmed. A write confirmed by one copy is lost if that node’s disk is.
A spread bucket is answered through its router only. Every operation on a bucket recorded spread — reads and writes by key, find, count, distinct, search, describe and dump — is answered through one node: the router, which is the node that owns the bucket’s name. While that node is down the bucket cannot be read or written at any replication factor, until it returns or remove-node hands the bucket’s name to a node that holds a copy of it. A whole bucket at factor 2 or more stays readable from its copies. A bucket that must stay readable while any one node is down belongs at placement: whole, which is the default in this build.
A replica that returns during heavy writes at one can stay behind while they run. The owner sends a returned replica what it missed one frame at a time, and on 4-vCPU hosts that can fall behind writes at one that arrive faster than the replica can apply them; if it stays behind, slow the writers or write at majority. The replica then catches up only once the writes slow down, and its reads are stale until it does. Meanwhile it is shown not in sync (hoarddb_replication_peers_not_in_sync), and majority and all writes do not wait for it: they answer at once with the copies that hold them. Writes at majority or all are throttled to what the replicas confirm, and a returned replica caught up within 30 seconds in every run we measured. Faster catch-up is planned after v1.0.
hoardDB never removes a node automatically. A peer that stays unreachable is reported; an operator decides whether and when to remove it.
A member that is unreachable, or whose stream is refused, holds every node’s write-ahead log. No node prunes its log past what that member confirmed, so logs grow until it returns, is repaired, or is removed with remove-node. This is what lets a member that was down receive every create database and drop database it missed; on a small cluster with few buckets it is new, because a node used to keep its log only for the members that hold one of its buckets. hoarddb_replication_peers_not_in_sync names the condition.
A node that joins does not learn an empty database. It learns a database with the first bucket it copies. A database with no bucket in it, created before the node joined and pruned from every log since, is missing on that node until something is created in it there.
Reads of a bucket that has just moved can answer “unavailable, retry” for up to a second, then go to its previous owner. After a join or a remove-node, a node can own a bucket before it has the bucket’s data. While the membership change settles, a read of such a bucket there waits up to one second for the node to take the bucket over and is answered from it; past that it is sent to the bucket’s previous owner, or it answers “unavailable, retry”. It never answers “not found” for a bucket the cluster holds. A read of a bucket whose only copy was on a removed node that does not answer is “not found” once the change has settled, and the answer names that node: if the bucket existed only there, its data is on it. Retry a refused read in a moment: nodes shows the members, and Is the cluster in sync? shows how to tell when every member has caught up.
During a join, the bucket counts in show databases can be short until the new owners’ copies land; show buckets stays complete.
Schema changes pause while a node joins. From the moment a node is added with join until every member has handed it the buckets it now holds, every create database, drop database, create bucket, drop bucket, alter bucket, restore, and ingest that has to create its bucket is refused with “unavailable, retry”; nothing is created, dropped or restored. Reads and writes to existing buckets carry on. The refusal and nodes name the joining node, how far the copying has got, the member the join waits on, and the ways out (Schema changes pause while a node joins). A member that is down at any time before the join has settled holds the pause until it is back or removed. With auto_rebalance off the pause lasts until the migrate commands that nodes lists have been run.
A node that has not yet heard of a join still accepts some schema changes. For up to a gossip round after a join, such a node takes create database and drop database and delivers them to every member, the new node included, once it has learned of the join. A bucket change it has to send to another node is refused with “retry”. A bucket change for a bucket it owns itself is accepted and answered like any write: success once its write concern is met, or “unknown” if the join moved the bucket off it while the change was in flight. Either way the bucket is handed to the nodes that hold it after the join before the pause lifts, so nothing is lost.
Removing a down member during a join can lose the buckets only it held. remove-node of a member that is not coming back lifts the pause once the others have handed off again, but a bucket whose only copy was on that member becomes unreachable. At a replication factor of 2 or more with one node down no bucket has its only copy there. Undo the join with remove-node of the new node first if you can bring the down member back.
Outside a join, a create or drop of a bucket at a node that owns it but has not copied it yet can be refused with “unavailable, retry”. The node asks every member first whether one holds the bucket. If one does, the request waits until the copy lands. If a member does not answer and the bucket’s placement changed since every member last finished a membership change, the request is refused, naming the members that did not answer; retry once they answer, or remove a member that is not coming back with remove-node. A bucket whose placement has not changed is created or dropped while a member is down.
A replica that cannot apply a write its owner applied stops replication from that owner. It refuses the write instead of acknowledging it, and the owner’s stream to it waits there, for every bucket, so writes at w=all to those buckets wait out their concern and the owner keeps its log, until the replica has copied the bucket from its owner again: on its own with auto_rebalance on, and when you run migrate <database> <bucket> on it with auto_rebalance off (see Replication).
Data written by a build before this one has no bucket identity. Every bucket now carries an identity, its incarnation, that tells a re-created bucket from the one it replaced. A bucket carried over from an earlier build reads as incarnation 0; beside a copy created on this build it is reported as a different bucket and never merged. Before the first release, data directories are not carried across builds; if you carried one, drop and re-create each bucket.
A member of an encrypted cluster that comes back with an unencrypted store holds a later join’s pause. join refuses an unencrypted node, but a member restarted from a fresh, unencrypted data directory is still a member, and no data is placed on it; while it is in a moved bucket’s set, a join never settles. Start it from a fresh data directory with encryption on, or remove it with remove-node.
An encrypted node refuses to add a node it cannot reach. In encryption.mode: on, join asks the new node whether its store is encrypted, and a node that is not running yet, or answers too slowly, is refused with “could not ask … whether its store is encrypted”. Nothing changes and nothing pauses; start the node, or wait for it to come up, and run join again.
A cluster in encryption.mode: on whose members are mostly unencrypted can still add an unencrypted node. The check runs on the encrypted members: when an unencrypted member proposes the join, an encrypted member’s refusal counts only as one vote against it, and a majority of unencrypted members commits it. Such a cluster already cannot place a bucket whose set includes an unencrypted member through its encrypted ones. Encrypting every member rules it out.
Adding a node does not relieve disk pressure; see Is this node full?
User-management fencing is a reachability check, not consensus. The master confirms a strict majority of the current members are reachable before accepting a user mutation, and refuses with a retryable “no quorum” error otherwise. It is not a replicated consensus log; two nodes with divergent membership views can each believe they own the reserved key, and the term/version fence is what makes that safe.
Nodes cannot be rolled back to an earlier build once they have written to their log on this one. An earlier build reads the newer log entries as a damaged tail and discards them, and after a crash the copies of a bucket can then silently disagree about writes that were acknowledged. Back up every node’s data directory before you upgrade: stop the node and copy the whole directory (unlike dump, that includes blob payloads). Going back means restoring that copy on every node, not starting the old build on the new data. The first time a node writes the new format it logs one INFO line, “this node now writes its log in the origin format: an earlier hoardDB opens the data directory but discards the newer entries as a damaged tail”. A log in segment format 2 is refused, not discarded: a node with a write-ahead log writes format 2 from its first start on this build, and an earlier build started on it stops with parsing sealed file header: flags must be 0x00 and changes nothing; start the newer build, or restore the copy taken before the upgrade. A cluster is upgraded with every node stopped. Before the upgrade, stop writes and wait one minute; an earlier build has no gauge for this. An earlier build started on such a data directory also cannot see this build’s replication positions, so it applies re-sent entries a second time and FIFO and LIFO buckets silently gain duplicate elements; it does not refuse and does not warn. Restore a backup taken before the upgrade, or remove the node and join it again with an empty data directory. See Backup and Restore.
Upgrade by stopping every node. A cluster that runs two builds at once is not supported, and its failures are quiet. Stop all nodes, install the new build on all of them, then start them. In a mixed cluster, replication to a node on an older build can stall, so a write that waits for that node (w=all) times out; an older node that encrypts its store does not check whether a new peer does, and keeps sending it data (see Encryption in a cluster); an older owner does not tell a replica with a smaller storage.memtable_mb that it applied a large batch, so that replica skips the batch without a word (see How big is my memtable); and a data directory written by an earlier build, restored into a running cluster, can reuse sequence numbers its peers already hold, so they skip those writes and acknowledge them.
Copying a bucket to another node holds the whole bucket in memory on both nodes. A join, a remove-node, a rebalance and migrate copy a bucket as one stream that the sending node builds in memory and the receiving node collects in memory before it writes it. A large blob bucket, whose payloads otherwise stay on disk, needs memory for all of them on both nodes while it moves. Do not move a bucket larger than a node’s spare memory; the memory is released when the copy ends. A receiving node refuses a copy larger than half of its memory limit before it fetches any of it, and names the bucket. Nodes that already hold the bucket are unaffected.
Dropping and re-creating a database while a node copies one of its buckets can leave that bucket unserved on that node. If a bucket is being copied in twice at once, from two incarnations of a database that was dropped and created again under the same name, and the older incarnation is dropped as the newer copy finishes, the node lists the bucket and does not serve it. Restart that node. Let a join or a migrate finish before you drop a database and create it again.
Entries a node applied before it was upgraded can be sent again after the first owner change that follows the upgrade, while its log holds them, to a bucket whose owner had already changed before the upgrade. A queue bucket then holds those elements twice.
Removing a node that is down discards what only it held. remove-node on a node that cannot be reached loses the writes that were acknowledged by that node alone (one, or copies_confirmed: 1). For a queue bucket, an element popped by such a write is served again.
A membership change can make a bucket refuse writes for longer than a moment. If the node that owned a bucket is gone and another node that may hold a copy cannot be reached, the bucket refuses writes until that node returns or is removed. This needs two nodes unreachable at once — the old owner and one more — and affects about one bucket in n on an n-node cluster; reads are still served. remove-node on the unreachable one ends it, needs a majority of the current members, and discards what only that node held. At three nodes with two lost there is no way out but to bring one back. With five nodes at a replication factor of three, two lost, removing both lost nodes restores a full copy of every bucket on the three survivors; see Replication.
A node removed while it could not reach the cluster keeps accepting writes until it reconnects, answers them copies_confirmed: 1, and they are discarded.
A node that missed more than 64 membership changes must be re-added. Remove it, empty its data directory and add it again.
If two copies of a bucket were created apart (a node removed while it held the only copy, and the name created again), the cluster keeps both and refuses writes to the name until you drop it or remove the node that holds the other one. The error and a warning in the log, at most once an hour, name both nodes.
A node keeps its copy of a bucket that has left its replica set until every member of the set holds what it holds. show buckets on that node prints leaving beside such a bucket. A member that is down holds the copy back until it returns or is removed; the node logs a warning naming the member once the copy has waited more than five minutes, and once an hour after that.
A create issued just after a node was removed can be refused once. The refusal reads “a snapshot of this bucket is being pulled onto this node”. It is the retryable kind: the create succeeds on the next try.
migrate can report success while a stalled copy stays stalled. If a copy refused entries from another incarnation of a bucket and ownership has moved since, migrate re-seeds the copy from the new owner correctly, but the stall can still be listed. Check status; if the stall remains, drop the bucket on that node and let it be copied again.
While a replica is down, the owner’s record of entries it refused grows with each refusal. The record is rewritten whole each time, about 22 bytes per refusal. Refusals are rare, but a replica that stays down for days under a high refusal rate should be brought back or removed.
A data directory written by a development build from before the first release is not supported. Start the node on an empty directory and join it to the cluster.
A failed read inside a multi-document update can leave a node’s copy different from its replicas. If a store read fails inside a filtered or keyless multi-document update and the node later restarts, the node logs an ERROR and tells you to dump the bucket on a replica and restore it over the node’s copy. It does not repair itself.
Seed fingerprints for nodes on one host with consecutive ring ports clash. A node’s internode address is its ring port plus one, so host:4433=... and host:4434=... claim the same address, and the dial fails with “does not match the configured seed fingerprint” although both fingerprints are right. Leave at least two ports between the ring ports of nodes that share a host.
A damaged wal/incarnation.bson is trusted if it still decodes. The file has no checksum, so a flipped bit inside a stored id or position passes, and the node rewrites it as it is. Restore the node’s data directory from a backup or from a replica if you suspect the disk.
After a crash, a node can apply a large batch that its replicas refused. A batch close to the entry bound is counted differently when the log is replayed than when it is applied live. If the node crashed between logging the batch and applying it, replay can apply a batch the other copies refused for size. Keep batches well under the bound given under Writes and batches.
If a data file is damaged, the space of buckets dropped after the damaged range may not be returned until the node is restored. The node logs the failed reclaim each second.

Source and distribution

The source is not published yet. hoardDB is source-available — the server under Business Source License 1.1, the client libraries under Apache-2.0 — but the public repository does not exist today. The history is being prepared for publication, and the source lands when hoardDB reaches v1.0.

Source: docs/user/limitations.md in the repository.