Pre-1.0 hoardDB is pre-1.0. Expect breaking changes.

Replication

This page has not been rewritten for ring mode. It describes the consistent hash ring, a write-ahead log, a replication factor per bucket and a write concern. Ring mode (hoarddb -ring, with HOARDDB_CLUSTER_ENABLED for a cluster) does not use any of it, and in ring mode replication, seeding a node that joins and databases with more than one copy are not usable end to end yet. What does work in ring mode is join on a cluster that holds no data: Configuration has the steps and Limitations has what is missing.

hoardDB replicates data between nodes over TCP+TLS. This document is the design record: the model, the wire path, what is guaranteed, and what is not implemented yet.

Status of the replication this page describes: working and tested for the model below — three nodes at RF=3 and five nodes at RF=3 (exactly three of the five hold each bucket), data verified present in each node’s own storage, write concern enforced, and a restarted replica repaired from the owner’s log. See server/cluster_replication_test.go.

What was here before

The previous implementation reported success without replicating anything. replicateEntryAsync dialled a peer, logged that it was sending append entries, and then set peer.lastAckSeq = entry.Seq on its own and signalled a local acknowledgement. WaitForAcks returned len(peerIDs) after a single local signal, so even w=all was satisfied by the leader alone. Nothing called it, either: no handler went through the write path, and storage.OpenWAL was never called, so the WAL was never open.

Two consequences worth remembering:

A durability guarantee that is not backed by a network round trip is worse than no guarantee, because a client will trust it.
Opcode.IsMutating() answers “is this unsafe to process from 0-RTT?”, not “does this change stored state?”. It returns false for OpPut and OpBatchPut, which were idempotent overwrites when it was written (an insert now refuses an _id_ the bucket holds). Routing by that predicate silently exempts every data write from replication. Route by !IsReadOnly() instead.

Model

Per-bucket owner with ring-placed replicas.

A bucket’s owner is chosen by hashing the bucket’s database and name together onto the ring (ring.Ring.Owner), so app/users and analytics/users are placed independently, like any two buckets. The owner is the serialisation point for writes to that bucket, and the node that names the replication sequence.
The replica set is the owner plus the next replication_factor - 1 distinct nodes clockwise (ring.Ring.Replicas), where replication_factor is the bucket’s own: the factor it declared when it was created, else the node default (HOARDDB_REPLICATION_FACTOR). With RF=3 and three members, every node holds every bucket; a bucket that declared 1 on the same cluster is held by its owner alone. See Replication Factor for the create form. show buckets and describe bucket print each bucket’s factor and whether it was declared or inherited, on a node that holds the bucket.
Reads are served by any node in the replica set that holds the bucket. A node that is not in the set, or is in the set the ring would compute for a bucket it has never held, asks the owner.

Membership changes at runtime, with no restart. Start the new node configured as a cluster member (HOARDDB_SEED_NODES, plus HOARDDB_SEED_FINGERPRINTS: see Seed pinning), then run join <address> in a CLI session on any existing member. That member proposes the change in three rounds: a majority of the current members promises to consider it, then a majority accepts it (an acceptance is recorded and acts on nothing), and only then does every member adopt the new membership epoch. When the command returns, every member that answered is on the new epoch; a member that did not answer learns it from the first node it exchanges with. At most one member set is ever adopted at an epoch, whatever fails or runs at the same time. remove-node <node-id> removes a member the same way.

What the command says, and what to do:

Success. The change is made. If the change was already made, by an earlier run of the same command that stopped, it says so and changes nothing.
“membership change not made: … nothing was offered and nothing changed”. Too few members answered, or another change from this member is running. Run nodes to see which members are down, then run the command again.
"… was offered to the members …, but this command stopped before a majority confirmed it". The change may still complete: a member that accepted it finishes it, usually within a minute, and within about five minutes while members are unreachable. Run nodes to see whether it did before you run the command again.
“another membership change (…) was in flight and is now committed …; your change (…) was not made”. A change another member started and did not finish was finished first. Run nodes, and run your command again if it is still wanted.
“the change (…) is committed …, but this node could not adopt it”. The change is made; the member you ran it on is behind and catches up by gossip once the cause it names is fixed. The command exits with an error because that member needs attention.

Every member must run the same release before you add or remove a node: a member on an older release refuses the new rounds, and the members log member <node> runs a build without this release's membership protocol; upgrade every member before adding or removing nodes. Reads, writes and replication are not affected by mixed releases. Do not move a member back to an older release while a change is running: the older release forgets what the member accepted. hoarddb_membership_changes_total{outcome} on /metrics counts the changes each member proposed, by outcome. Each node derives the ring from the epoch’s member list. Because placement must agree across nodes, the member sets are compared on every gossip exchange, and a disagreement makes the node refuse mutations rather than scatter writes across replica sets its peers disagree with.

While a join or remove-node is being learned by every node — normally under a second — a write can be refused with write_outcome: not_applied and a message saying the membership is changing. Retry it. Nodes on different membership epochs do not replicate to each other or hand each other a bucket’s copy until the one that is behind has caught up. A write whose bucket moves to another owner while it is in flight is answered with an error saying so, and write_outcome: unknown: it is stored on the node that took it and is not replicated from there, so read it back and retry it if it is missing. A node that has been removed refuses writes and says which member to connect to: at once when it is reachable as the removal commits, and after a restart too. Every write verb is refused, including claim, ack, release, extend, requeue and discard, with write_outcome: not_applied and nothing changed. It keeps the copies it holds, and the members can still copy them from it, until you empty its data directory. While it runs it asks each bucket’s new owner to take the bucket from it, including a bucket whose only copy it holds, and logs every bucket this removed node holds is served by its members once they all have: empty its data directory only after that line. It asks about one bucket and one member at a time, so on a node with many buckets the line can take a long time to appear. A bucket that was dropped after the removal is not handed back. A bucket whose name was created again before the hand-off keeps the new bucket: the removed node warns, naming the bucket, that its own copy holds writes the cluster does not have, and it does not print the line above. With auto_rebalance off it moves nothing and logs how many buckets it holds; run migrate <database> <bucket> on each bucket’s owner. A removed node that cannot reach any member does not know it was removed, and the writes it accepts until it reconnects are not kept, so do not point clients at a removed node. To add a removed node back, stop it and empty its data directory first: join refuses a node that still holds the record of its removal, and says so. join also refuses a node that does not answer: start it first.

A node started to join serves nothing until it has been added. Until join runs on a member, it answers every request that names a database or bucket with “not added yet”, naming the join to run, and status on it shows Membership: NOT ADDED YET - run 'join <address>' on a member. The members carry on as before while it waits.

Schema changes pause while a node joins

A node that joins owns some buckets before it holds them. Until every member has handed it the buckets it now holds, a schema change could create an empty bucket over the real one, so while a join settles every schema change on the cluster is refused with “unavailable, retry”: create database, drop database, create bucket, drop bucket, alter bucket, restore, and an ingest that has to create its bucket. Nothing is created, dropped or restored. Reads and writes to buckets that exist carry on, and a restore --dry-run is answered, saying that a real restore would be refused now. Data writes carry on as they always did: pop, and the lease verbs claim, ack, release and extend and the dead writes requeue and discard, are not schema changes and are not paused. The pause lifts on its own when the join settles, normally seconds after the copying finishes.

The refusal says who is joining, how many of the buckets the change moved have been copied, and what the join waits on:

Schema changes are paused while n4 joins the cluster: at least 412 buckets of this membership change have been copied to their new places. ...
Waiting on: n3, unreachable for 5m12s. n1 and n2 cannot finish handing off until n3 answers.
Ways out: start n3 again; or undo the join with 'remove-node n4' run on another member; or if n3 is not coming back, 'remove-node n3' - buckets whose only copy is on n3 then become unreachable, and a bucket of the same name created afterwards is a new, empty bucket.

A member that is down before the join has settled holds the pause: every bucket whose new set includes it waits for its answer. The ways out, in order: wait; start the down member again; undo the join with remove-node of the new node, run on another member; or, if the down member is not coming back, remove it with remove-node, at the cost the refusal states. After a remove-node the pause lifts once the remaining members have handed back what the join moved. With auto_rebalance off nothing is copied for you: the refusal and nodes list the migrate <database> <bucket> commands to run, each on the node that lacks the copy or holds one that does not follow the new owner. Run each one on the node it names, for example on n3:

[h]oardDB> migrate app Orders

That list can grow once. Once the new owner has its copy, a member that took an older copy of the bucket from the previous owner’s log finds its copy does not follow the new owner, and is named too, with a migrate of its own. Run nodes again after each round and run every command it lists, the new ones as well. The pause lifts when nodes lists no migrate command.

nodes prints the same below its table: Schema changes: allowed (every member settled at epoch 7), or Schema changes: PAUSED and the refusal’s paragraph.

Nodes cannot be rolled back to an earlier build once they have written to their log on this one. An earlier build reads the newer log entries as a damaged tail and discards them, so after a crash the copies of a bucket can silently disagree about writes that were acknowledged. Back up every node’s data directory before you upgrade (stop the node, copy the whole directory; see Backup and Restore), and go back by restoring that copy, never by starting the old build on the new data. The first time a node writes the new format it logs one INFO line saying so. A cluster is upgraded with every node stopped. Before the upgrade, stop writes and wait one minute; an earlier build has no gauge for this.

A node’s log is also written in segment format 2 from its first start on this build, and an earlier build does not discard it: it refuses to start, with an error that ends parsing sealed file header: flags must be 0x00, and changes nothing. Start the newer build, or restore the copy taken before the upgrade. Changing storage.wal_preallocate is a rolling restart of the same build: change it on one node and restart it, wait until hoarddb_replication_peers_not_in_sync is 0 and status shows the expected WAL row, then do the next.

On its first start this build moves the node’s replication positions from replication_state.bson in the data directory into the node’s store and deletes the file. If that file cannot be read, the node does not start: without the positions in it, entries that peers re-send may be applied twice. The error names the file. To start without them, move it aside (mv replication_state.bson replication_state.bson.unreadable) and start the node again. On a node whose store is encrypted, delete the moved file once you no longer need it: it holds peer ids and bucket names in plaintext.

An earlier build started on such a data directory does not refuse to start and does not warn. It cannot see the replication positions this build keeps in the node’s store, and the state file it reads is gone, so it applies every entry its peers re-send a second time: FIFO and LIFO buckets silently gain duplicate elements. To go back, restore the data directory from a backup taken before the upgrade, or remove the node from the cluster and join it again with an empty data directory.

When a bucket’s owner changes, the new owner takes over before it serves. A join or a remove-node can move a bucket to another node. The node that owned it stops accepting writes to it and finishes sending what it had; the new owner waits until it holds all of it, and only then accepts writes. Every other copy is brought to the new owner’s, or copied again from it. While that happens a write to that bucket is refused with write_outcome: not_applied. Retry it. When the new owner already holds a copy this takes well under a second; when it does not — every bucket that moves to a node that has just joined — it takes one copy of the bucket, and a join that moves many buckets copies them one after another. Other buckets keep accepting writes; while a node’s copy of one of them is being settled, a write to it can report a copies_confirmed below the replication factor. A node that leaves a bucket’s replica set keeps its copy until every member of the set holds everything it holds; until then show buckets on that node prints leaving beside the bucket.

A write is kept as long as one node that holds it is still in the cluster. A write’s response says how many copies hold it (copies_confirmed). If the node that owned a bucket is removed while it cannot be reached, the writes that only it held are lost: a write acknowledged at one, and any write whose response said copies_confirmed: 1. Remove a node while it is running whenever you can — it then hands over everything it holds. A node that was removed while it could not reach the cluster does not know it was removed. It keeps accepting writes, answers them copies_confirmed: 1, and those writes are discarded when it reconnects.

If the old owner is gone, the new owner asks every other node before it serves. It takes the most complete copy any of them holds. If one of them cannot be reached, writes to the buckets that were moving are refused until it returns or you remove it; the error names it. This happens only when the old owner and another node are both unreachable; one node down is never enough. About one bucket in n on an n-node cluster is affected (the ones the lost node owned), and reads are still answered from their copies. remove-node on the unreachable node ends the wait, needs a majority of the current members to answer, and loses the writes that only that node held. With three nodes and two of them lost there is no way out but to bring one back.

With five nodes at a replication factor of three, two lost, the way back is to remove each lost node. Run remove-node <node-id> for the first lost node on a surviving member, then for the second. Once both are removed, the three survivors take over the buckets and hold a full copy of every bucket again, and majority writes and reads are served from them. A write acknowledged by a copy on each of the two lost nodes is lost with them; a write at all is on a survivor. The rebalance is complete when nodes lists three members and each survivor holds every document of the bucket; the node log and metrics show it (nodes does not show copies).

A node’s ring identity is the address it is listed under; HOARDDB_NODE_ID is only needed when you want an identity that is not the address (written as id@host:port). Node identity is validated at startup: a node whose identity is not on the ring is a configuration error and startup fails with the member list rather than running in a state where it would never take ownership of anything. A node takes the entry listed at its own address; it takes an entry by node id only when it listens on a wildcard such as 0.0.0.0 and sets HOARDDB_NODE_ID (the host name default never counts). Give every node the same HOARDDB_SEED_NODES: members compare where each node id lives, and a node whose list places an id at another address than a member’s is never confirmed, so it serves no database, bucket or user change, and its status names the id and both addresses.

Raising a bucket’s factor

alter bucket Sessions { replication_factor: 5 } raises the number of copies a bucket keeps (HQL: alter bucket has the answers and refusals). The command is answered when the bucket’s owner has recorded the new factor, not when the new copies exist. Each member the new factor places copies the bucket from the owner by itself; you do not run anything on it. A member copies one bucket at a time per pull, and the copy is one bucket’s data, once.

Until a copy lands, the member is not sent the bucket’s writes and the bucket carries on at the copies that hold it. A write’s copies_confirmed counts the copies that hold it, so for a bucket raised from 1 to 3:

Moment	`one`	`majority`	`all`	`copies_confirmed` / `replication_factor`
both copies pending	1	1	1	1 / 3
one copy landed	1	2	2	2 / 3
both landed	1	2	3	3 / 3

Every concern is served throughout; none waits for a copy. A write with no concern named takes the new factor’s default at once: majority for a bucket whose factor is now above one. consistent get reads the bucket’s whole replica set, so after a raise it asks the new members too; wait for the owner’s show buckets row to read N/N before a read that must see every copy.

A full member. A raise onto a member whose disk is above its high-water mark is accepted. That member’s copy waits: it logs, once an hour for the bucket, why and what to do (free space, then migrate <database> <bucket> on it, or remove-node), and retries with a back-off that grows to thirty seconds. The bucket’s other copies and every other bucket carry on.

With auto_rebalance off, nothing is copied for you. The member logs the number of buckets waiting for a copy and up to ten names, each with the migrate <database> <bucket> command to run on that member.

Only one change at a time per bucket. A raise is refused while a join is settling, while the owner is still recording a membership change, and while an earlier raise has copies that have not landed on a member the new factor keeps; each refusal says when to retry.

Lowering a bucket’s factor

A decrease (alter bucket Sessions { replication_factor: 2 }) deletes the copies it removes once the remaining copies have shown that they hold everything the removed copies hold; their space is reclaimed in the background after the delete. When the cluster has fewer members than the factor, no copy is deleted: every member already holds one at both factors.

Write path

client ──▸ coordinator (any node)
              │  not the owner? forward the entry
              ▼
            owner
              1. append to WAL (fsync)      ← the entry is durable here
              2. apply locally
              3. ship the entry to the bucket's other replicas
              4. wait for the write concern
              5. answer the client
replica ──▸ 1. append to WAL (fsync)
            2. apply locally
            3. acknowledge the sequence number

An entry is the original client opcode plus its original request payload (ReplEntry). A replica applies an entry through the same Server.dispatch call a local client request takes, so a mutation has exactly one implementation and cannot drift between the client path and the replication path.

Write concern is evaluated against confirmed sequence numbers, never against local state, and it counts the copies that are in sync: the owner, and every replica that is in sync. A replica stops being in sync when its stream fails, is refused or falls behind its queue, and counts again only once the owner has sent it the log up to the tail. While it is still being sent what it missed, writes do not wait for it, and the node’s log says “replication to a peer is back in sync” when it counts again.

one — the owner’s own apply. No replica wait.
majority — a majority of the copies in sync; the owner counts as one.
all — every copy in sync.

A write at majority or all waits for the copies that are in sync — reachable and caught up. If a replica is down or behind, the write succeeds on the copies that remain, and the response says how many hold it: copies_confirmed beside replication_factor. With every replica out, majority and all are confirmed by the owner alone and answer copies_confirmed: 1. Check the number if one copy is not enough for you.

Every write’s response carries copies_confirmed and replication_factor. The CLI prints stored on N of M copies on stderr when the first is below a majority of the second (1 of 3, 2 of 5); a write a majority holds prints nothing.

Fewer copies than the replication factor

stored on N of M copies means the write is on N of the bucket’s M copies, and N is below a majority of M (1 of 3, 2 of 5). It is not an error: the write is durable on the N copies that confirmed it, and the rest catch up. The CLI cannot know which cause it is, so it points you at status and the node logs. The common causes, and the check for each:

A write that a majority holds prints no note, so on three copies you never see 2 of 3: at the default majority concern the owner answers as soon as two copies hold the write, and the third follows within milliseconds. That also means a write on 2 of 3 copies is silent when one copy is out; status and nodes are how you see a member that is down, and the write’s copies_confirmed and replication_factor fields in the response carry the exact count.

A replica is still connecting just after create bucket. The replica set is placed when the bucket is created, and a replica may not have opened its stream yet. Wait a moment and check status; the count reaches the factor on its own, with nothing to fix.
A node is down. status on the reachable nodes names it, and nodes lists it as unreachable. Start it again; what it missed is sent from the owner’s log when it returns, and the count recovers on its own.
A node is over its disk high-water mark. status shows the disk row red (disk_high_water_pct, HOARDDB_DISK_HIGH_WATER, default 85). The node refuses to copy a bucket in and logs the refusal. Free space on that node, or raise HOARDDB_DISK_HIGH_WATER; replication resumes on its own (Is this node full?).
A node refused it for another reason. The refusal is in that node’s log and names the bucket and the fix. On a Docker Compose stack, docker compose logs <node>; otherwise read the node’s log file.

status on each node is the first check: it shows Healthy: false and names the resource when one is red. The node log carries the full refusal and the step that clears it.

The first all write after a replica stops answering waits for one send to it to fail — at once if the node is down, up to half the write-concern timeout (never under 2 seconds) if it is silent — and later writes do not wait for it.

An error for a write carries write_outcome: not_applied (retry freely), applied (it is stored; do not retry a push or an $inc), or unknown. A single node says the same: a write it refuses before anything is stored (a bucket that does not exist, a permission, a size limit, a duplicate key, a type refusal) carries write_outcome: not_applied. A write that is still waiting for its copies when the write-concern timeout (5s) ends is answered TIMEOUT with write_outcome: applied and the copies that hold it; it reaches the others when they are reachable. A write the server refuses (a $inc of a text field, for example) is answered with that refusal and write_outcome: not_applied, even when a slow replica makes the wait that follows time out: nothing was written. A pop that removed an entry is the exception in three cases: its write concern is not met in time, the bucket’s owner changes while it is in flight, or one of its replicas is refused for not encrypting its store. A pop’s answer is the only copy of the entry it removed, so in those three cases the answer is a success that carries the entry, copies_confirmed below replication_factor, write_outcome: applied, and a message that says what fell short. Keep the entry; do not pop again to get it back, because a pop takes the next entry. Any other error for a pop does not carry its entry, even when the owner applied the pop: a forwarded pop whose answer is lost between nodes is UNAVAILABLE with write_outcome: unknown, and its entry may be gone (Limitations). A write answered success: false because the owner’s own store failed to apply it carries write_outcome: unknown too: its replicas apply it, and the owner applies it when it restarts. Its copies_confirmed counts only the replicas that hold it. Read it back before you retry a push. Retry an insert with upsert. An insert whose answer you did not get, or one answered write_outcome: unknown, may be stored. Sent again as an insert it is refused DOCUMENT_EXISTS if it was; sent as upsert it is stored either way, so upsert is the retry that is safe to repeat. An insert is decided once, by the bucket’s owner: two inserts of one _id_ through different nodes give one success and one DOCUMENT_EXISTS, and every replica, and every replay after a restart, keeps the owner’s answer.

This changed in this release. all used to mean every copy of the bucket’s factor, and a write at majority or all failed when the copies it named did not confirm in time, or at once when fewer nodes were running than the concern needed. It now means every copy that can confirm the write, and the response says how many did. A program that relied on all failing compares copies_confirmed with replication_factor instead.

A replica that is down does not make writes fail, however long it is down. What it missed is sent from the owner’s log when it returns.

A batch answers write_outcome per item. A BATCH request (HQL batch { ... }) is many independent writes and reads, not a transaction: each item is its own entry in its owner’s log, with its own sequence and its own wait for the write concern, and its answer carries its own write_outcome with the meaning above. One write concern, on the batch, covers every write item. The node the client reached sends each bucket owner one forwarded request carrying that owner’s items, not one per item. A batch refused whole (a malformed batch, an item the session may not run) is refused before any item runs, with write_outcome: not_applied when it carries a write. When an owner’s answer to its part of a batch is lost, its write items are unknown and its read items changed nothing, as for a lone forwarded request.

Write concern can be set per request ("write_concern": "majority" on an insert or batch insert, HOARDDB_WRITE_CONCERN for the node default). The default is per bucket: majority for a bucket whose factor is above one, one otherwise, resolved by the bucket’s owner. HOARDDB_WRITE_CONCERN sets one level for every bucket instead.

The accepted values are 1 (or one), majority and all, in any letter case. Anything else, in a request or in HOARDDB_WRITE_CONCERN, is refused: a request gets an INVALID_ARGUMENT error that lists the values and nothing is written, and a node with a mistyped setting does not start. A typo is never served as one.

Wire protocol

The cluster opcodes in the protocol spec are used, and one was added for membership changes:

GOSSIP (0x90) — announce this node, exchange member lists, verify agreement. Also the join handshake.
REPL_PUSH (0x91) — apply these entries. One frame serves both hops: with return_response the owner’s response is relayed to the client (coordinator → owner), without it the acknowledgement is the applied-through sequence (owner → replica).
REPL_ACK (0x92) — the response opcode for a replication push.
MEMBERSHIP_CHANGE (0x9E) — one round of a membership change (prepare, accept or commit), named by a field of the request. CLUSTER_PROPOSE (0x96), the vote of earlier releases, is answered with a refusal that says to upgrade every member.

Node-to-node sessions authenticate with the cluster token, which already exists on every node and is exactly the credential that authorises cluster membership (HOARDDB_CLUSTER_TOKEN, or the cluster.token file in the data directory). The transport rejects every non-auth opcode on an unauthenticated session, so a node authenticates before it can replicate. Tokens are compared in constant time against an in-memory copy. A token that does not match is checked against the file only when the file has changed since it was last read, which is what a rotation does, so a wrong token never costs a file read.

A failed write returns a defined error response rather than a handler error: the transport turns a handler error into a fatal stream reset, which would tear down a client’s session over something as ordinary as an unmet write concern.

Durability, recovery and repair

The WAL is the authority for what a node has durably accepted. Every entry is appended before it is applied, on the owner and on every replica.
On startup a node replays its WAL through the ordinary dispatch path. Replay is best-effort per entry (a create for a bucket that already exists is expected, not fatal) and runs once per process.
One sender per peer. Each node sends what it logged to each other member from one place, in the order it logged it. A write is put on that peer’s queue (1,024 entries) only while the peer is keeping up; the writer never waits for the queue. When the queue is full, or a send fails or is refused, the node empties the queue and stops queueing for that peer: everything the peer has not confirmed is in the log.
After a start, a node takes no write of its own until a majority of its peers has answered it (for three members, both of the others; for five, three of the four). Each answer says how far that peer holds this node’s log, and before its first write the node moves its log past the highest of them. A node whose log went backwards — restored from a snapshot of its VM or disk, or whose wal/ directory was lost — therefore never gives a new write a sequence a peer already holds another write at, which would make the new write look older than the one it replaced. A normal restart costs one answer from each peer, which a node exchanges anyway. Until enough peers answer, a write to a bucket the node owns, or a create database or drop database sent to it, waits as long as it would wait for its copies (the write-concern timeout, 5 seconds by default) and is then refused, UNAVAILABLE, with “Nothing was written” and the peers not yet heard from named: start them or restore the network to them, or remove a peer that is gone for good with remove-node. A plain read is not held. When the log had to move, the node logs one warning at start: “this node’s log ended at T and its peers hold it through P …”. A peer that answers only after the node has started taking writes and holds the log further moves it again, with a warning naming that peer.
A replica that missed entries — it was down, slow, or its queue was full — is repaired from the owner’s log when it answers again: the sender asks the peer whether it is there before it reads any of the log, sends the entries the peer has not confirmed in sequence order, filtered to the buckets that peer holds (a create database or drop database goes to every member), and only then sends new writes from the queue. A new write cannot overtake the repair. The acknowledgement position advances only as far as the peer says it applied. A sender whose peer does not answer tries again after half a second, then waits twice as long each time up to 30 seconds, and at once when the peer answers the node’s membership check or finishes copying a bucket from it.
A membership change starts every sender again from the log under the new placement. An entry of a bucket this node no longer owns is not sent by it after the change, and neither is one it wrote before an earlier change took the bucket away and a later one gave it back. Each such write is counted once in hoarddb_replication_entries_stranded_total when the node’s log is next read for a peer, and the node logs one warning naming the peer and up to five of the buckets. A write a replica lacks is counted once the node’s log is read for that replica, so the count is never below what any one replica lacks, and it can be above it: it also includes writes the bucket’s other copies did receive.
Replication is at-least-once: an acknowledgement lost in flight makes the sender re-send, so a receiver never applies the same sequence twice. The receiver’s position for each sender is rebuilt at start from its own write-ahead log, and checkpointed in the node’s store where the log cannot carry it; nothing is written for it per replicated frame. Without that, non-idempotent store types (FIFO, LIFO, heap, btree) would silently gain duplicate elements.
A failed send is never dropped, and nothing is sent ahead of it. This is not a nicety: the acknowledgement is a high-water mark, so an entry that is skipped while a later one is acknowledged becomes permanently invisible — the repair starts above it, and the deduplication guard rejects it if it is ever sent again. Measured before the one-sender design: with a replica down longer than its queue, 213 of 1,238 writes were never delivered to it once it returned; after an owner restart, live writes overtook the repair and up to 297 of 1,000 earlier writes never arrived. Both are delivered now.
The check that a replica’s unique index agrees with its owner’s (hoarddb_replica_constraint_disagreements_total) compares every write, live or sent from the log: the owner records in its bucket state each write it did not apply, until its log no longer holds it.
A drop reaches every member, not only the bucket’s current replica set, so a copy left on a node outside the set is dropped too. It removes exactly the bucket it names: a member that already holds a bucket of the same name created after the drop keeps it. Each member remembers the drop until no member, and no node removed in the last 64 membership changes, still holds that bucket, so a copy of a dropped bucket is never copied back.
A replica never acknowledges a write it did not apply. If the owner applied a write and a replica cannot — for example its copy of the bucket is a different bucket of the same name, or it cannot read the bucket’s declared field types — the replica refuses it, counts it in hoarddb_replica_divergences_total, and logs one ERROR naming the bucket and the owner. Replication from that owner to that replica then stops for every bucket at that write, writes at w=all to those buckets time out, and the owner’s log is kept; the owner’s retention WARN names the replica and the bucket. With auto_rebalance on (the default) the replica repairs itself: it replaces its copy with the owner’s, and replication resumes. With it off, run migrate <database> <bucket> on the replica to do the same. When the replica refused a create because it holds a different bucket of that name, dropping the bucket through its owner ends the stop too, and the replica keeps its own copy.

Is the cluster in sync?

Each node reports its own streams on /metrics (turn it on with metrics.enabled; see Configuration). Alert on two gauges:

hoarddb_replication_peers_not_in_sync is how many members this node’s stream to is stalled: the member is down, its connection hangs, it refused an entry, or this node could not queue one for it. 0 is healthy. Above 0, majority and all writes are confirmed by fewer copies (the response’s copies_confirmed says how many), and what the member misses waits in this node’s log. The gauge returns to 0 once the member answers and the log has been sent to it up to the tail; writes do not have to stop first, except writes at one faster than the member can be sent them (Limitations). Alert when it stays above 0 for longer than a restart of a node takes.
hoarddb_replication_lag_seconds is how long the member that is owed entries has gone without confirming any of them. It stays near 0 while every member keeps up, and reads 0 when every member has confirmed everything sent to it. A reading that keeps growing is a member that is not confirming. Alert when it passes the delay you accept for a replica’s reads. It counts only from the first entry sent to a member, so a member that is down and has been sent nothing reads 0 here; hoarddb_replication_peers_not_in_sync still counts it, so alert on both.

Four more say how much and why:

hoarddb_replication_lag_entries is how many entries of this node’s log the furthest-behind member has not confirmed. A member that is sent nothing, because it holds none of the buckets being written, counts 0.
hoarddb_membership_changes_total{outcome} counts the membership changes this node proposed, an operator’s or one it finished for a member that stopped, by outcome: committed, finished_another, already_committed, refused, no_majority and indeterminate. An indeterminate change may still complete; nodes says whether it did.
hoarddb_cluster_peers_unreachable is how many members fail this node’s membership check: down or cut off. A member whose replication connection hangs while it still answers that check is not in sync and is not unreachable.
hoarddb_cluster_members_joining is how many joined members this node has not yet seen every member settle. 0 is healthy. Above 0, schema changes are paused (Schema changes pause while a node joins), and nodes says what the join waits on.

After a join or a remove-node, hoarddb_handoffs_in_progress reads 0 on every node when every bucket that moved has been taken over. A node that has no copy of a moved bucket yet reads 0, and so does every node in the first seconds after the change, before it has worked out what moved: read every node, more than once. Three counters say how the hand-off went: hoarddb_handoff_copies_total (buckets a node copied whole), hoarddb_handoffs_without_previous_owner_total (buckets taken over without their previous owner, after it was removed while down) and hoarddb_handoff_entries_reconciled_total (writes an old owner delivered after it lost the bucket).

Each is one series per node. None is per bucket or per database.

The node’s log names the member. A stream that leaves the in-sync set logs one WARN, “replication to a peer left the in-sync set”, with node_id, the reason, and the bucket when the member refused one it does not hold yet. Later failures of the same stream are not logged again. When the stream returns, one INFO, “replication to a peer is back in sync”, says how long it was out (out_for) and how many entries were sent to it (entries_replayed).

While a member is out, every node’s log grows. No node prunes its write-ahead log past what every member has confirmed. So every node’s log grows until the member returns or is removed, not only the logs of the nodes whose buckets it holds. Watch hoarddb_node_data_bytes{part="wal"} beside the data directory’s free space (Is this node full?). Each node also logs a WARN, “WAL retention on this node is held”, that names the member and the fix: restart it, or remove it with remove-node if it is not coming back. The first comes at the first retention pass that finds the member out, then one every five minutes. Just after a node starts, the WARN says the node is waiting for the member’s first handshake: until every member has answered once, nothing is pruned.

Never start a copy of a running node

A node’s data directory holds its node id, its record in the member list, its TLS key and the cluster token. A copy of it (a cloned VM or disk image, a file-level backup restored beside the live node, a copied data_dir for a test node) started while the original runs would run as the same node: both would own its buckets and, if it is the cluster master for user changes, both would accept them. To add a node, start it with an empty data directory and its own HOARDDB_NODE_ID, then run join <node-id>@<address> on a member.

A copy that listens on a specific address other than the one the member list gives for its node id stops at start:

cluster: this node listens on 192.0.2.9:4433, but the member set in .../membership.json
lists its node id "n2" at 192.0.2.2:4433, another address: this looks like a copy of
that node's data directory ...

A copy listening on a wildcard address (0.0.0.0) cannot tell its address from the listed one and starts. The cluster then detects it, within about a minute: every process sends a random instance id on each membership exchange, and a member that keeps hearing two of them under one node id reports it.

The member logs one WARN, “two processes are running as node n2: …”, naming where each process was seen and both instance ids, with the fix.
status on that member shows a Duplicate: row with the same text, and nodes marks the node DUPLICATE IDENTITY.
hoarddb_duplicate_node_id_detected_total counts each detection. It is one series per node.
Both processes under the node id are told. Each shows a Duplicate: row in its own status and refuses user changes until only one runs, so there are never two cluster masters.

What to do: stop the copy, empty its data directory, and start it again with its own HOARDDB_NODE_ID before you join it. The warning clears about two minutes after one of the two processes stops: the member that saw both instances keeps the detection for one 60 s window after its last evidence, and the duplicated node keeps the member’s report for another 60 s after that. A node that restarts shows a new instance id once and is never reported.

Moving a standalone node into a cluster

A node started without HOARDDB_SEED_NODES keeps its users under its own signing key, auth.token.key in its data directory. A clustered node signs with a key derived from the cluster token instead, so every member can check every other member’s logins. A standalone data directory cannot become a cluster member in place. If you start it with HOARDDB_SEED_NODES, or with HOARDDB_REPLICATION_FACTOR above 1, it stops at start and changes nothing:

this data directory (/var/lib/hoarddb) ran as a standalone node: its users are stored
under its own signing key, auth.token.key (key f4517766), ...

Started without those settings, it runs standalone as before. To move its data and users into the cluster:

Start the node standalone again, connect with hoarddb, and note each user and their roles: user list, then user describe <name> for each.
Dump its data: hoarddb dump --address <its client address> --user <admin> --out ./standalone-dump.
Stop it. Start the cluster member on an empty data directory (a new HOARDDB_DATA_DIR), with HOARDDB_SEED_NODES set.
Restore the data into the cluster: hoarddb restore --address <a member's client address> --user <admin> --dir ./standalone-dump.
On the cluster, create each user again with the roles you noted: create user <name> roles [role@db, ...]. Each user gets a new password, because a dump does not carry users or passwords (see Backup and Restore and User management).

Keep the old data directory until the cluster has served the restored data.

Upgrading a cluster that changed membership on an older release

On releases before this one, a membership change that failed after some members had agreed to it, or two changes run at once, could leave a member on a member set that never took effect, at the same epoch as the set that did. Such a member could take buckets over as a second owner, and a later change could replace the real owner’s copy with its copy, losing writes the real owner had acknowledged. This release cannot repair that by itself, because which set took effect cannot be told for certain afterwards. nodes does not show it: once that member has caught up with later changes it shows the same epoch and set as every other. Check with nodes history, which prints a member’s membership history, one line per epoch with a short digest of the set:

epoch 3: n1 n3 n5 n9  (digest 5c1e0a7b)

Upgrade every member. The upgrade changes no membership. Make no membership change until step 6 is done.
List the nodes to compare: every member, and every node removed at or after the earliest epoch any two histories disagree on that is still running. A removed node is not listed by nodes, but it is still asked for its copies when a bucket changes owner, so a removed node that is running is as dangerous as a member. The removed nodes are the ids that appear in an epoch’s line and not in a later one. Run nodes history against each. Treat a removed node that does not answer as one on the wrong set (step 4).
Compare the lines. If every node shows the same digest for every epoch it holds, stop: the cluster did not diverge within the epochs recorded (but read the note on the 64-epoch limit below first).
For each epoch where the nodes disagree, find which set took effect, by one rule: a set fits when the set every node agrees on at the next epoch differs from it by exactly one node. If exactly one of the sets fits, it is the one that took effect, and every node that holds another set at that epoch is on the wrong set, whether it agreed to that set or learned it from another node. In every other case, stop and rebuild the cluster from a backup: no set fits; the epoch is the last one recorded; the nodes disagree on the next epoch too; or both sets fit, which is the likeliest history, when an operator ran the change that seemed to fail again. Do not decide by counting which set more nodes hold: the wrong set can spread to a majority.
Stop every node on the wrong set at once, before removing any, the removed-but-running ones included: one left running while another is removed can still hand its copy over the real owner’s. Empty the data directory of each removed one now. Then check that a majority of the current members is still running; if not, the cluster cannot remove the stopped ones, so rebuild it from a backup.
With every one of them down, for each member on the wrong set in turn: empty its data directory, remove it (remove-node, run on a running member), and wait until the removal has handed its buckets over (nodes shows schema changes allowed). When every one is removed, add each back as a new node, one at a time.

Removing a node on the wrong set while it is running loses acknowledged writes: the hand-over that the removal starts copies its copy over the real owner’s, and a running one can do the same during another node’s removal.

The 64-epoch limit is a limit on what can be checked, not on the danger. nodes history shows at most the last 64 epochs. A copy taken over on the wrong set outranks the real owner’s copy at any age when the real owner’s copy has not changed hands since, and with auto_rebalance off nothing moves it. A cluster whose history does not reach back to the first membership change it made on an older release, and which ever had a change fail or run beside another on such a release, should be rebuilt from a backup rather than checked.

After the upgrade, a member that reports a different member set at its own epoch can only be one that diverged on an older release, or a configuration error: its writes are refused with a message pointing at nodes history and this procedure.

Upgrading a cluster that already holds data: bucket placement changed

Builds before this change placed a bucket by its name alone, so every database’s users bucket shared one owner and one replica set. A bucket is now placed by its database and its name together. On a cluster with data, most buckets therefore belong on different nodes after the upgrade — about (N−1)/N of them on N nodes — and they move once, on their own:

Upgrade every node, then start them. Stop all nodes, install the new build on each, start them all. Do not run old and new builds side by side: they disagree about where a bucket lives, and nothing detects that disagreement. Upgrade every encrypted node before you admit an unencrypted one: an encrypted node on an earlier build does not check its peers and keeps replicating to an unencrypted node. See Encryption in a cluster.
On start, each node hands off what it should no longer hold. A node that holds a bucket outside the bucket’s new replica set asks the new owner and replicas to copy it, owner first. Once every member of the set holds everything this node’s copy holds, this node removes its own copy; until then show buckets on it prints leaving beside the bucket. The node logs relocating buckets this node holds outside their replica set when it starts moving and relocation of misplaced buckets finished when it is done. A member that is not up yet is retried with back-off, up to 30 seconds between tries, until it answers.
While a bucket moves, requests for it can fail. Its new owner refuses writes and reads of it until it has taken it over (“ownership of <database>.<bucket> is moving to this node … Retry in a moment”), even when it already holds a copy as a replica: that copy may lack writes the previous owner acknowledged. A refused request starts the take-over. A read sent there before its copy has started finds no bucket. Retry them; the move of one bucket is one snapshot copy.
With auto_rebalance off (HOARDDB_AUTO_REBALANCE=false), nothing moves. The node logs a warning with the number of buckets it holds outside their replica set. Run migrate <database> <bucket> on each such bucket’s new owner (nodes lists the members).

What it costs: each bucket that moves is one snapshot transfer, plus one request to each member of its replica set. A node with nothing to move pays one ring lookup per bucket it holds at start, and sends nothing.

A fresh cluster, or a single node, has nothing to move and notices nothing.

Limitations

These are real and worth knowing before relying on the cluster:

Start every founding node before you use a new cluster. On a cluster’s first start each node’s member list comes from its seed settings, and a node serves database and bucket requests only once every node in that list has answered it and agrees; until then it answers “not added yet”, and status shows Membership: NOT ADDED YET. Starting all of them is enough — they confirm each other within a gossip round. A node added later with join is confirmed by the join itself.
Eventual consistency by default. A read is served from whichever replica the client reaches. A replica that is behind (or was just offline and has not been repaired yet) can return stale or missing data. A midpoint reads as locally-consistent-per-replica, not globally consistent.
Opt-in strong consistency: consistent get reads the key from every member of the bucket’s replica set, returns the copy with the highest version, and repairs a stale or missing replica through the normal replication path. The ordinary get is unchanged — still eventual, still served from whichever replica is reached. What consistent get guarantees:
- A version is ordered by membership change first. After a join or a remove-node moves a bucket, a write through its new owner outranks every write its old owner made, so an owner change does not let an older value win.
- A key the owner has deleted is not brought back from a replica that missed the delete. When the bucket’s owner does not hold the key, its copies that the owner has already passed are answered “not found” and removed. A write made while the read runs is kept.
- While the bucket’s owner cannot be read, consistent get answers only if every other copy answered and they agree; otherwise it is refused, and the message names the owner. It returns every write acknowledged at majority or all. A write acknowledged at one, whose owner is then down, may not be returned. The cost: with the owner down and one more copy down, or the other copies differing, the read is refused until the owner is back.
- While a bucket changes owner: if the new owner already held a copy (a replica that a remove-node promoted), consistent get is refused until it has taken the bucket over. If it did not (a node that just joined), the read answers from the other copies when all of them answered and agree, and is refused otherwise.
- It is refused for the moments after a membership change in which members disagree on the membership, and while a member runs an earlier build of hoardDB. Each refusal says what to wait for; retry.
- It compares only copies of the bucket the owner serves. A copy of another bucket of the same name, created apart, is left out, and the message names the node that holds it.
Database-level DDL is not covered by write concern. create database has no bucket to place by, so it is broadcast to every member on the same ordered stream as bucket writes and not waited for. Ordering guarantees a bucket created after its database arrives after it; acknowledgement does not. It is still not waited for, and it is delivered to a member that was down: every node keeps its log until every member has confirmed it, so a member that was down or behind is sent what it missed when it answers again, whether or not the node that took the DDL restarted in between.
The WAL is truncated only up to what every member has confirmed. WAL.Truncate can be called, but the server’s retention pass (server/retention.go, every HOARDDB_WAL_RETENTION_INTERVAL seconds) prunes only segments every other member has confirmed and every local store has been synced past, and it syncs the Badger stores first. Until a member has answered since this node started, nothing is pruned. A member that is down, or whose stream is refused, keeps every node’s log from shrinking while it is down — which is exactly when the log is the repair tool. A member this node has sent nothing to for a while does not: each retention pass tells it how far nothing is owed to it, it records that before it answers, and the next pass can prune past it. WAL.FirstSequence() is the retained horizon, and the sender refuses (rather than silently skipping) any peer whose position is below it, and logs the repair: remove the node (remove-node), empty its data directory and add it again, so it is seeded by a snapshot. migrate does not re-seed a node that already holds a copy. (Before retention existed the WAL was never truncated and this footgun was latent.)
A node that comes to hold a bucket copies it whole. When a membership change makes a node a bucket’s owner or replica and it holds nothing, it copies the bucket (every engine: hash, btree, heap, fifo, lifo and blob) and is then kept current by replication. A new owner copies from the node that holds the bucket’s most complete copy; a new replica copies from the bucket’s owner once the owner has taken the bucket over. Adding a node is also not the fix for a node whose disk is filling: see the disk row of Is this node full?.
A copy kept by an earlier version can be stale. A node that leaves a bucket’s replica set — after a membership change or the placement upgrade above — now deletes its copy once every member of the set holds everything it holds, so if it rejoins that set later it fetches a fresh copy. Earlier versions kept the copy when replication_factor was above 1. A copy an earlier version left on a node that is still outside the bucket’s set is deleted after the upgrade when auto-rebalance is on (the default): on the node’s first start, or at the next membership change. A copy on a node that had already rejoined the bucket’s set before the upgrade is not detected, and can be missing every write made while the node was out. If a node left a replica set and rejoined it on an earlier version, wipe its data directory and re-add it.
A wiped node can be re-added under its old node id. Each node names every life of its write-ahead log with an id kept in wal/incarnation.bson, and its peers keep their position in that log together with the id. A node whose data directory was emptied starts a new log under a new id, so its peers apply what it sends from the start of that log, whatever they held for the old one. (Before this release a peer that remembered a higher position took the new writes for ones it already held, skipped them, and still confirmed them.)
A node restored from a file-level copy of its data directory comes back behind its replicas, and says so. Its log ends where the copy was taken, and its replicas hold writes after that point that it no longer has. It logs one line per replica, naming it: “… had applied this node’s log through P; this log now ends at T …”. Before it takes a write it moves its log past P (see Durability, recovery and repair above), so new writes replicate normally, are never taken for the ones it lost, and order above them. A peer that was not among the majority that answered at start can still hold the log further; the node then moves its log again when that peer answers and warns that the writes it took meanwhile may order below that peer’s. To make the copies identical, remove that replica (remove-node), empty its data directory and add it again. Let every replica be repaired before you change the membership. Back up and restore through dump and restore (Backup and Restore) to avoid this.
wal/incarnation.bson belongs to the log. Copy it with wal/, never without it. A node that finds its log and not that file cannot tell which of its peers’ positions are current, and does not replicate to a peer that holds one until the file is restored or the node is removed, emptied and added again; it logs the remedy. The file has no checksum: damage that leaves it readable is not detected, and the node trusts it and writes it back, so it then misjudges its peers’ positions without an error. Keep a copy of wal/ with every backup of the data directory.
A membership change needs a majority. join and remove-node commit only when a majority of the current members accept, so a cluster that has lost its majority cannot change its membership. Placement changes with membership, and the data moves on its own unless auto-rebalance is off (HOARDDB_AUTO_REBALANCE=false).
A node that encrypts its store gives no bucket’s data to a node that does not. Each node encrypts what it stores under its own key, so a bucket handed to an unencrypted node would be written there in the clear. When a membership change places a bucket held by an encrypted node on an unencrypted one, the encrypted node refuses to hand the bucket over — to the rebalance and to migrate alike — and refuses creates and writes whose replica set includes that node. It also refuses to answer a read, a dump or a write for its buckets that reaches it through an unencrypted node: connect clients to encrypted nodes. The error names the node; the data stays where it is until that node is started from a fresh data directory with encryption on, or removed with remove-node. Two things do reach every member, by design: the names of databases and buckets, and the user accounts. A node with encryption.mode: require or on refuses to add a node at join that says its store is unencrypted, and says how to fix it. See Encryption in a cluster.
The CLI does not set write concern per request yet, even though requests carry the field.
A node outside a bucket’s replica set does not hold the bucket at all. Bucket DDL is placed on the replica set rather than broadcast, so show buckets on a non-replica node lists nothing for it even though reads and writes work there (they are forwarded). With 5 members and RF=3 that is two of five nodes.
Route reads to the holders, not to every node. A forwarded read costs about 2.4× a local one (measured: 5,024/s forwarded against 11,927/s local at one read in flight), so a client that round-robins over all five members of a 5-node RF=3 cluster is slower than one that uses a single node: 65,820/s versus 74,105/s, while round-robining over the three holders gives 95,602/s (+29%). Anything in front of the port — a load balancer, a proxy, DNS — must know which nodes hold a bucket, which is why clients are expected to use the ring rather than a generic balancer. Details: bench/RESULTS.md.

Testing

server/cluster_replication_test.go starts three real TCP+TLS servers on loopback with a real ring and a real cluster token — nothing is mocked, because only an end-to-end test can show that data actually arrives.

TestClusterPropagatesWritesToEveryReplica writes 30 documents to one node and reads each one back from every node’s own storage.
TestWriteConcernFailsWhenReplicasAreDown stops two of three nodes and asserts majority and all fail while one still succeeds.
TestRestartedReplicaCatchesUp stops a replica, writes at majority, restarts it, and asserts the missed writes are repaired from the owner.
TestReplicaReceivesEntriesRetriedAfterAFailedSend stops a replica, writes at majority, restarts it, and asserts the entries arrive without the test asking for a repair, because the sender must not discard them (see the failed-send note above).
TestEntryThatCouldNotBeQueuedIsOnTheReplicaAfterItReturns, TestLiveWritesDoNotOvertakeCatchUpAfterAnOwnerRestart and TestAFullQueueNeverBlocksAWriter hold the one-sender guarantees: a replica down longer than its queue, an owner restart with live writes during the repair, and a frozen replica under 2,000 writes at one.
TestClusterReplicationFactorThreeOnFiveNodes runs 5 members at RF=3 and asserts each bucket lands on exactly 3 of them, that the other 2 hold none, that one/majority/all all succeed (write concern is measured against the replica set, not the member count), and that every node can serve a read — including the two that hold no copy.

For throughput, see bench/RESULTS.md. To reproduce a three-node cluster on one host, give each node its own ring port, spaced ten apart: a node’s internode port is its ring port plus one, so ring ports 4433, 4434 and 4435 would put one node’s internode port on the next node’s ring port and break the pins. The steps are a shared cluster token, a certificate per node (every seed must be given a fingerprint, so each certificate exists before its node starts), and then the three nodes:

export HOARDDB_ROOT_USER=admin HOARDDB_ROOT_PASSWORD='choose-a-long-password'
export HOARDDB_CLUSTER_TOKEN_FILE=/tmp/cluster.token
head -c 32 /dev/urandom | od -An -tx1 | tr -d ' \n' > /tmp/cluster.token

# a certificate per node, and the pin list built from their fingerprints
PINS=""
for n in 1 2 3; do
  mkdir -p /tmp/n$n/keys
  openssl req -x509 -newkey ed25519 -nodes -days 365 -subj "/O=hoardDB/CN=hoarddb-node" \
    -keyout /tmp/n$n/keys/server.key -out /tmp/n$n/keys/server.crt 2>/dev/null
  chmod 600 /tmp/n$n/keys/server.key
  fingerprint=$(openssl pkey -in /tmp/n$n/keys/server.key -pubout -outform DER | tail -c 32 | sha256sum | cut -d' ' -f1)
  PINS="$PINS,127.0.0.1:$((4423 + 10 * n))=SHA256:$fingerprint"
done
export HOARDDB_SEED_FINGERPRINTS="${PINS#,}"
export HOARDDB_SEED_NODES=127.0.0.1:4433,127.0.0.1:4443,127.0.0.1:4453
export HOARDDB_REPLICATION_FACTOR=3

# ring ports 4433, 4443, 4453 (internode 4434, 4444, 4454); client ports 7433, 7434, 7435
for n in 1 2 3; do
  HOARDDB_DATA_DIR=/tmp/n$n HOARDDB_LISTEN=127.0.0.1:$((4423 + 10 * n)) \
  HOARDDB_TCP_LISTEN=127.0.0.1:$((7432 + n)) ./hoarddb-server > /tmp/n$n.log 2>&1 &
done

Wait a few seconds, then write through node 1 and read through node 2:

printf 'create database app;\nuse app\ncreate bucket notes {type: hash};\ndb.notes.insert({text: "hello from node 1"})\nexit\n' | \
  ./hoarddb --insecure --user admin --password "$HOARDDB_ROOT_PASSWORD" --address localhost:7433
printf 'use app\ndb.notes.find({})\nexit\n' | \
  ./hoarddb --insecure --user admin --password "$HOARDDB_ROOT_PASSWORD" --address localhost:7434

The second command prints the document inserted through node 1.

Those addresses are loopback, which is right on one host and wrong on several. For a cluster across machines, every node sets HOARDDB_LISTEN to the address the others reach it on (not 127.0.0.1), and lists the same addresses in HOARDDB_SEED_NODES:

# on the machine 192.0.2.1 (and likewise 192.0.2.2, 192.0.2.3 with their own address)
HOARDDB_LISTEN=192.0.2.1:4433 HOARDDB_TCP_LISTEN=0.0.0.0:7433 \
HOARDDB_SEED_NODES=192.0.2.1:4433,192.0.2.2:4433,192.0.2.3:4433 \
HOARDDB_SEED_FINGERPRINTS=... HOARDDB_REPLICATION_FACTOR=3 ./hoarddb-server

A node listening on 0.0.0.0 advertises 127.0.0.1 to its peers, so on several hosts set HOARDDB_LISTEN to each host’s reachable address, or use HOARDDB_NODE_ID with id@address seeds.

A node with seeds on another machine that is still on the loopback default is refused at start, and the message names HOARDDB_LISTEN. The internode port (the node address’s port plus one, 4434 here) must be reachable between the nodes and from nothing else; see Security.

A node starts happily while its peers are still down: the ring is built from configuration, and unreachable peers are retried in the background.

Source: docs/user/replication.md in the repository.