Pre-1.0 hoardDB is pre-1.0. Expect breaking changes.

Load a file: hoardDB-cli ingest

hoardDB-cli ingest loads a BSON file or a JSON Lines file into one bucket, and creates the bucket if it is not there. It is one command, it reports what it did, and if it is interrupted you run the same command again and it carries on.

It reads two kinds of file:

• a .bson file — BSON documents written one after another, as a BSON dump tool writes them;
• JSON Lines (.jsonl, .ndjson or .json) — one document per line, in plain JSON or Extended JSON.

It does not read a JSON array, a compressed file or a directory. Each refusal names the command that fixes it (see When it says no).

Load your first file

Start a server (Getting started), then, from the directory the server runs in:

hoardDB-cli ingest users.bson --into app.users --address 127.0.0.1:7433 --insecure

--into app.users means database app, bucket users; the text before the first dot is the database. The bucket is created as a hash bucket. To create a different store type, say so:

hoardDB-cli ingest events.jsonl --into app.events --type fifo --address 127.0.0.1:7433 --insecure

A real run, against a fresh server (this one listens on port 27533; 5,000 documents, then 3,000 lines), followed by counting what is in the buckets:

$ hoardDB-cli ingest users.bson --into app.users --address 127.0.0.1:27533 --insecure
WARNING: TLS certificate verification is disabled (--insecure); the connection to 127.0.0.1:27533 cannot be trusted and may be intercepted. Use this only against a local development server.
Authenticated as admin
ingesting users.bson → app.users: 1,000 documents (19%)
ingesting users.bson → app.users: 2,000 documents (39%)
ingesting users.bson → app.users: 3,000 documents (59%)
ingesting users.bson → app.users: 4,000 documents (79%)
ingesting users.bson → app.users: 5,000 documents (100%)
ingest: users.bson → app.users — 5,000 read, 5,000 applied in 0.2s (32,848 docs/s)
  note: re-running this command applies the same documents to the same keys; the count will not change.

$ hoardDB-cli ingest events.jsonl --into app.events --type fifo --address 127.0.0.1:27533 --insecure
WARNING: TLS certificate verification is disabled (--insecure); the connection to 127.0.0.1:27533 cannot be trusted and may be intercepted. Use this only against a local development server.
Authenticated as admin
ingesting events.jsonl → app.events: 1,000 documents (32%)
ingesting events.jsonl → app.events: 2,000 documents (66%)
ingesting events.jsonl → app.events: 3,000 documents (100%)
ingest: events.jsonl → app.events — 3,000 read, 3,000 applied in 0.2s (17,338 docs/s)
  note: app.events is a fifo bucket — re-running this command appends a second copy of every document.

$ cat counts.hql
use app
db.users.count()
db.events.count()
exit
$ hoardDB-cli -address 127.0.0.1:27533 -insecure < counts.hql
Switched to database 'app'
5000
3000

(The shell also prints its banner and the sign-in line; they are left out here.)

Progress and the summary go to standard error, so --json can have standard output to itself. The connection flags are the ones dump takes: --address, --user, --password, --auth-token, --config and --insecure.

Flags may come before or after the file name.

What ends up in the bucket

Every field of every document comes back with the same value and the same BSON type it had in the file: dates stay dates, object ids stay object ids, a Decimal128 stays a Decimal128. Field order is not preserved, and it differs by store type. Nothing is renamed, and nothing else is rewritten.

A JSON Lines file has no date type of its own. {"$date": "2024-05-01T00:00:00Z"} becomes a datetime; a plain string such as "2024-05-01T00:00:00Z" stays a string.

The key of each document

A hash, btree or heap bucket is keyed by a string. Ingest builds that key from the document’s _id and stores it in _id_, leaving _id as it was:

Keys are not tagged with their type, so the number 5 and the string "5" are the same key, and one overwrites the other. Ingest counts the overwrites it can see inside a batch (overwritten) and warns once when a file mixes _id types.

A fifo or lifo bucket is not keyed: documents are pushed in file order, and the last document in the file is the head of a lifo bucket. Their _id is not touched. A heap bucket is keyed like a hash bucket and every document goes in at priority 0.

When it says no

Ingest stops and tells you the fix rather than guessing. Each of these leaves the target as it was.

• A JSON array, or one pretty-printed object — jq -c '.[]' events.json > events.jsonl, then ingest events.jsonl.

• A gzip file (found by its first bytes, whatever its name) — gunzip -k users.bson.gz.

• A blob bucket — a .bson file holds documents, not payloads. Load the documents into a hash bucket, or keep the files on your own storage with one small document per file (Example: large file stores).

• A bucket of another type than the one you asked for with --type — ingest never changes an existing bucket’s type. Leave --type out to fill it as it is:

  $ hoardDB-cli ingest users.bson --into app.events --type btree ...
  ingest: Bucket "events" in database "app" is type fifo; ingest was asked to create it as type btree. Ingest never changes an existing bucket's type. Run 'hoardDB-cli ingest users.bson --into app.events' to fill it as a fifo, or, in the shell, run 'use app' and then 'drop bucket events' first if you meant to replace it.
  

• A bucket with an expiry index — documents older than the expiry are deleted within one sweep after they are loaded, and a string in the date field never expires. Pass --allow-expiring if that is what you want.

• A document it cannot key — a double, a Decimal128, a date, a boolean, a null, a document or an array as _id, an empty string, or an _id_ that is not a string. --id-as extjson keys such a document by its Extended JSON text instead. --on-error skip loads the rest.

• A line with more than one value, text after the value, or a repeated key — it names the line number. A missing newline between two concatenated files is the usual cause.

• A document over 33.5 MB, the most a request can carry.

• A bucket that does not exist, with --no-create.

Creating a bucket needs the db-manager role on the database; filling one that exists needs only writer. The messages say which, and quote the grant command.

Skipping bad documents

By default ingest stops at the first document it cannot load, after loading everything before it. To load the rest and keep the bad ones:

hoardDB-cli ingest users.bson --into app.users --on-error skip --rejects users.rejects.bson

A real run over a four-line file whose second line has a double as _id and whose fourth has an empty string:

$ hoardDB-cli ingest mixed.jsonl --into app.mixed ...
ingest: mixed.jsonl line 2: _id is a double and hoardDB keys documents by a string. Pass --id-as extjson to key such documents by their JSON text, or --on-error=skip to import the others and list the skipped ones in a file (--rejects). Re-exporting with a string or object-id _id also works.
ingest: mixed.jsonl → app.mixed — 2 read, 1 applied in 0.1s (16 docs/s)

$ hoardDB-cli ingest mixed.jsonl --into app.mixed --on-error skip --rejects mixed.rejects.jsonl ...
ingest: mixed.jsonl → app.mixed — 4 read, 2 applied, 2 skipped in 0.1s (25 docs/s)
  skipped: 2 document(s) (lines 2, 4); rejects in mixed.rejects.jsonl

$ cat mixed.rejects.jsonl
{"_id":3.5,"n":2}
{"_id":"","n":4}

The rejects file holds each skipped document exactly as it was in the input, so it is itself a valid input: fix the problem (or pass --id-as extjson) and ingest it. It is created with mode 0600 and never overwritten. The summary names skipped documents by position, never by content.

A unique index that refuses one document refuses the whole batch it was in. With --on-error skip, ingest sends that batch again one document at a time so that exactly the violating documents are skipped.

Interrupted? Run it again

A run writes a resume file next to the input, users.bson.hoardingest, before it sends each batch and again once the batch is confirmed. Press Ctrl-C once and the run finishes the batch in flight, records where it is and exits with code 4. Press it twice and it exits at once. Either way, running the same command again resumes.

$ hoardDB-cli ingest users.bson --into app.bigusers --state s4.state --address 127.0.0.1:27533 --insecure   # Ctrl-C after a second
...
ingesting users.bson → app.bigusers: 86,000 documents (21%)
ingest: interrupted. Every document before byte 8,835,780 is in app.bigusers; run the same command to resume from there (resume file: s4.state).
ingest: users.bson → app.bigusers — 86,000 read, 86,000 applied in 1.0s (85,938 docs/s)

$ hoardDB-cli ingest users.bson --into app.bigusers --state s4.state --address 127.0.0.1:27533 --insecure
...
ingesting users.bson → app.bigusers: 400,000 documents (100%) · 87,719 docs/s
ingest: users.bson → app.bigusers — 400,000 read, 400,000 applied in 3.6s (86,257 docs/s)
  note: re-running this command applies the same documents to the same keys; the count will not change.

$ hoardDB-cli -address 127.0.0.1:27533 -insecure < counts.hql     # db.bigusers.count()
400000

(Lines marked ... are the connection banner and earlier progress lines, left out here. The second run started at byte 8,835,780, and the count is exactly the 400,000 documents in the file.)

What a resume can promise depends on the store type:

When a reply is lost (a timeout, a dropped connection) ingest cannot know whether the server applied the batch, and exits with code 8. On a keyed bucket the next run simply sends it again. On a fifo or lifo bucket it refuses, and says so:

The last run stopped with a batch of 500 document(s) (bytes 1,288,490,188 to 1,290,588,340) in doubt: it may already have been applied. app.events is a fifo bucket, so re-sending it appends a second copy. ... then either pass --accept-duplicates to re-send it, or --from-offset 1290588340 to skip it.

At most that one batch can be added twice.

A resume refuses to go on when the situation it was interrupted in has changed:

• the input changed (a different length, or a different first megabyte) — it names the file and both lengths;
• the server address is different;
• the bucket’s indexes are different;
• the bucket now holds fewer entries than the interrupted run saw in it — the usual cause is a bucket dropped and recreated, and resuming would skip everything before the offset without a word.

--resume-anyway overrides the last three. --from-offset N starts at byte N of the file, which is how you resume deliberately after editing the file after the point where it stopped. Deleting the resume file starts over: on a keyed bucket that puts everything again; on a fifo or lifo bucket it appends a second copy of everything already loaded.

What the check cannot see: a bucket that was dropped, recreated and then filled by someone else to at least the number of entries the run had seen, or a different server that answers at the same address.

A run that finished is remembered. Ingesting the same file again into a fifo or lifo bucket is refused unless you pass --again.

--no-resume keeps no resume file. Reading standard input (-, with --format) keeps none either, and says so. If the resume file cannot be written, ingest refuses before it connects and names --state and --no-resume.

Two ingests of one input cannot run at once: a .lock file beside the resume file stops the second. If a run was killed outright, the lock is left behind; delete it, as the message says.

A dry run

--dry-run connects, checks the target and the file the way a real run would, reads the whole file, and writes nothing: no documents, no bucket, no resume file. It reports how many documents it would load and skip.

Machine-readable output

--json prints exactly one JSON object on standard output and keeps the text on standard error:

{
  "ok": true,
  "partial": false,
  "dry_run": false,
  "input": "users.bson",
  "format": "bson",
  "database": "app",
  "buckets": [
    {
      "bucket": "users2",
      "store_type": "hash",
      "created": true,
      "documents_read": 5000,
      "documents_applied": 5000,
      "documents_skipped": 0,
      "documents_overwritten": 0,
      "ids_minted": 0,
      "idempotent": true,
      "batch_budget_final": 2097152
    }
  ],
  "documents_read": 5000,
  "documents_applied": 5000,
  "documents_skipped": 0,
  "bytes_read": 502780,
  "elapsed_ms": 127,
  "resume_file": "",
  "in_doubt_batch": null,
  "indexes_reported": [],
  "warnings": [],
  "errors": []
}

(The object is printed on one line; this one was run with --no-resume, so resume_file is empty.)

errors lists skipped documents as {"document": 2, "reason": "_id is a double"}: the position (a line number for JSON Lines), never the content. When a run stops early, stopped_because holds the message.

Exit codes

Before this release a run that finished with skipped documents exited 2, the same number as a refusal, so a script could not tell “stopped” from “finished with skips”. It now exits 3, the number dump and restore already use for “some applied, some not”. A script that treated any non-zero exit as failure needs no change; one that matched 2 for skips needs to match 3.

Two connection changes in this release. A run with no credentials to send exits 1 (usage); it used to exit 7. And ingest no longer asks whether to trust a server it has not met, even on a terminal: it refuses. Connect once with hoardDB-cli and confirm the fingerprint to record the server, or pass -insecure against a development server.

Sizes, and what the store commits at once

A batch is applied all-or-nothing, and the store commits only so much at once (about 20 MB by default, 5 MB on a node set to the smallest memtable, counting the index entries each document adds). Ingest sends batches of up to 2 MiB of documents, and if the store still refuses one it halves the batch and tries again, keeping the smaller size for the rest of the run (batch_budget_final). A single document whose indexed fields alone make one write too large is refused with the fix: index a smaller field, or load without the index and declare it afterwards.

What ingest does not carry over

• Indexes. Indexes can only be declared when a bucket is created. Ingest never reads or applies index definitions; if a users.metadata.json sits beside users.bson, ingest says so and leaves it alone. Create the bucket first, with the indexes you want (Indexes), then ingest into it.
• Users and roles, and blob payloads.
• Elements of nested buckets. Ingested documents carry none.
• Field order (above).

Three behaviours of the write path apply to ingested documents exactly as they do to any write, and are limits to know about:

• A sub-document that is exactly {"$$now": true} is replaced by the current date when it is written.
• A unique index on a dotted path such as a.b does not look inside nested documents, so it sees no value there.
• Keys are not tagged with their type (above).

Every flag

Source: docs/user/ingest.md in the repository.