Append-only time-series storage for Yuneta. Records are grouped into topics, keyed by a primary id, and indexed for fast access by time range or row id.
Higher-level stores built on top¶
tr_msg2db— dict-style message store (one value per key).tr_queue— message queue with at-least-once semantics (used by the MQTT broker).TreeDB — graph memory database with hook/fkey relationships. See the
tr_treedbAPI and the TreeDB crash course for the full rules.
Core API (selected)¶
tranger2_startup(...) / tranger2_shutdown(...)
tranger2_open_topic(...) / tranger2_close_topic(...)
tranger2_append_record(...)
tranger2_open_iterator(...) / tranger2_iterator_get_page(...) / tranger2_close_iterator(...)
tranger2_open_list(...) / tranger2_close_list(...)Persistence semantics¶
Every call to tranger2_append_record() writes one record to disk,
increments the per-key g_rowid (monotonic, never resets) and i_rowid
(row position in the .md2 index file).
Limits¶
timeranger2.h defines two limits for values from outside: a
record’s time, and a version.
TRANGER2_MAX_TIME (0x00000FFFFFFFFFFF, 17592186044415) is the
largest time that an md2 row keeps, as __t__ and as __tm__. The row
keeps 44 bits of the time. The 16 bits above them hold the user flag and the
system flag, so tranger2_append_record() writes
the flags over the high bits of a larger time. Then the record keeps another
time than the name of its file. In seconds the last time is
559444-03-08T09:40:15 UTC. In milliseconds (sf_t_ms, sf_tm_ms), it is
2527-06-23T06:20:44.415 UTC. The time of the tkey is a time too, and a
string tkey is a DATE with no year limit (approxidate_careful() reads
"@99999999999999 +0000" as that many seconds, times 1000 on an sf_tm_ms
topic). tranger2_append_record() keeps a time
that does not fit CUT, as it always did, and logs an ERROR since SDK 7.27.0.
A caller that takes times from outside asks first, with
tranger2_record_tkey_time() for the tkey:
C_TRANGER’s add-record (see C_TRANGER), and
the treedb’s treedb_create_node() and
treedb_update_node() refuse such a record:
BOOL fits = TRUE;
tranger2_record_tkey_time(topic, record, &fits);
if(__t__ < 0 || (uint64_t)__t__ > TRANGER2_MAX_TIME || !fits) {
gobj_log_warning(gobj, 0,
"function", "%s", __FUNCTION__,
"msgset", "%s", MSGSET_PARAMETER,
"msg", "%s", "a time that a record cannot keep, refused",
NULL
);
return -1;
}TRANGER2_VERSION_MAX (0x7FFFFFFF, 2147483647) is the largest
topic_version or schema_version that a command takes from outside. A
version goes up one at a time, and save-schema writes the version plus 1,
so a version near the end of a json_int_t overflowed there. The code keeps
and compares versions as json_int_t, whatever their size. It is asked by
tranger2_topic_var_fault() and by
C_TREEDB’s open-treedb and create-topic for a peer’s schema, and a
peer’s save-schema is refused when it tries to raise the schema_version or
a topic_version past it (since SDK 7.27.0). So a treedb a peer opens AT
the maximum cannot save an edit again: give a peer’s schema versions far
below it.
# from a peer: refused
command-yuno id=<id> service=treedbs command=create-topic treedb_name=treedb_x topic_name=t2 topic_version=2147483648 cols='{"id":{"header":"Id","type":"string","flag":["persistent"]}}'
# -> -1 <role^name>: cannot create topic 't2' in treedb 'treedb_x': a topic_version that is not an integer from 0 to 2147483647
# a peer's treedb opened at "schema_version": 2147483647, and a column edited in __system__
command-yuno id=<id> service=treedbs command=save-schema treedb_name=treedb_x
# -> -1 <role^name>: cannot save the schema of 'treedb_x': the version 'schema_version' would pass 2147483647, the largest version a command can publishFilesystem watcher¶
src/fs_watcher.c implements an inotify-based watcher (fs_event_t)
used by root-linux/C_FS and by the stores themselves to react to
on-disk changes. See the fs_watcher page in the sidebar.
CLI companions¶
| Tool | Purpose |
|---|---|
utils/c/tr2list | List records in a topic |
utils/c/tr2keys | List keys in a topic |
utils/c/tr2search | Search by filter |
utils/c/tr2migrate | Migrate from legacy timeranger v1 |
utils/c/list_queue_msgs2 | List a tr_queue queue |
utils/c/msg2db_list | List a tr_msg2db topic |
utils/c/treedb_list | List TreeDB nodes |
Tests¶
tests/c/timeranger2, tests/c/tr_dt_unknown, tests/c/tr_msg,
tests/c/tr_msg2db, tests/c/tr_queue, tests/c/tr_treedb and the other
tests/c/tr_treedb_* directories (tr_treedb_load_failed,
tr_treedb_failed_save, tr_treedb_link_events, tr_treedb_files,
tr_treedb_hook_hygiene, ...), and, through C_TREEDB,
tests/c/c_treedb_literal_wins. The test suite lists
them, and each directory has a README.md.
Source code¶
Function reference¶
tranger2_append_record()¶
Appends a new record to a topic in the TimeRanger database. Master-only. If
__t__ is zero a timestamp is assigned (milliseconds when the topic is sf_t_ms,
else seconds). The new record’s metadata is returned through the required
md_record_ex out-parameter — the return value is a status code, not the
metadata. jn_record is owned (consumed, even on error).
int tranger2_append_record(
json_t *tranger,
const char *topic_name,
uint64_t __t__,
uint16_t user_flag,
md2_record_ex_t *md_record_ex,
json_t *jn_record
);Parameters
| Key | Type | Description |
|---|---|---|
tranger | json_t * | Pointer to the TimeRanger database instance. |
topic_name | const char * | Name of the topic where the record will be appended. |
__t__ | uint64_t | Timestamp of the record. If set to 0, the current time is used. |
user_flag | uint16_t | User-defined flag associated with the record. |
md_record_ex | md2_record_ex_t * | Pointer to the metadata structure of the record. This field is required. |
jn_record | json_t * | JSON object containing the record data. Ownership is transferred to the function. |
Returns
Returns 0 on success, or a negative value on failure.
Notes
The function makes sure that the record is appended to the specified topic in tranger2_startup(). If the topic does not exist, it must be created using tranger2_create_topic() before calling this function.
A time past the 44 bits of an md2 row is kept CUT, and said. The row
keeps 44 bits of __t__ and of __tm__ (TRANGER2_MAX_TIME).
The flags use the other bits. A __t__, or a time of the tkey
(tranger2_record_tkey_time()), that does
not fit is appended cut, as it always was, and logged as an ERROR (without a
stack, since SDK 7.27.0, where before it was cut without a word): “Record time past what
an md2 row keeps (44 bits): appended CUT”, with topic, key, __t__,
__tm__ and tm_wrapped (the multiplication by 1000 of a date of an
sf_tm_ms topic passed 64 bits). The append returns 0. A caller that takes
the time from a peer refuses the record before the append:
/* topic `raw_t`, tkey `when`, sf_string_key */
tranger2_append_record(tranger, "raw_t", 0, 0, &md,
json_pack("{s:s, s:s}", "id", "r1", "when", "@99999999999999 +0000"));
/* 0: appended, its __tm__ cut; ERROR "Record time past what an md2 row keeps (44 bits): appended CUT" */A tkey that is a text is read by
tranger2_time_of_json_text(), the rule
of a text time bound: "" is no tkey value (0), the text of a JSON integer
is that integer in the unit of tm, a text written as another number (" 5",
"5 ", "10.5", "007", "1,5", "5 5", "1,5 days ago") is refused, a string with a NUL inside
("5\u0000junk") is refused, and any other text is a date. A text
that is refused is refused with a WARNING (the text is from a peer or a
user) and nothing written: “Cannot append record, its tkey is a text that
is no date”, with why, topic, tkey and the first 64 bytes of the text
in value. The append returns -1, and the trace shows the first 256 bytes
of the record, not all of it. The same in
tranger2_append_record_text(). Up to SDK
7.29.3 approxidate() read a text that is no date as NOW, in silence, and
"5" as the 5th of this month. On an sf_tm_ms topic, "1761000000000" was
stored as NOW times 1000 (up to 7.29.2 as the year 57773, cut). Now it is the
integer 1761000000000, as a json integer tkey is.
/* topic `raw_tm_ms`, tkey `tm`, sf_int_key|sf_tm_ms */
tranger2_append_record(tranger, "raw_tm_ms", 0, 0, &md,
json_pack("{s:I, s:s}", "id", (json_int_t)2, "tm", "garbage"));
/* -1, nothing written; WARNING "Cannot append record, its tkey is a text that is no date",
why "a text that is no date" */
tranger2_append_record(tranger, "raw_tm_ms", 0, 0, &md,
json_pack("{s:I, s:s}", "id", (json_int_t)2, "tm", "10.5"));
/* -1, nothing written; the same WARNING, why "a number that is not the text of a JSON integer" */
tranger2_append_record(tranger, "raw_tm_ms", 0, 0, &md,
json_pack("{s:I, s:s}", "id", (json_int_t)2, "tm", "1761000000000"));
/* 0: md.__tm__ 1761000000000 */
tranger2_append_record(tranger, "raw_tm_ms", 0, 0, &md,
json_pack("{s:I, s:s}", "id", (json_int_t)2, "tm", "2000-01-03T12:00:00+0000"));
/* 0: md.__tm__ 946900800000 */Reading the metadata back. The metadata of the new record is returned in
md_record_ex, and that is where a caller reads it: md.g_rowid is the
key’s global rowid (the one that never resets), md.rowid the row’s position
inside its .md2 file (i_rowid), then __t__, __tm__, __offset__,
__size__, system_flag, user_flag. Hand the function your only reference:
md2_record_ex_t md;
if(tranger2_append_record(tranger, "topic", 0, 0, &md, record) == 0) {
json_int_t g_rowid = (json_int_t)md.g_rowid;
}A string of a record can hold NUL characters. json_dumps() writes each
one as the escape \u0000, and a read hands it back (the content is loaded
with JSON_ALLOW_NUL). A consumer that reads the string with kw_get_str()
gets the C string up to its first NUL, but json_string_length() gives the whole
length:
json_t *rec = json_pack("{s:s, s:s#}", "id", "A", "blob", "a\0b", 3);
tranger2_append_record(tranger, "topic", 0, 0, &md, rec);
/* a load hands {"id": "A", "blob": "a\u0000b"}: json_string_length() is 3 */A content that is not json when it is read logs “Bad data, the content of the
record is not json” (a CRITICAL, with topic, key, __t__, __offset__,
__size__, the jansson error and its position), and the record is not
handed. A content that this process has not the memory to parse logs “Cannot
read the record, this process has not the memory to parse its content
(MEM_MAX_BLOCK)” (the same fields, and the jansson error is “not enough
memory”), and the record is not handed either. The largest memory block
(gbmem_get_maximum_block()) is set by MEM_MAX_BLOCK for each yuno, and a
content smaller than that block can need more to parse: jansson doubles the
buffer of a string as it reads it, and the table of an array or an object. A
string of 40 000 bytes does not parse with a block of 64 KiB, nor does an
array of 10 000 numbers. A yuno with a larger block reads the record. Up to
7.25.4 such a string crashed the process (jansson wrote past its buffer, see
the lexer fix of Yuneta’s jansson in kernel/c/linux-ext-libs/HACKS.md).
The function does not add a __md_tranger__ to the record for the caller.
It adds one (g_rowid, i_rowid, t, tm, offset, size,
system_flag, user_flag, the same dict a load attaches) only to the record
it hands to a realtime list that takes it: one that wants the key and is not
an only_md feed (an only_md feed is handed NULL). A list open on another
key does not make the append build it. Nor is a __md_tranger__ ever stored:
one the record carries in, from an earlier append or from a load, is the
metadata of THAT record and is dropped before the content is written.
The record is owned by the function, and these changes are made to the
object itself, not to a copy: a list hands it on without copying it, on the
hot path of every append. So a caller that keeps a reference of its own sees
them, and reads the metadata from md_record_ex anyway:
json_incref(record); // the caller keeps one
tranger2_append_record(tranger, "topic", 0, 0, &md, record);
// No list takes the key: record has no __md_tranger__ (a carried one is gone)
// A list takes the record: record HAS the list's fresh __md_tranger__
// On an sf_rowid_key topic: record also has "__rowid__"
json_decref(record);A __t__ that belongs to an earlier file. The record goes to the file that
its __t__ selects (through the topic’s filename_mask), even if that file is
not the newest one. A key’s global rowid counts the records in the order of its
files, so the new record takes its place in that order, and every record of
the later files moves one place up:
/* filename_mask "%Y-%m-%d"; key 1 has A (2000-01-01), B (2000-01-02) */
tranger2_append_record(tranger, "topic", t_2000_01_01 + 10, 0, &md, jn_c);
/* C gets g_rowid 2, and B is now 3: an iterator serves A, C, B */The g_rowid in md_record_ex is its place, the same number a reload gives
(and stores in the loaded record’s __md_tranger__). If you keep global rowids, for example as a page position, a
record written into an earlier file makes the rowids of the later records
stale. Records written with __t__ = 0 (now) always go to the last file, and
move nothing. Until 2026-09-15 the cache in memory disagreed with the disk
until the next reload: it served the wrong record, and a follower re-published
the whole file.
An append that fails is not half written. The content (.json) is written
first and the md2 row after. When the md2 cannot be opened or created, sought
or written, the function returns -1 and cuts the content file back to where
the record began (a row written in part is cut from the .md2 too): the file
keeps no record that no row names. If the cut fails as well, it logs “Cannot
cut back the content of an append whose md2 row was not written: the content
file keeps a record no row names”. A kill or a power cut between the two
writes still leaves that shape. The next open ignores it with a warning (see
After a restart too, under tranger2_open_iterator()).
Up to 7.25.4 the content was left behind.
The cut comes BEFORE the critical that reports the failure. That matters with
the exit bit of the tranger’s on_critical_error (LOG_OPT_EXIT_ZERO, 2,
the default of C_TRANGER, and what C_TREEDB passes with its
exit_on_error): gobj_log_critical() then ends the process inside the log
call, and nothing after it runs. When the md2 cannot be opened or created, the cause is
logged without leaving, the content is cut back, and then one more critical
exits:
CRITICAL create_file: Cannot create json file path=<store>/items/keys/A/2000-01-06.md2
CRITICAL get_topic_wr_fd: Cannot open file to write path=<store>/items/keys/A/2000-01-06.md2
CRITICAL tranger2_append_record: Cannot append record, its md2 file cannot be opened: its content
was cut back topic=items key=A file_id=2000-01-06 <- exit(0) here, with 2The same order holds for a content write that fails part way: its part is cut back before the critical.
A write that stops part way (the file size limit, a full disk) is not said as
a write FAILED with an errno: write() returned a count, and errno says
nothing then. It is said as a SHORT write, with the bytes written and the bytes
expected. A read that returns fewer bytes than a row is said as a SHORT read
the same way:
CRITICAL tranger2_append_record: Cannot append record, short write of its content: the file size
limit or the disk is full topic=items written=13 expected=45
CRITICAL tranger2_append_record: Cannot save record metadata, short write: the file size limit or
the disk is full topic=items written=13 expected=32
CRITICAL read_md2_row: Cannot read a record of md2 file, short read
path=<store>/items/keys/A/2000-01-02.md2 row=last offset=64 read=16 expected=32The same holds for the files that are replaced whole (topic_var.json,
topic_cols.json) and for the zero wipe of
tranger2_delete_instance():
ERROR replace_json_file: Cannot replace topic_var.json, short write: the file size limit or
the disk is full path=<store>/items/topic_var.json
CRITICAL tranger2_delete_instance: Cannot zero the payload, short write: the file size limit or
the disk is full topic=items key=A remaining=53 written=4 expected=53A row always goes on a row boundary. When the md2 row of a failed append
was written in part and the cut of that part failed too (“Cannot cut back the
md2 of an append whose row was not written whole: its size is not a whole
number of rows”), the md2 ends in a torn row. The next append cuts the md2
back to its whole rows first, with the same warning the open gives (see A
.md2 whose last row is torn under
tranger2_open_iterator()), and writes its row
there:
WARNING tranger2_append_record: md2 file of the key ends in a part of a row: an append that was
never acknowledged was cut back topic=items key=A file_id=2000-01-02
old_size=141 new_size=128If that cut fails, the append is refused with -1 and its content is cut back
(“Cannot append record, its md2 file ends in a part of a row that cannot be
cut back: the append is refused”). The whole rows stay readable, and the next
append tries the cut again. The append cuts only a tail that is a torn row
after good rows, the same check as the open (see
tranger2_open_iterator()). For any other tail it
does not cut: it is refused with -1, its content is cut back, the md2 does
not change, and the file is flagged as the open flags it. Every load of the
key in the same process says load_failed, and the next append into the file
counts it again first, so it is refused too (see below) until the file is
repaired. The logs, after the CRITICAL that names the shape:
ERROR flag_file_unreadable_at_append: md2 file of the key flagged unreadable at
an append: every load of the key says load_failed
topic=items key=A file_id=2000-01-02
CRITICAL tranger2_append_record: Cannot append record, its md2 file ends in a part
of a row that must not be cut back: the append is refused
topic=items key=A file_id=2000-01-02 md2_size=161The file is flagged BEFORE that CRITICAL. With the exit bit of
on_critical_error (the default, see above) the yuno ends inside the log call,
and the next start reads the file again and flags it at its open. So the flag
in memory matters only when on_critical_error does not exit.
When the check cannot RUN (the .json cannot be opened, for example with
EMFILE or ENFILE, or cannot be read, or there is no memory), it found
nothing wrong with the file. The append is refused with -1 and its content is
cut back, but the file is NOT flagged: the whole rows stay readable, and the
next append checks again. With the default on_critical_error the refusal’s
CRITICAL exits the yuno, and the next start checks the file at its open (and
flags it when the check still cannot run). The CRITICAL of the check comes
first (one of the three listed in
tranger2_open_iterator()), then the refusal:
CRITICAL check_torn_md2_tail: Cannot check the torn tail of a md2 file, its content file
cannot be read: not cut topic=items key=A file_id=2000-01-02
path=<store>/items/keys/A/2000-01-02.json errno=24 serrno="Too many open files"
CRITICAL tranger2_append_record: Cannot append record, the torn tail of its md2 file
cannot be checked now: the append is refused, the file is not flagged
topic=items key=A file_id=2000-01-02 md2_size=161Up to 7.25.4 nothing cut a torn row back: an append wrote its row after the torn bytes, and the cache build left the whole file out of the key, so no read found that row.
An append into a file flagged unreadable (a .md2 that the cache build was
not able to count, see the same section) counts the file again first. If it can be read
now, it gets its cell, the flag of that file goes (“md2 file of the key
readable again: it is counted, and the key is not flagged for it”), and the
append goes on. If not, the append is refused with -1 and writes nothing
(“Cannot append record, its file is flagged unreadable: its row would follow
rows no cell counts”). Without this refusal, its row is counted as the file’s first,
and a load reads an OLD row of the file in its place.
/* key A: the md2 of day 2 could not be opened at the open (mode 0000) */
tranger2_append_record(tranger, "topic", t_day2, 0, &md, rec1); // -1, nothing written
chmod(path_md2_day2, 0660);
tranger2_append_record(tranger, "topic", t_day2, 0, &md, rec2); // 0: day 2 counted,
// A loads whole againThe file counted again gets its cell in the MIDDLE of the key, and every
global rowid after it moves up by the rows of the file. The iterators open on
the key follow: an unfiltered one takes its segments again at its next page. A
FILTERED one (a paging iterator with an index, see
tranger2_iterator_get_page()) takes its
segments and its index again at once, as a new open builds them now. So
its index gains the rows of the file, with their new rowids. Like every append
after the open of a filtered iterator, the append that caused the count is not
in its index. For example:
/* key A: v1 (day 1), v2 (day 2, flagged), v3 (day 3), v4 (day 4) */
it = tranger2_open_iterator(tranger, "topic", "A",
json_pack("{s:I}", "from_t", (json_int_t)t_day1), NULL, "pager", "me", NULL, NULL);
/* its page: total_rows 3, v1 v3 v4 */
chmod(path_md2_day2, 0660);
tranger2_append_record(tranger, "topic", t_day2, 0, &md, rec_v9); // counts day 2
/* its page: total_rows 4, v1 v2 v3 v4 */tranger2_append_record_text()¶
Appends a record whose content is already json TEXT, for example a frame as a peer sent it. The text is written as it is: the function does not parse the whole record and does not dump it again. Master-only. For a record of about 1 KB, this removes the parse, the free and the dump that tranger2_append_record() does with a json: about 18 µs per record on a Xeon E-2386G.
The store reads what it needs from the text with json_scan_object_members(). It makes sure that the text is one json object that a read of the record can parse (json_loadb() with JSON_ALLOW_NUL). It takes the pkey and the tkey from the text, as tranger2_append_record() takes them from a json. A text that it refuses is not written. Thus a topic cannot hold a record that nobody can read back.
int tranger2_append_record_text(
json_t *tranger,
const char *topic_name,
uint64_t __t__,
uint16_t user_flag,
md2_record_ex_t *md_record_ex,
const char *content,
size_t content_len
);Parameters
| Key | Type | Description |
|---|---|---|
tranger | json_t * | Pointer to the TimeRanger database instance. |
topic_name | const char * | Name of the topic. The topic must be keyed by a string (sf_string_key) or by an integer (sf_int_key). |
__t__ | uint64_t | Timestamp of the record. If set to 0, the current time is used. |
user_flag | uint16_t | User-defined flag associated with the record. |
md_record_ex | md2_record_ex_t * | Required. The function writes the metadata of the new record here. |
content | const char * | content_len bytes of json text, followed by a NUL: content[content_len] must be 0. Not owned, not modified. |
content_len | size_t | The number of bytes of the text, without the NUL. |
Returns
0 on success, -1 on error. Every error is logged.
Example
/*
* topic "raw_events": pkey "id" (sf_string_key), tkey "tm"
*/
const char text[] = "{\"id\":\"DEV-1\",\"tm\":1700000001,\"m\":{\"v\":229.0}}";
md2_record_ex_t md;
if(tranger2_append_record_text(tranger, "raw_events", 0, 0, &md, text, sizeof(text) - 1) < 0) {
// Error already logged: nothing was written
return -1;
}
/* md.g_rowid: the key's global rowid; md.__tm__ is 1700000001 */Notes
Everything else is the same as in tranger2_append_record(): the checks of the key (not empty, at most NAME_MAX, not a path), the write (one content and one md2 row, and the same cut back of a failed write), the cache and the realtime lists. When a realtime list wants the record, the function parses the text one time for all the lists.
These texts are refused with an ERROR, and nothing is written:
A text that is not one json object: “Cannot append record, its content is not one json object”.
A text that is not followed by a NUL: “Cannot append record, content NULL or not followed by a NUL”.
A topic keyed by rowid (
sf_rowid_key): its records get a"__rowid__"member, and a text cannot get one. Usetranger2_append_record().A text that has a
"__md_tranger__"member. That member is the metadata of another append. The json append removes it, but the text append cannot cut it.A string pkey that has a NUL in it (
"dx\u0000evil"). The json append takes it as"dx".
The text keeps its spaces and its escapes. A read gets back the same json, not the bytes that json_dumps() writes.
tranger2_record_tkey_time()¶
The time that tranger2_append_record() keeps as
the __tm__ of record in topic, and whether an md2 row keeps it whole.
The append and the callers that check a record first share this one
computation, so a check cannot disagree with what is written.
uint64_t tranger2_record_tkey_time(
json_t *topic, // NOT owned
json_t *record, // NOT owned
BOOL *fits
);Parameters
| Key | Type | Description |
|---|---|---|
topic | json_t * | The json of an open topic (tranger2_topic()): its tkey and system_flag are read. Not owned. |
record | json_t * | The record about to be appended. Not owned, not changed. |
fits | BOOL * | Set to FALSE when an md2 row cannot keep the time: negative, past TRANGER2_MAX_TIME, a date of an sf_tm_ms topic whose multiplication by 1000 passes the 44 bits, or a string that tranger2_time_of_text() refuses: one that is no date, or one written as a number that is not the text of a JSON integer (the return is then 0, and up to SDK 7.29.3 a text that is no date was NOW and fitted). TRUE otherwise. |
Returns
The time the append keeps: the value of the tkey -- an integer as it is, a
string read by tranger2_time_of_text() (the
text of a JSON integer as that integer, a date multiplied by 1000 on an
sf_tm_ms topic) --, or 0 when the topic has no tkey or the record no
value of it (a value of another type, and "", are 0 too). A negative integer comes
back as a large unsigned value, and *fits is FALSE.
Notes
The append keeps a time that does not fit CUT, and logs it as an ERROR, but it
does not refuse it. A caller that takes the record from a peer asks this
first and refuses the record: C_TRANGER’s add-record, and the treedb
before a create or a saved update of a node (since SDK 7.27.0).
Example
json_t *topic = tranger2_topic(tranger, "events"); // tkey "when", sf_tm_ms
BOOL fits;
json_t *rec = json_pack("{s:s}", "when", "@1700000000 +0000");
uint64_t tm = tranger2_record_tkey_time(topic, rec, &fits); // 1700000000000, fits TRUE
json_decref(rec);
rec = json_pack("{s:s}", "when", "@20000000000 +0000");
tranger2_record_tkey_time(topic, rec, &fits); // fits FALSE: 2e13 ms is past 44 bits
json_decref(rec);
rec = json_pack("{s:I}", "when", (json_int_t)-1);
tranger2_record_tkey_time(topic, rec, &fits); // fits FALSE: negative
json_decref(rec);
/* the check of a peer's record (yours), before the append */
tranger2_record_tkey_time(topic, record, &fits);
if(!fits) {
gobj_log_warning(gobj, 0,
"function", "%s", __FUNCTION__,
"msgset", "%s", MSGSET_PARAMETER,
"msg", "%s", "a time in its tkey that a record cannot keep, refused",
NULL
);
JSON_DECREF(record)
return -1;
}
tranger2_append_record(tranger, "events", 0, 0, &md, record); // owns recordtranger2_time_of_text()¶
The time a TEXT says, read by the one rule that timeranger2 applies to a
text time bound of a match_cond (from_t, to_t, from_tm, to_tm)
and to a text tkey. C_TRANGER reads the before of delete-key-files
and archive-key-files with it too. The rule for a number is the command
parser’s for an integer.
int tranger2_time_of_text(
const char *text,
BOOL in_ms,
BOOL end_of_second,
json_int_t *value,
const char **why
);Parameters
| Key | Type | Description |
|---|---|---|
text | const char * | The text. NULL is read as "". |
in_ms | BOOL | The axis is in milliseconds (t of an sf_t_ms topic, tm of an sf_tm_ms one): a date is multiplied by 1000. An integer is never converted: it is already in the unit of its axis. |
end_of_second | BOOL | An end bound: a date on an axis in milliseconds takes the last millisecond of its second (+999). |
value | json_int_t * | The time read. 0 when the text is refused. |
why | const char ** | When not NULL, the cause of a refusal (NULL when the text is read). |
Returns
| Return | Text |
|---|---|
0, *value 0 | "": no time (no bound, no tkey value). |
0, *value the integer | The text of a JSON integer, `-?(0 |
-1, “a time text longer than 256 bytes” | Any other text of more than 256 bytes (also as a json string, in tranger2_time_of_json_text()). No date needs more, and the reader takes time that grows with the length of a text that can be from a peer: a 140 KB text of "at", comments and "the" took it seconds. The text of a JSON integer is not of these: of any length it is an integer, and past INT64_MAX it is -2 (below). |
-1, “a number that is not the text of a JSON integer” | A text written as another number: " 5", "5 ", "007", "-007", "+5", ".5", "5.", "10.5", "1E5", and with a decimal comma, "1,5", "10,5", ",5", "5,". It is never read as a date. |
-1, “a comma between digits, not a date” | A comma between two digits among other words: "1,5 days ago", "1,5days ago", "2026,10,01", "10:00,5", "1,5:29", "yesterday 1,2026". approxidate() reads them as September 26 (1 and 5 days ago), January 10 of 2026, the 5th, the 1st at 05:29, September 30. Three commas of a date are kept: (1) the decimal comma of the seconds of a clock time, ISO 8601’s ("2026-10-01T10:00:00,500Z", "10:00:00,5"). (2) A comma that closes a date before its hour, a clock time or an hour with am/pm: "2026-10-01,10:00", "1 Oct 2026,10:00", "01 10 2026,10:00", "Oct 1,3 pm". What is before it must be a date, or the comma is refused: "1 2,10:00", "12 2026,10:00", "2026,10:00", read as February 1, October 12 and the year 2026. (3) A comma between a day (1 or 2 digits) and a year (4 or 2 digits) after the name of a month: "Oct 1,2026", "Oct 1,26". Not when the number after the comma is a count: "Oct 1,50 days ago", as "1,5 days ago" is refused. |
-1, “numbers separated by blanks or commas, not a date” | Words that are all digits: "5 5", "10 5", "2026 10 01", "2026, 10, 01", ", 5", "10 2026", "5_5", or two or more of them in a row among other words: "5 5 ago", "1 5 days ago", "2026 10 01 10:00", "12 2026 10:00", "yesterday 1 2026", "2026_10_01", "5 the 5". approxidate() reads "5 5" as May 5, "2026 10 01" as January 10 of 2026, ", 5" as the 5th of this month, "1 5 days ago" as September 26, "12 2026 10:00" as October 12 and "yesterday 1 2026" as September 30. The words are parted as approxidate_careful() parts them: by blanks, commas, _, and a . that is not between two digits ("Oct.. 1 2026", "3 pm.", but "01.10.2026" is one word), where letters and digits touch ("Oct1" is "Oct 1", "1 2thu" is "1 2 thu", "5the5" is "5 the 5"), but for an ordinal ("5th") and the T of ISO 8601 ("2026-10-01T10:00", "20261001T1000"), and by the - or / of a date with a month name ("Oct-1-5" is "Oct 1 5", "01/Oct/2026" is "01 Oct 2026"). A comment ("(CEST)") is passed over whole, as the reader passes over it, also between a count and its am/pm or unit ("Oct 1 5 (CEST) pm" is 17:00 CEST, and up to 7.29.3 the reader read 2005-10-01). ago, at, on, the, of and a weekday are passed over, as it passes over them: they part no digits ("5 the 5" is "5 5", May 5) and no month from its day ("march the 5 2026" is March 5). A run of digits is a date only as a person writes one: day month year with a year of 4 digits ("01 10 2026", "01 10 2026 10:00", but with numbers alone a year of 2 digits is refused, because approxidate() reads "1 2 34" as February 1, 2034), or after the name of a month a day, or a day and a year of 4 or 2 digits ("Oct 1", "1 Oct 2026", "Oct 1 2026", "Oct 1, 2026", "Oct 1 26"). A month name is 3 letters or more of its English name, in any case ("Oct", "Sept.", "october"), as approxidate_careful() reads one. Any other word before the run ("yesterday", a weekday) does not make it a date. After a day and a month name, in either order, only a year of 4 or 2 digits can follow ("1 Oct 2026", "Oct 1st 26", "Thu Oct 1 10:00:00 2026"). "1 Oct 5", "01-Oct 5", "Oct 1st 2" and "Oct 1 10:00 5" are refused (read as 2005 and 2002). An ordinal ("1st", "5th") is a day: after it, with no month name, only a month and a year of 4 digits ("1st 10 2026"). "5th 2026" (the 5th of a month the text does not say), "5th 2" (February 5) and "1st 10" are refused. A number alone is no hour: "Oct 29 10" is 2010-10-29, a year of 2 digits as in "Oct 1 26". After at (or at the), a person means a time, and the reader refuses a number alone there, of any value ("Oct 1 at 10", "at 3", "Oct 1 at 24", "Oct 1 at the 10"), a count back ("at 5 days", "at 2 fridays ago", "at -1d", "at last friday") and a number with dots and no year of 4 digits ("at 10.30", "at 10.30.00"), “a text that is no date”. Up to 7.29.3, "Oct 1 at 24" was 2024-10-01, "at 10.30" October 30 and "at 10.30.00" 2000-10-30. "Oct 1 at 10:00", "Oct 1 at 10 pm", "at 15h" and "at 10h30" (15:00 and 10:30, as Spanish and French write them, and up to 7.29.3 15 hours back, and 10 hours back from the 30th), "at 5 Oct 2026" and "at 5 of October" (a day before a month name) are dates. The reader takes ago only right after its count: "1 day 5 ago" is no date (it was October 4). A number that a unit, am/pm or a weekday follows is a count ("5 days", "3 pm", "3pm", "2 fridays"): it ends a run and is not in it, and the run before it must be a date, even of one number. "Oct 1 2026 3 pm", "01 10 2026 3 pm", "Oct 29 3 PM", "01 10 2026 5 days ago" are dates. "12 10 pm" (read as the 12th at 22:00, where 12:10 pm was meant), "5 5 pm", "5 at 3 pm", "2026 3 pm" and "1 5days ago" are refused. After a month, a day or a year, a number before a weekday is that date’s, not a count: "Oct 1 2026 Thu" is October 1. So is a year (4 digits, 1970-9999) before any weekday: "Oct 1 2026 thursdays" is no date, and "2026 Thu", a weekday with no day, is refused by the reader, as "monday 10:00" is (see approxidate_careful()). The weekday is not checked against a date: "Oct 5 2026 friday" is Monday, October 5. Beside a day counted from today, the reader refuses a weekday that is not that day’s ("now Mon" on a Wednesday: the day the text counted, in the zone said or the UTC offset of now), and a second weekday ("Mon Thu now"). A 12 that noon or midnight follows is their hour, right after at or with no date before it, as the reader reads it, and it ends a run too: "Oct 1 2026 at 12 noon" is a date, "2026 12 noon" is refused. "1700000000 +0000" (git’s raw date), "01.10.2026" and epoch seconds alone with a comment ("1761000000 (UTC)", "(EST) 1761000000", 9 digits or more and no leading zero, refused for a while) are not of these shapes, and are dates. Without a comment, a blank or a mark about them leaves a number: ", 1761000000" and "_1761000000" are refused, as " 1761000000" is. |
0, *value the date | Any other text that approxidate_careful() reads: "2000-01-02T00:00:00+0000", "2000-01-02 00:00:00 +0000", "yesterday". |
-1, “a text that is no date” | Any other text: "garbage", "0x10". |
-2, “an integer past what a time holds” / “a date past what a time holds” | An integer past INT64_MAX, of any length ("9223372036854775808", 300 nines, and a development version answered -1 “a time text longer than 256 bytes” from 257 digits on, and a tkey of them was refused as “a text that is no date”), or a date that does not fit a json_int_t in its unit. |
Notes
Nothing is logged: the text is from a peer or a user, and the caller
says it. Up to SDK 7.29.3 a bound with no T was cut by strtoll() (" 5"
was 5, "10.5" 10, "2026-10-01 00:00:00" 2026) and a tkey was read by
approxidate() ("5" the 5th of this month, a text that is no date NOW).
Unreleased development versions after it read every text that was not
-?[0-9]+ as a date: " 5" was the 5th, "10.5" May 10, in silence. A
decimal comma was too (approxidate() reads "1,5" as May 1, "10,5" as May 10),
and digits in groups ("5 5" May 5, "2026 10 01" January 10 of 2026).
Later a word more kept the misreading ("1,5 days ago", "5 5 ago",
"2026 10 01 10:00", ",5"), and "01 10 2026", a date read right, was
refused. Then an hour with am/pm after a date ("Oct 1 2026 3 pm"), the
decimal comma of the seconds ("2026-10-01T10:00:00,500Z"), a comma before
a clock time ("2026-10-01,10:00") and a two-digit year after a month name
("Oct 1 26") were refused, and a day and a year after any word were a
date ("yesterday 1 2026" September 30).
A dot between digits cannot be refused by its shape, as a comma is: it
is how a day and a month are written ("1.10 2026" and "01.10.2026" are
October 1). Only a text that is a number and nothing else ("10.5", ".5")
is refused. Any other dot is for approxidate_careful() to judge.
Since the release after SDK 7.29.3 (BREAKING) a comma before an hour is kept
only when it closes a date, and a lone number before an hour with am/pm or a
unit is refused: up to 7.29.3 "1 2,10:00" was February 1, "12 2026,10:00"
October 12, "12 10 pm" the 12th at 22:00 and "1 5days ago" September 26.
The words approxidate_careful() passes over ("Oct.. 1 2026", "march on 5 2026", "Oct 1 2026 3 pm.") are dates, as they were in 7.29.3. So is an
ordinal day with its year and an hour ("Oct 1st 2026 3 pm", "October 1st, 2026 at 3 pm"). Letters and digits that touch are parted as the reader parts
them: up to 7.29.3 "5the5" was May 5, "2026on10on01" January 10 and
"Oct-1 5" 2005-10-01, and a development version read "1 2thu" as
February 1 and "Oct1 5 friday" as 2005-10-01. A day and a month name take
only a year of 4 or 2 digits, and an ordinal with no month name only a month
and a year: up to 7.29.3 "1 Oct 5" was 2005-10-01, "5th 2" February 5
and "5th 2026" the 5th of the current month.
approxidate_careful() stays lenient past these shapes: "yesterday 10.30"
is NOW, and "1/10/2026" January 10 (the order of a date with slashes is
month/day). A time that must be exact is written as an integer, or as
ISO 8601 ("2026-10-01T10:30:00+0200").
A local time with no zone does not depend on the text read before it: one
that a change of hour repeats is its first one, and one that a change of hour
skips is read with the offset in force before the change (in Madrid
"2026-10-25 02:30" is 02:30 +0200 and "2026-03-29 02:30" 03:30 +0200, and up
to 7.29.3 the repeated hour was +0200 or +0100 according to the text read
before it). am/pm with no hour ("today pm"), a day word right after at
("at yesterday", "at now") and "never" with a zone ("never (+0200)")
are no date, and an offset after a date or a day is its zone ("yesterday +0000", as "yesterday UTC"): not one glued to the digits of a date with
only a day before it ("2026-10-01-10:00" is no date), but one right after
the digits of a clock time ("2026-10-01T10:00-05:00") or after a clock time,
noon or epoch seconds said before it, where no clock time can follow
("10:00 2026-10-01-0500", "noon 2026-10-01-0500", "@1790000000-0500",
"1790000000-0500"). A minus sign, a count and its unit is a duration there,
as anywhere ("10:00 2026-10-01-1d" is September 30 at 10:00), but where an
offset can stand four digits are that offset, as ISO 8601 writes it: "10:00 -0500 hours", "noon -0500 hours" and "2026-10-01 -0500 hours" are no date.
With no clock time an offset outside -12:00..+14:00 is no
date ("now-2026", "yesterday -2300"). A run of 12 digits with a sign
after it is no date ("202610011000-0500"): it is a date and a time written
with no T ("20261001T1000-0500"), and epoch seconds have 11 digits at
most there. The same 12 digits with nothing else are the text of an integer,
read as that integer. A count right after a
month name, with no day in the text, is no date ("Oct 1 pm", "Jan 1 pm",
"Oct 5 days ago", and up to 7.29.3 they took the day of today), and nor is a day
counted back that the zone skipped whole ("yesterday noon" in Apia on
2011-12-31). See approxidate_careful().
A text in C stops at its first NUL: a json string is read with
tranger2_time_of_json_text(), which judges
it whole.
Example
json_int_t v;
const char *why;
tranger2_time_of_text("1761000000000", TRUE, FALSE, &v, &why); // 0, v 1761000000000
tranger2_time_of_text("", FALSE, FALSE, &v, &why); // 0, v 0: no bound
tranger2_time_of_text(" 5", FALSE, FALSE, &v, &why); // -1, why "a number that is not the text of a JSON integer"
tranger2_time_of_text("2000-01-02T00:00:00+0000", TRUE, TRUE, &v, &why); // 0, v 946771200999
tranger2_time_of_text("garbage", FALSE, FALSE, &v, &why); // -1, why "a text that is no date"
tranger2_time_of_text("1,5", FALSE, FALSE, &v, &why); // -1, why "a number that is not the text of a JSON integer"
tranger2_time_of_text("Oct 1 2026 3 pm +0000", FALSE, FALSE, &v, &why); // 0, v 1790866800
tranger2_time_of_text("march the 5 2026 00:00:00 +0000", FALSE, FALSE, &v, &why); // 0, v 1772668800
tranger2_time_of_text("12 10 pm", FALSE, FALSE, &v, &why); // -1, why "numbers separated by blanks or commas, not a date"
tranger2_time_of_text("1 2,10:00", FALSE, FALSE, &v, &why); // -1, why "a comma between digits, not a date"
tranger2_time_of_text("Oct 1st 2026 3 pm +0000", FALSE, FALSE, &v, &why); // 0, v 1790866800
tranger2_time_of_text("5th 2026", FALSE, FALSE, &v, &why); // -1, why "numbers separated by blanks or commas, not a date"
tranger2_time_of_text("Oct1 5 friday", FALSE, FALSE, &v, &why); // -1, why "numbers separated by blanks or commas, not a date"
tranger2_time_of_text("Oct 5 2026 friday 00:00:00 +0000", FALSE, FALSE, &v, &why); // 0, v 1791158400: Monday, the weekday is not checked
tranger2_time_of_text("Oct 1 2026 5 (UTC) pm", FALSE, FALSE, &v, &why); // 0, v 1790874000: the comment parts 5 from its pm
tranger2_time_of_text("1761000000 (UTC)", FALSE, FALSE, &v, &why); // 0, v 1761000000
tranger2_time_of_text(", 1761000000", FALSE, FALSE, &v, &why); // -1, why "numbers separated by blanks or commas, not a date"
tranger2_time_of_text("Oct 1 at 10", FALSE, FALSE, &v, &why); // -1, why "a text that is no date": 10 am or pm?
tranger2_time_of_text("Oct 1 at 24", FALSE, FALSE, &v, &why); // -1, why "a text that is no date": no hour
tranger2_time_of_text("at 5 Oct 2026 00:00:00 +0000", FALSE, FALSE, &v, &why); // 0, v 1791158400
tranger2_time_of_text("Oct 1 2026 at 15h +0000", FALSE, FALSE, &v, &why); // 0, v 1790866800
tranger2_time_of_text("1 day 5 ago", FALSE, FALSE, &v, &why); // -1, why "a text that is no date"
tranger2_time_of_text("monday 10:00", FALSE, FALSE, &v, &why); // -1, why "a text that is no date": which monday?
tranger2_time_of_text(<a text of 300 bytes>, FALSE, FALSE, &v, &why); // -1, why "a time text longer than 256 bytes"
tranger2_time_of_text(<300 digits>, FALSE, FALSE, &v, &why); // -2, why "an integer past what a time holds"tranger2_time_of_json_text()¶
The time a json STRING says, judged whole: what
tranger2_time_of_text() reads in its text, and
a refusal for a string with a NUL inside. This is how timeranger2 reads a
text time bound of a match_cond and a text tkey, and how C_TRANGER reads
the before of delete-key-files and archive-key-files.
int tranger2_time_of_json_text(
json_t *jn_text, // NOT owned
BOOL in_ms,
BOOL end_of_second,
json_int_t *value,
const char **why
);Parameters
| Key | Type | Description |
|---|---|---|
jn_text | json_t * | The json string. Not owned. |
in_ms | BOOL | As in tranger2_time_of_text(). |
end_of_second | BOOL | As in tranger2_time_of_text(). |
value | json_int_t * | The time read. 0 when the text is refused. |
why | const char ** | When not NULL, the cause of a refusal. |
Returns
What tranger2_time_of_text() returns for its
text, and -1 with “a text with a NUL in it” for a string whose length is
not the length of its C text ("5\u0000junk": from json_stringn(), or a
json loaded with JSON_ALLOW_NUL, as records are), or -1 with “a time that
is no text” when jn_text is not a string.
Notes
json_string_value() gives the text up to its first NUL. Up to 7.29.3 the
tranger read a tkey and a bound with it: {"id":1,"tm":"5\u0000junk"} was
appended with __tm__ 5, and a before of
"2026-10-01T00:00:00Z\u0000garbage" purged the files before that date.
Example
json_int_t v;
const char *why;
json_t *jn = json_stringn("5\0junk", 6);
tranger2_time_of_json_text(jn, FALSE, FALSE, &v, &why); // -1, why "a text with a NUL in it"
json_decref(jn);
jn = json_string("2000-01-02T00:00:00+0000");
tranger2_time_of_json_text(jn, TRUE, FALSE, &v, &why); // 0, v 946771200000
json_decref(jn);tranger2_backup_topic()¶
Creates a backup of a topic in the TimeRanger database. If backup_path is empty, the topic path is used. If backup_name is empty, the backup file is named topic_name.bak. If overwrite_backup is true and the backup exists, it is overwritten unless tranger_backup_deleting_callback returns true.
json_t *tranger2_backup_topic(
json_t *tranger,
const char *topic_name,
const char *backup_path,
const char *backup_name,
BOOL overwrite_backup,
tranger_backup_deleting_callback_t tranger_backup_deleting_callback
);Parameters
| Key | Type | Description |
|---|---|---|
tranger | json_t * | The TimeRanger database instance. |
topic_name | const char * | The name of the topic to back up. |
backup_path | const char * | The directory where the backup will be stored. If empty, the topic path is used. |
backup_name | const char * | The name of the backup file. If empty, topic_name.bak is used. |
overwrite_backup | BOOL | If true, an existing backup is overwritten unless tranger_backup_deleting_callback prevents it. |
tranger_backup_deleting_callback | tranger_backup_deleting_callback_t | A callback function that determines whether an existing backup must be deleted before overwriting. |
Returns
The new, empty topic (not yours), created with the description, cols and var of the old one, or NULL (logged) when the backup fails.
Notes
If overwrite_backup is true and the backup exists, tranger_backup_deleting_callback is called. If it returns true, the existing backup is not removed.
Only a master can back up a topic (the backup moves its directory): on a replica it returns NULL with “Only master can back up”. The topic name follows the topic name rule.
The backup CLOSES the topic first, then moves its directory. A failure after that point (the backup exists and overwrite_backup is false, topic_desc.json does not load, or the rename() fails -- the backup name is taken by a file, another filesystem) returns NULL and opens the topic again as it was, with a warning: “Backup of topic failed: the topic is opened again as it was, not backed up”. Every pointer to the old topic is stale all the same, as after a successful backup: take the topic again by name.
So does a failure AFTER the move, when the new topic cannot be created (ENOSPC, a mkdir that fails): what the create left is removed, the backup is moved back where the topic was, and the topic is opened again, with the same warning. Only when the backup cannot be moved back is nothing opened: a CRITICAL, “Backup of topic failed, and the backup cannot be moved back: the data of the topic is in the backup”, with both paths. Up to 7.25.4 the data stayed in the backup and nothing was opened: a queue had no topic until a restart, and, its size read as 0, its backup was never tried again (tests/c/tr_queue/test_tr_queue_backup_failed).
That holds for EVERY tranger, also one opened with on_critical_error LOG_OPT_EXIT_ZERO (the MQTT broker’s queues, C_TRANGER’s default): the create of a backup runs with the exit bits of on_critical_error off (LOG_OPT_EXIT_ZERO, LOG_OPT_EXIT_NEGATIVE, LOG_OPT_ABORT), because its failure is recovered. The CRITICALs are still logged, and the tranger’s value is back when the create returns. If the create exits there, the data of the topic stays in <topic>.bak until moved back by hand.
json_t *topic = tranger2_backup_topic(tranger, "queue", 0, 0, TRUE, 0);
if(!topic) {
// Error already logged: not backed up; the topic is still the old one
topic = tranger2_topic(tranger, "queue");
}Up to 7.25.4 a failed rename() also leaked the topic_var it had loaded, and the topic stayed closed: a queue (trq_check_backup()) was left with no topic.
tranger2_close_all_lists()¶
Closes the iterators, rt_mem and rt_disk lists of a topic that belong to a
given creator. An empty creator closes all of them. A non-empty creator
with an empty rt_id closes all of that creator’s. With both set it narrows to a
single rt_id.
int tranger2_close_all_lists(
json_t *tranger,
const char *topic_name,
const char *creator, // if empty, remove all
const char *rt_id // if empty, remove all lists of creator
);Parameters
| Key | Type | Description |
|---|---|---|
tranger | json_t * | Pointer to the TimeRanger database instance. |
topic_name | const char * | Name of the topic whose lists must be closed. |
creator | const char * | Creator identifier. If empty, all lists are removed. |
rt_id | const char * | Realtime list identifier. If empty, all lists of the creator are removed. |
Returns
Returns 0 on success, or a negative value on failure. No-op (0) if the topic
is not opened.
Notes
This closes the iterators, rt_mem and rt_disk lists associated with a specific
creator, preventing resource leaks. If creator is empty, every list of the
topic is closed regardless of rt_id.
tranger2_close_iterator()¶
Closes an iterator in the TimeRanger 2 database, releasing associated resources.
int tranger2_close_iterator(
json_t *tranger,
json_t *iterator
);Parameters
| Key | Type | Description |
|---|---|---|
tranger | json_t * | A pointer to the TimeRanger 2 database instance. |
iterator | json_t * | A pointer to the iterator object to be closed. |
Returns
Returns 0 on success, or a negative value on failure.
Notes
Closing an iterator makes sure that any allocated memory or resources are properly released.
tranger2_close_list()¶
The function tranger2_close_list() closes a previously opened list, which can be a real-time memory list (rt_mem), a real-time disk list (rt_disk), or a non-real-time list.
int tranger2_close_list(
json_t *tranger,
json_t *list
);Parameters
| Key | Type | Description |
|---|---|---|
tranger | json_t * | A pointer to the TimeRanger database instance. |
list | json_t * | A pointer to the list object to be closed. |
Returns
Returns 0 on success, or a negative value on failure.
Notes
This function makes sure that resources associated with the list are properly released. If the list is a real-time list, it stops receiving updates.
tranger2_close_rt_disk()¶
The tranger2_close_rt_disk() function closes a previously opened real-time disk stream in the TimeRanger database, releasing associated resources.
int tranger2_close_rt_disk(
json_t *tranger,
json_t *disk
);Parameters
| Key | Type | Description |
|---|---|---|
tranger | json_t * | A pointer to the TimeRanger database instance. |
disk | json_t * | A pointer to the real-time disk stream to be closed. |
Returns
Returns 0 on success, or a negative value on failure.
Notes
This function must be called when a real-time disk stream is no longer needed to free resources.
The directory of the feed, <topic>/disks/<id>/, is renamed away first
(disks/.closing.<pid>-<start>.<seq>) and then removed. The master keeps hard-linking
the new records into that directory until it hears the feed is gone. Removed
where it was, a key directory the master made in between made the removal
fail (“rmdir() FAILED ... Directory not empty”), the directory stayed, and the
master went on feeding it for ever (up to 7.25.21, seen at 3000 records/s).
Renamed, the master cannot reach it, and its watch of disks/
(FS_FLAG_MOVED_AS_DELETED) hears the rename as the close of the feed. A
.closing.* left by a reader that died while removing it is removed by the
master the next time it reads disks/ (only when that process is gone).
<start> is when the reader’s process started (field 22 of
/proc/<pid>/stat): a pid alive with another start time is a pid reused by
another process, and the leftover is removed too. Up to 7.25.21 the name had
the pid alone, and such a leftover stayed while the unrelated process lived.
A name of the old form (.closing.<pid>.<seq>) is judged by its pid alone.
tranger2_close_rt_mem()¶
The tranger2_close_rt_mem() function closes a real-time memory stream associated with a given TimeRanger instance.
int tranger2_close_rt_mem(
json_t *tranger,
json_t *mem
);Parameters
| Key | Type | Description |
|---|---|---|
tranger | json_t * | A pointer to the TimeRanger instance managing the real-time memory stream. |
mem | json_t * | A pointer to the real-time memory stream to be closed. |
Returns
Returns 0 on success, or a negative value on failure.
Notes
Closing a real-time memory stream using tranger2_close_rt_mem() makes sure that resources are properly released.
tranger2_close_topic()¶
The tranger2_close_topic() function closes an open topic in the TimeRanger database, releasing associated resources.
int tranger2_close_topic(
json_t *tranger,
const char *topic_name
);Parameters
| Key | Type | Description |
|---|---|---|
tranger | json_t * | A pointer to the TimeRanger database instance. |
topic_name | const char * | The name of the topic to be closed. |
Returns
Returns 0 on success, or a negative value on failure.
Notes
Closing a topic does not delete its data. It only releases resources associated with the open topic.
tranger2_create_topic()¶
The tranger2_create_topic() function creates a new topic in the TimeRanger database if it does not already exist. If the topic exists, it returns the existing topic metadata. The function makes sure that the topic is properly initialized with the specified primary key, time key, system flags, and additional metadata.
json_t *tranger2_create_topic(
json_t *tranger,
const char *topic_name,
const char *pkey,
const char *tkey,
json_t *jn_topic_ext,
system_flag2_t system_flag,
json_t *jn_cols,
json_t *jn_var
);Parameters
| Key | Type | Description |
|---|---|---|
tranger | json_t * | Pointer to the TimeRanger database instance. If the topic exists, only tranger and topic_name are required. |
topic_name | const char * | Name of the topic to be created. |
pkey | const char * | Primary key field of the topic. If not specified, the key type defaults to sf_int_key. |
tkey | const char * | Time key field of the topic. |
jn_topic_ext | json_t * | JSON object containing additional topic parameters. See topic_json_desc for details. This parameter is owned by the function. |
system_flag | system_flag2_t | System flags for the topic, defining its behavior and properties. |
jn_cols | json_t * | JSON object defining the topic’s columns. This parameter is owned by the function. |
jn_var | json_t * | JSON object containing variable metadata for the topic. This parameter is owned by the function. |
Returns
Returns a JSON object representing the topic metadata. The returned JSON object is not owned by the caller and must not be modified or freed.
NULL (logged) for a refused name, a topic that does not exist on a replica,
an empty pkey, no key type, a NEW topic that cannot be made whole (below), a
topic_cols.json / topic_var.json that cannot be written at a version
change, or a topic that cannot then be opened (see
tranger2_open_topic()).
A new topic is made whole, or not at all. The master creates the topic
directory, topic_desc.json, topic_cols.json, topic_var.json, keys/ and
disks/. When any of them cannot be made (no space, a mkdir that fails),
the directory it made is removed, nothing is opened or kept in memory, the
call answers NULL and logs “Cannot create topic: it is not whole, what was
made is removed” after the cause. The next create starts again from nothing.
Up to 7.25.4 each failure was logged and the create went on: a topic
whose keys/ (or disks/) was not made was opened and returned as
created, and a queue backup (see
tranger2_backup_topic()) took that half topic as
the queue’s new one.
It is whole or not at all under LOG_OPT_EXIT_ZERO too. The failures of
the create (the mkdirs, the write of topic_desc.json) are CRITICALs logged
without the exit bits of on_critical_error. “Cannot create topic: it is not
whole, what was made is removed” is logged after the removal, as a CRITICAL
at the tranger’s own on_critical_error. So a tranger that exits on a
critical -- LOG_OPT_EXIT_ZERO, the default of C_TRANGER and what
C_TREEDB and the MQTT broker’s queues pass -- exits there, with nothing of
the topic left on disk. Up to 7.25.4 it exited in the log of the first
failure: the topic_desc.json, topic_cols.json and topic_var.json stayed
without keys/ or disks/, and the next start opened that half topic (or
was not able to open it at all, when the failure was the write of
topic_desc.json).
json_t *topic = tranger2_create_topic(tranger, "devices", "id", "tm", NULL,
sf_string_key, json_pack("{s:s, s:I}", "id", "", "tm", (json_int_t)0), 0);
if(!topic) {
return -1; // Error already logged; nothing of "devices" on disk or in memory
}Notes
This function is idempotent. This means that if the topic already exists, it will return the existing topic metadata instead of creating a new one. If the primary key (pkey) is not specified, the function defaults to sf_string_key if pkey is defined, otherwise it defaults to sf_int_key.
Only the master writes, and it asks first. On a replica, or on a master
that lost its lock while it was stopped (see tranger2_stop()),
an existing topic is opened read-only and nothing is written, and a new one is
refused with “Cannot open TimeRanger topic. Not found and no master”.
Nothing about tm order is written. From 7.25.5 to 7.25.22 a new topic
carried "marks_tm_unordered": true and the master marked a file whose tm
went back (<file>.tm_unordered). Both are gone, and a store that has them
keeps them, ignored. A tm condition is a filter on every row: see
tranger2_open_iterator().
{
"topic_name": "readings",
"pkey": "id",
"tkey": "tm",
"system_flag": 1
}A new topic_version replaces topic_cols.json, then topic_var.json,
each through a temporary file and a rename(), both fsync’ed with their
directory, so the new version never survives a power cut its cols did not. The
last_rowid_id counter of treedb is carried over. The cols go first because
the version is what says they moved. A file that cannot be written refuses the
topic: the call answers NULL, logs “Cannot re-create topic_cols.json for a
new topic_version: the topic keeps its version, and is not opened” (or the
error of topic_var.json), and the next create tries again -- the version on
disk has not moved. “Re-Creating topic_var.json” and “Re-Creating
topic_cols.json” are logged only for a file written. Up to 7.25.4
topic_cols.json was REMOVED first and a failed write went unnoticed: the
topic opened with no cols file under the new version.
Topic names. A topic name is ONE directory under the database, and a segment of the backtick kw paths (topics`<name>`cols). Every call that takes a topic name (create, open, delete, backup, tranger2_topic_path(), tranger2_write_topic_var() / _cols()) refuses, with the error “Invalid topic name (path metacharacters not allowed)”:
An empty name,
.and...Any name that holds
/or`.
A leading . is accepted, unlike a key: MQTT queues are named <client_id>-IN / -OUT, and the broker accepts a client_id such as .foo.
The id of a disk feed follows the directory half of the rule. The id
of tranger2_open_rt_disk() (the rt_id of a
follower’s open-rt / open-list) becomes the directory
<topic>/disks/<id>/, and the follower removes that directory before creating
it. An empty id is refused with “Invalid rt id (empty)”. ., .., or an id
holding / is refused with “Invalid rt id (path metacharacters not allowed)”, and an id
longer than NAME_MAX (255 bytes on Linux) with “Invalid rt id (longer than
NAME_MAX)” (its directory cannot exist). A leading . is refused too, with
“Invalid rt id (a leading dot is reserved)”: disks/.<name> is where a feed
goes while it is removed (see tranger2_close_rt_disk()).
A backtick is accepted, since an rt id is not a segment of any kw path (treedb
names its own feeds <treedb>`<topic>`<id>). The id can be from a peer,
so a refused id is logged as a warning, without a stack.
tranger2_open_rt_disk(tranger, "raw_events", "", NULL, cb, "db_history^7301", "", NULL); // ok
tranger2_open_rt_disk(tranger, "raw_events", "", NULL, cb, "a`b`c", "", NULL); // ok
tranger2_open_rt_disk(tranger, "raw_events", "", NULL, cb, ".hidden", "", NULL); // NULL: reserved
tranger2_open_rt_disk(tranger, "raw_events", "", NULL, cb, "../x", "", NULL); // NULL: escapesOne id, one feed. The directory is keyed by the id alone, so an id already
in use by a live feed of the topic is refused whatever the creator, with a
warning: “rt disk id already in use by another creator, refused”, or “rt
disk id already in use by the same creator, refused”. A second feed used to
take the first one’s directory over, and its close removed it. The guard is
per process: it reads the feeds of this tranger, so two processes that
follow one store with the same id still take each other’s directory.
tranger2_open_rt_disk(tranger, "users", "", 0, cb, "gui-42", "gui", 0); // OK
tranger2_open_rt_disk(tranger, "users", "", 0, cb, "../../etc", "", 0); // refused
tranger2_open_rt_disk(tranger, "users", "", 0, cb, "", "", 0); // refused: empty
tranger2_open_rt_disk(tranger, "users", "", 0, cb, "gui-42", "other", 0); // refused: in use
tranger2_open_rt_disk(tranger, "users", "", 0, cb, "gui-42", "gui", 0); // refused: in use
char long_id[NAME_MAX + 2];
memset(long_id, 'x', sizeof(long_id) - 1);
long_id[sizeof(long_id) - 1] = 0; // 256 bytes
tranger2_open_rt_disk(tranger, "users", "", 0, cb, long_id, "gui", 0); // refused: > NAME_MAXtranger2_create_topic(tranger, "users", "id", "tm", 0, sf_string_key, 0, 0); // OK
tranger2_create_topic(tranger, ".foo-IN", "", "", 0, sf_rowid_key, 0, 0); // OK
tranger2_create_topic(tranger, "../other_db/users", "id", "tm", 0, 0, 0, 0); // refused
tranger2_create_topic(tranger, "users/keys", "id", "tm", 0, 0, 0, 0); // refusedtranger2_delete_key()¶
The tranger2_delete_key() function deletes a whole record (= a
primary key, with every instance stored under it) from the specified
topic in the TimeRanger database. It removes the keys/<key>/
directory and drops the key from the topic cache. Irrecoverable. Only
the master can delete.
The deletion is propagated to subscribers:
The master removes the live
keys/<key>/directory FIRST, and signals the delete only once it is done (up to 7.25.4 it signalled first, so a key whose directory the master was not able to remove was heard deleted while it was still on disk). It signals it in the directory of every rt_disk feed:topic/disks/<rt_id>/<key>/, where it exists (the feed received records of the key), is removed, and then a directory.d<seq>.<key>is created and removed at once -- that is the signal.seqis the topic’s delete sequence (<topic>/delete_seq.json, recorded durable before the signal, growing across restarts of the master), and it is what lets a follower tell a delete it applied from a new one (see fs_watcher, the rt_disk feed). A key longer than NAME_MAX - 23 bytes is signalled as.h<seq>.<sha256 of the key>. A feed that received no record of the key since it opened hears the delete too. The sequence is taken only whendisks/has a feed to tell. Its record costs two disk flushes per delete then. Since 7.26.0. Up to 7.25.22, the signal was the key’s own directory (removed, or created and removed), so master and followers of a topic upgrade together. Whendisks/cannot be listed (itsopendir()fails with anything butENOENT, or areaddir()fails half way), the feeds not reached do not hear it, and that is logged: “Cannot tell the rt_disk feeds that a key was deleted, opendir() of disks/ FAILED” or “Cannot tell every rt_disk feed that a key was deleted, readdir() of disks/ FAILED”. The delete itself is done and answers0. Up to 7.25.4 both were silent.Each follower watches its own
disks/<rt_id>/. It clears the key from its topic cache and fires the callback of that feed only, once.A key deleted and written again (once or more) before a follower reads the delete: the feed hears
deleted(once or more), then the records of the key’s LAST life from rowid 1, and keeps the key, also behind a master in ANOTHER process. Each key directory is watched with a descriptor of its own (FS_FLAG_DIR_FDS, fs_watcher_dir_fd), and a link is taken through the descriptor of the directory its event came from (openat/unlinkat): a link of a directory gone reaches nothing. The link is the md2 of its life: the records are read through descriptors checked to be that life’s (the md2 against the link, the content once per life by path), so they are that life’s even ifkeys/<key>/is another one by then. Up to 7.25.21 a follower that lagged behind such a master read and unlinked by path, and was able to hand a later life before an earlier delete ([R2 DEL], the live key out of the cache, the next append handing R1..R4, and under a random two-process load 7-11 of 12 feed and key pairs got a life out of order, none now). The records of a life deleted before the follower read it are not handed -- they are gone from disk with it. The follower reads a key directory only if it is still the directory it looked at before it asked where its queue ended, and only once its stream is past that end: for any key, also one it never saw. A file linked in a directory that waits to be read is read with it, in order. When it hears a delete, the follower closes the descriptors it holds on the files of the key: looked up by the key, as the master does at its delete (up to 7.25.21 every open key was walked, once per feed that heard it: 4.3 ms per delete and feed with 50000 keys open, 17 us now). The first feed to hear a delete asks where the stream of each other feed ends (fs_queued_events_end()), to note what each one owes: a cost linear in the feeds, 17 us per delete and feed with one feed, 30 with sixteen. Where the filesystem keeps no birth time (some NFS, ext4 with 128-byte inodes) a key directory is told from another by its inode alone, and an inode freed by a delete can be given to the key written again: said once, with a warning (“No birth time of a directory ...”). So is a container whose seccomp profile refusesstatx()(EPERM): the inode is then read fromlstat(). Any other failure to tell which directory a key directory is (EIO,EACCES) is an ERROR, once per errno in a row, and the directory is not read then. Up to 7.25.21 every failure ofstatx()took the directory for gone, in silence. Up to 7.25.20 the directory was read early ([R1 DEL],[DEL R1 DEL], the live key out of the cache), and a key read before, deleted and written again in the same day file, was read through the old file’s descriptor (short reads, records lost). Details in fs_watcher.A key directory that cannot be read for a PASSING cause -- it cannot be told, opened or listed, or one of its links cannot be opened or removed, with
EMFILE,ENFILE,ENOMEM,EIO,EINTRorEAGAIN-- keeps its links, and nothing brings them back by itself: the master links a file into the feed only when the link is not there, so the key’s next records in that file make no event. Its read is OWED: each failure is logged where it happens, the debt is said once per feed (“rt_disk feed: a key directory could not be read for a passing cause, its read is owed to the end of the next batch of events of the feed”, a WARNING), and the read is tried again at the end of the feed’s next batch of events, if the directory is still the one seen. The first one that fails again ends that round, and the rest wait for the next batch. When nothing is owed any more it is said (“rt_disk feed: no read of a key directory owed any more”, an INFO). A link heard meanwhile in a directory that owes its read is left to that read, which takes every link in order. A failure for another cause (EACCES) is logged and not owed. No timer: a feed that hears no event at all keeps the read owed until it hears one -- a record of any key that makes a new link, a new file of a key, a delete, or the echo of a link it consumes itself. Until then, the keys it owes are behind the master. A record that cannot be read once its link is consumed is read with the key’s next link, as before. Up to 7.27.6 such a key stayed behind the master for good -- until a new file of the key (a new day) or a new feed -- and a key directory watched by its path (the open of its descriptor failed when it was made) was listed by its path. It is now opened and read through that descriptor, like the others (tests/c/timeranger2/test_rt_disk_emfile):follower with 2 descriptors free, master writes a new day of keys 0..59 -> ERROR Cannot open a key directory to read its links (one per key) -> WARNING rt_disk feed: a key directory could not be read ... owed ... (once) descriptors free again, master appends to keys 0..59 -> no event: their links are there; the follower serves 2 records, the master 13 master appends to key 60 (a new key: an event, a batch of the feed) -> keys 0..59 read at the end of the batch: 13 records each -> INFO rt_disk feed: no read of a key directory owed any moreIn-process
rt_mem,open_iterator, andrt_disksubscribers without a watcher (no loop) whosekeyfilter matches receive theirtranger2_key_deleted_callback_tdirectly, if registered.
For per-instance delete (one row of a key’s .md2 index —
irrecoverable, no resurrection), see
tranger2_delete_instance().
int tranger2_delete_key(
json_t *tranger,
const char *topic_name,
const char *key
);Parameters
| Key | Type | Description |
|---|---|---|
tranger | json_t * | A pointer to the TimeRanger database instance. |
topic_name | const char * | The name of the topic from which the record will be deleted. |
key | const char * | The key identifying the record to be deleted. |
Returns
Returns 0 on success, or a negative value on failure.
Notes
A key with no directory on disk (its stat() fails with ENOENT) is logged
(“key directory not found”, an error) and the call still answers 0: the key
is removed from memory all the same. Any OTHER failure of that stat()
(EACCES on keys/, EIO) says nothing of the key: the call answers -1
with “Cannot delete key, stat() of its directory FAILED”, and nothing is
deleted, dropped from memory or announced. Up to 7.25.4 it was taken as
“not found”: the key left the cache, the delete was announced to every feed
and follower, and the call answered 0 over files still on disk, which came
back at the next open.
key A, 3 rows; stat(keys/A) fails with EIO -> -1, A keeps its 3 rows, no callback
the same delete once keys/A can be stat'ed -> 0, one callbackThe directory of the key is first moved out of keys/, whole, to
<topic>/.deleted_key.<pid>.<start>.<n> (a rename(), atomic: the key is
there whole, or its name is gone), and then removed. So a delete never leaves
some of a key’s files without the others. Up to 7.27.5 a key without a base
was removed in place, file by file: a removal that failed half way left some
of its files, and the key, read again from them, numbered its rows anew --
its next records took g_rowids that followers and clients had seen.
A directory that cannot be moved out answers -1 (“Cannot delete key,
rename() of its directory out of keys/ FAILED: the key stays as it was”) and
announces nothing. The key is on disk as it was. Its cache is read again from
it, and its iterators take their segments again from it: an unfiltered one at
its next page, a FILTERED one at once, its index built again as an open builds
it. A moved directory that cannot be removed is the key deleted all the same:
0, announced, logged (“Key deleted, but its directory moved aside cannot be
removed: it is swept at the next open of the topic”), and removed at the
master’s next open of the topic:
key A, 3 rows; a filtered pager of A (from_t) -> total_rows 3
tranger2_delete_key(A), keys/A of mode 0550 -> -1, A keeps its 3 rows, no callback
the same pager, next page -> total_rows 3
keys/A holds a directory that cannot be emptied -> 0, one callback, keys/A gone,
<topic>/.deleted_key.* left
the next open of the topic by the master -> the directory left removed (a warning)The iterators of the key, in this process, lose what they took from it: see
tranger2_iterator_get_page().
The legacy name tranger2_delete_record() is kept as a source-level
alias in timeranger2.h (#define tranger2_delete_record tranger2_delete_key)
so existing callers keep compiling unchanged. New code must use
tranger2_delete_key().
tranger2_delete_key_files()¶
The tranger2_delete_key_files() function deletes the OLD FILES of a key and
keeps the g_rowid of every record left. It is the third delete granularity,
between a whole key (tranger2_delete_key()) and one
instance (tranger2_delete_instance()), and it is
what a retention period is made of. Only the master can delete. The design is
in kernel/c/timeranger2/DESIGN-retention.md.
It deletes every file of the key whose id sorts before the id of the file
that before_t belongs to. The cut is by file and by the time of the append
(__t__), never by tm, and no row is read: at least the period asked for is
kept, and at most one file more. With filename_mask %Y-%m-%d, “30 days”
keeps 30 to 31 days.
The rows deleted go to the base of the key, keys/<key>/.purged:
{"upto": "2026-09-20", "rows": 1823411, "t": 1792573238}The base is written with an fsync() of the file and of its directory
before any file is deleted, so a crash leaves the key as it was or purged,
never renumbered. The live rowids of the key are then purged+1 .. purged+rows, and:
tranger2_topic_key_size()counts the live rows.tranger2_topic_key_range()adds"purged".A
from_rowid/to_rowidof a match_cond counts from the base (a positive bound inside the purged rows is taken to the first live one), and a negative one from the end, as before.Position N of a page of an unfiltered iterator is
g_rowid = purged + N, with the base the key had when the iterator opened: a purge since the open moves no position (the ones of the rows it took are returned empty), and a filtered iterator keeps its index, skipping the rows purged.The next append gets
purged + rows + 1.
json_int_t tranger2_delete_key_files(
json_t *tranger,
const char *topic_name,
const char *key,
uint64_t before_t
);Parameters
| Key | Type | Description |
|---|---|---|
tranger | json_t * | A pointer to the TimeRanger database instance. It must be the master. |
topic_name | const char * | The topic of the key. The file ids of its filename_mask must sort as time (tranger2_mask_sorts_as_time(): the year first, numeric fields of a fixed width, and %s, a %G that ends the mask, %Y-%d and the %E/%O modifiers are refused). |
key | const char * | The key whose old files are deleted. |
before_t | uint64_t | The cut, in the topic’s unit of t (milliseconds with sf_t_ms). The files of the records appended before the period of before_t are deleted. |
Returns
The rows deleted, 0 when there was nothing to delete, or -1 (logged). A
-1 is almost always a refusal, with nothing deleted. One -1 is a purge
APPLIED: the base was written but the fsync() of its directory failed, so
the cache and the followers took the purge, and the files stay on disk until
the next open of the master removes them (see the notes below).
Example
/*
* Retention of 30 days on a topic of one file a day (in seconds), a slice
* of the keys per tick of the owner's timer
*/
uint64_t cut = time_in_seconds() - 30*86400;
json_t *keys = tranger2_list_keys(tranger, "raw_events");
size_t n = json_array_size(keys);
for(size_t i = priv->purge_cursor; i < n && i < priv->purge_cursor + 100; i++) {
const char *key = json_string_value(json_array_get(keys, i));
if(tranger2_delete_key_files(tranger, "raw_events", key, cut) < 0) {
// Error already logged: refused, the key as it was -- or the purge
// applied with its base not durable, the files removed at the next open
}
}
priv->purge_cursor += 100;
if(priv->purge_cursor >= n) {
priv->purge_cursor = 0; // the next tick starts the next round
}
JSON_DECREF(keys)key K: files 2000-01-01 (rows 1-3), 2000-01-02 (4-6), 2000-01-03 (7-9)
tranger2_delete_key_files(K, before_t = 2000-01-02T00:00:00) -> 3
files left: 2000-01-02 (4-6), 2000-01-03 (7-9), base {upto 2000-01-01, rows 3}
tranger2_topic_key_size(K) -> 6, the next append -> g_rowid 10
from_rowid 1 or 3 -> from 4; from_rowid -1 -> 10; to_rowid 3 -> nothingNotes
Refused with
-1, logged, nothing deleted: a cut in the future (its file sorts after the file of now: if accepted, the cut takes the file being written, and every append of the period after). Afilename_maskwhose ids do not sort as time (tranger2_mask_sorts_as_time(): the"queue"of a tr_queue topic,%d-%m-%Y,%Y-%b, but"tracks-%Y-%m-%d"is fine). A topic of a treedb or a msg2db open on the tranger (its keys are their nodes). A key flaggedunlisted, or with a file flagged unreadable in the period cut (its rows are in no cell, and otherwise the base misses them). A key not in the cache. A replica.An append into a purged period is refused afterwards (“Cannot append record, it falls in a purged period of its key”): otherwise, its file is taken for a purged one at the next open. Appends with
__t__ = 0(now) never meet it.A key purged whole keeps its directory and its base, with 0 rows, and a new append continues the numbering.
tranger2_delete_key()removes the base with the key, and a key written again starts at 1.At the open, the files at or before
uptoare not counted: the master removes the ones a crash left behind (and a.purged.new, a purge that did not happen), a replica skips them. A.purgedthat cannot be read flags the keyunlisted(“The base of a purged key cannot be read”): it is not a base of 0, because a base of 0 numbers the key from 1 again. A load reads the base again when the directory or the base changed, or when the read failed for a passing cause (EMFILE, ENFILE, ENOMEM, EIO, EINTR, EAGAIN: up to 7.27.6 such a key stayed flagged, every load of it empty). A replica that lists a key while the master purges it, and finds a md2 gone at its open, reads the base again. When it covers the file, the replica numbers the key as the master (INFO “md2 file of the key purged by the master while its cache was built: its rows are in the base, read again”).Followers. The master makes and removes
disks/<rt_id>/.p.<key>(.q.<sha256>for a key longer than NAME_MAX - 23 bytes) in every feed directory. A follower that hears it reads the base, drops the cells at or beforeupto, closes their descriptors and takes the segments of its iterators again. One that did not hear it (an overflow, no feed, down) finds it before its next load or page of the key, with onestat()of the key’s first md2. A follower that cannot read the base (EMFILE, EIO) is not flagged: it keeps its numbering (its cells sum the master’s base, unless it missed a late row of the files purged) and its new records, drops the cells whose files are gone, and reads the base again at every read until it can. A purge that moves the numbering of a follower (rows it had missed, a file it flagged unreadable) is said with a WARNING, “The numbering of the key moved to the master’s: the records handed before had g_rowids below the master’s”, with itscauseanddelta. Master and followers of a topic upgrade together: an older follower ignores the signal.Cost. One small file and two
fsync()per key purged, and two or three unlinks per file: 0.9 to 1.8 ms per key on two Rocky/Debian test VMs. The hard linksdisks/<rt_id>/<key>/<id>.md2that a dead feed left are removed too, BEFORE the files ofkeys/: while one of them cannot be removed (“Cannot remove a feed’s link of a purged period, unlink() FAILED: the files of the period stay, removed at the next open”), the files of the period stay as well, and the next open of the master removes both. Up to 7.27.6 the files went first, and a link left by a crash or a failedunlink()was never removed.When the directory of the base cannot be
fsync()ed after its rename, the purge is applied (the cache and the followers take it) but the files stay until the next open, and the call answers-1.tranger2_delete_key()moves the directory of the key out ofkeys/whole (arename(), atomic) before it removes it, base included (since 7.27.6): a delete that cannot move it answers-1with the key as it was, numbered by its base, and a moved directory that cannot be removed is the key deleted all the same (0), swept at the master’s next open.By hand: the
delete-key-filescommand ofC_TRANGER.To keep the files instead of losing them:
tranger2_archive_key_files().
tranger2_archive_key_files()¶
The tranger2_archive_key_files() function is the purge of
tranger2_delete_key_files() -- the same cut,
the same guards, the same base, the same answer -- and the files it takes out
of the topic are kept: each purge of a key hard-links them into a new
batch of archive_directory. What is done with a batch (sent to a backup
machine, compressed, loaded elsewhere) is the job of another process, which
removes the batch when it is done. The design and its crash table are in
kernel/c/timeranger2/DESIGN-retention.md §12. Added after 7.28.11: it ships
in the next release.
<archive_directory>/<database>/<topic>/<key>/<upto>@<t>/
<file_id>.json <file_id>.md2 [<file_id>.unordered] [<file_id>.tm_unordered]
topic_desc.json topic_cols.json the schema, as it is at the purge
manifest.jsonupto is the last file id taken and t the time of the purge (the t of
the base). The manifest says what the batch holds and how it is numbered:
{
"database": "tr_raw_events",
"topic": "raw_events",
"key": "dev1",
"files": ["2026-09-19", "2026-09-20"],
"rows": 5712,
"first_g_rowid": 1817700,
"last_g_rowid": 1823411,
"upto": "2026-09-20",
"t": 1792573238
}Row N of the k-th file of files has the g_rowid first_g_rowid + the
rows of the files before it + N - 1.
When the base is written, an empty ready mark is created in
<archive_directory>/.ready/. Its name is the path of the batch under the
archive, with % written %25 and / written %2F:
<archive_directory>/.ready/tr_raw_events%2Fraw_events%2Fdev1%2F2026-09-20@1792573238A reader watches that one directory (a rename() gives no fs event, and the
tree has a directory per key). The mark is a hint, not a record: a batch
without one is found by a scan of the tree. The SDK’s reader is the
archive_sender yuno.
json_int_t tranger2_archive_key_files(
json_t *tranger,
const char *topic_name,
const char *key,
uint64_t before_t,
const char *archive_directory
);Parameters
| Key | Type | Description |
|---|---|---|
tranger | json_t * | A pointer to the TimeRanger database instance. It must be the master. |
topic_name | const char * | The topic of the key, as in tranger2_delete_key_files(). |
key | const char * | The key whose old files are archived. |
before_t | uint64_t | The cut, in the topic’s unit of t, as in tranger2_delete_key_files(). |
archive_directory | const char * | Where the batches go: an absolute path of a directory that EXISTS, on the filesystem of the topic (a hard link cannot cross filesystems), neither inside the directory of the tranger nor holding it. The levels <database>/<topic>/<key> under it are created. |
Returns
The rows archived (and taken out of the topic), 0 when there was nothing to
take, or -1 (logged). A refusal deletes nothing and leaves no batch. The
-1 of a purge APPLIED with its base not durable is as in
tranger2_delete_key_files(), and its batch
stays.
Example
/*
* Retention of 30 days that keeps what it takes: the batches go to
* /yuneta/store/archive, where a reader yuno sends them to a backup
* machine and removes them
*/
uint64_t cut = time_in_seconds() - 30*86400;
json_int_t rows = tranger2_archive_key_files(
tranger, "raw_events", "dev1", cut, "/yuneta/store/archive"
);
if(rows < 0) {
// Error already logged: refused, nothing archived nor deleted
}key dev1: files 2000-01-01 (rows 1-3), 2000-01-02 (4-6), 2000-01-03 (7-9)
tranger2_archive_key_files(dev1, before_t = 2000-01-02T00:00:00, "/yuneta/store/archive") -> 3
topic: files 2000-01-02 (4-6), 2000-01-03 (7-9), base {upto 2000-01-01, rows 3}
archive: .../dev1/2000-01-01@<t>/ 2000-01-01.json 2000-01-01.md2 topic_desc.json
topic_cols.json manifest.json {rows 3, first_g_rowid 1, last_g_rowid 3}
tranger2_archive_key_files(dev1, before_t = 2000-01-03T00:00:00, ...) -> 3
archive: a second batch .../dev1/2000-01-02@<t>/, first_g_rowid 4Notes
A batch whose name starts with
.is not finished. It is built as.new-<upto>@<t>and renamed when it is whole and durable: a reader that watches the directory takes only the names that do not start with..The batch is durable BEFORE the base. A crash while it is built leaves a
.new-*, removed by the next archive of the key (INFO “Batch of an archive that was not finished removed: its purge did not happen”). A crash between the batch and the base leaves a whole batch of files still live: the next purge archives them again in another batch. Until then the first batch holds hard links to live files, so a late append,tranger2_write_user_flag()ortranger2_delete_instance()changes them there too, after itsmanifest.json: two batches of one file id can differ, and it is possible that the older one does not match its manifest. A reader takes a file id from the batch with the latest@<t>. A base that cannot be written removes the batch and answers-1.Refused, besides the refusals of
tranger2_delete_key_files(), with-1, logged, nothing deleted: noarchive_directory(“what archive_directory?”), a relative one, one that does not exist (“cannot be resolved: it must exist”), a file, one on another filesystem (“a hard link cannot cross it”), one inside the directory of the tranger or holding it, and a batch that already exists with the name of this one.The archive frees no disk until its reader removes a batch. The SDK puts no limit on it: while the reader is down it grows by what retention takes.
Cost. Three
fsync()more per key than a delete (the manifest, the batch, the directory of the key), four the first time a key is archived: about three times a delete (measured: 12 ms against 37 ms per key on a busy NVMe, 3,000 keys).The followers see what a delete shows them: an archive changes nothing on their side.
By hand: the
archive-key-filescommand ofC_TRANGER, into itsarchive_directoryattribute.
tranger2_mask_sorts_as_time()¶
Do the file ids of a filename_mask (a strftime() format) sort as time?
tranger2_delete_key_files() deletes the files
whose id sorts BEFORE the id of the cut, comparing them as strings, so it
refuses a topic whose mask fails this test: with such a mask, it cannot tell
which of its files are old.
The ids sort as time when every field is numeric, of a fixed width, the
YEAR first, and each field the NEXT one down from the last, with literal text
anywhere: a field left out mixes the ones below it (%Y-%d puts the 9th of
March before the 10th of February). The fields that can follow each one:
| Field | Can be followed by |
|---|---|
| (first) | %Y, %G, %F (%Y-%m-%d) |
%Y | %m, %j |
%m | %d |
%G (the year of the ISO week) | %V (and %G cannot end the mask) |
%V | %u |
%F, %j, %d, %u | %H, %R (%H:%M), %T (%H:%M:%S) |
%H | %M |
%M, %R | %S |
%% is a literal. %s is refused: the epoch has no fixed width (9 digits
before 2001-09-09).
BOOL tranger2_mask_sorts_as_time(
const char *filename_mask
);Parameters
| Key | Type | Description |
|---|---|---|
filename_mask | const char * | The mask, as in the filename_mask of a topic. |
Returns
TRUE when the ids of the files it names sort as time.
Example
tranger2_mask_sorts_as_time("tracks-%Y-%m-%d"); // TRUE
tranger2_mask_sorts_as_time("series-1d-%Y"); // TRUE
tranger2_mask_sorts_as_time("%FT%T"); // TRUE
tranger2_mask_sorts_as_time("%G-W%V"); // TRUE (the year of the week, then its week)
tranger2_mask_sorts_as_time("%Y-%d"); // FALSE: the month left out
tranger2_mask_sorts_as_time("%s"); // FALSE: the epoch has no fixed width
tranger2_mask_sorts_as_time("queue"); // FALSE: a tr_queue topic, one file
tranger2_mask_sorts_as_time("%d-%m-%Y"); // FALSE: the year last
tranger2_mask_sorts_as_time("%Y-%d-%m"); // FALSE: the day before the month
tranger2_mask_sorts_as_time("%Y-%b"); // FALSE: "Aug" sorts before "Jun"
tranger2_mask_sorts_as_time("%Y-%e"); // FALSE: " 1" is not "01"tranger2_record_in_purged_period()¶
Does a record fall in a purged period of its key? TRUE when
tranger2_append_record() refuses it for that
(“Cannot append record, it falls in a purged period of its key”): its __t__
names a file at or before the upto of the key’s base
(tranger2_delete_key_files()). The append logs
that refusal as an ERROR. A caller that must give the cause to a PEER asks
first, and answers it. C_TRANGER’s add-record does:
“record refused by topic ‘raw_events’: a t in a purged period of its key:
its old files were deleted”.
__t__ and record are the ones the append takes. FALSE for __t__ = 0
(now: no purge reaches it), a key with no base, a key not in the topic, and a
record whose key or time cannot be read (the append says why). A key flagged
unlisted (its .purged, or its directory, was not readable when the topic
opened) has no base in its cache: its base is read from the disk, quietly, as
the append reads it when it lists the key again. A base that can be read gives
the same answer as the append. A base that still cannot be read (a .purged
that is corrupt or unreadable, a directory that cannot be listed) gives
FALSE, and the append refuses the record for that other cause, with an
ERROR “Cannot append record, its key cannot be listed: its row would follow
rows no cell counts”: the two agree only that the record is not appended,
and only the append says so, in the log. Silent, but for a topic that does not
exist (logged).
BOOL tranger2_record_in_purged_period(
json_t *tranger,
const char *topic_name,
uint64_t __t__,
json_t *record // not owned
);Parameters
| Key | Type | Description |
|---|---|---|
tranger | json_t * | The TimeRanger database. |
topic_name | const char * | The topic of the record. |
__t__ | uint64_t | The time of the append, in the topic’s unit of t (milliseconds with sf_t_ms). 0 is now. |
record | json_t * | The record, with the topic’s pkey. Not owned. |
Returns
TRUE when the record falls in a purged period of its key.
Example
/*
* Key "dev1" of a topic of one file a day, purged up to 2026-09-22
* (tranger2_delete_key_files(tranger, "raw_events", "dev1", <2026-09-23>))
*/
json_t *record = json_pack("{s:s, s:I}", "id", "dev1", "tm", (json_int_t)1790035200);
if(tranger2_record_in_purged_period(tranger, "raw_events", 1790035200, record)) {
// 2026-09-22: refused, say so to whoever sent it
} else {
md2_record_ex_t md;
tranger2_append_record(tranger, "raw_events", 1790035200, 0, &md, json_incref(record));
}
JSON_DECREF(record)tranger2_delete_instance()¶
The tranger2_delete_instance() function deletes a single instance
(one row of a key’s .md2 index) without touching the surrounding
rows. The row’s metadata is mutated in place: sf_deleted_instance
(bit 0x0400 of system_flag2_t, on the inherited side of the mask
so rt_by_disk followers see the same tombstone) is ORed into the
system flag bits. The .json data log stays append-only.
Read paths (tranger2_open_iterator history loop,
tranger2_iterator_get_page, and the rt_by_disk follower) skip rows
whose tombstone bit is set. Treedb is downstream of those paths and
inherits the skip with no tr_treedb change. tranger2_read_record_content()
and tranger2_read_user_flag() deliberately do not filter — the
caller has already located the row — so they remain available for
audit / wipe-verification tooling.
Irrecoverable. Master-only. Second delete of the same row is a silent
no-op (returns 0).
int tranger2_delete_instance(
json_t *tranger,
const char *topic_name,
const char *key,
uint64_t __t__,
uint64_t rowid, // per-key i_rowid, slot in .md2, based 1
BOOL zero_payload
);Parameters
| Key | Type | Description |
|---|---|---|
tranger | json_t * | A pointer to the TimeRanger database instance. |
topic_name | const char * | The topic that owns the key. |
key | const char * | The key whose instance is to be deleted. |
__t__ | uint64_t | Time of the instance (matches the value passed to tranger2_append_record). |
rowid | uint64_t | Per-key slot index in the .md2 file, based 1. |
zero_payload | BOOL | If TRUE, also overwrites the matching __size__ bytes at __offset__ in the .json data file with zeros. Opt-in: after the wipe the JSON is no longer a parseable concatenation — only the .md2 index makes sense of it. Use for sensitive-data wipes. |
Returns
Returns 0 on success, a negative value on failure.
Notes
Side effects to be aware of:
rowids do not renumber. Slot N stays slot N (dark).iterator_size(),total_rows,pagescount slots, not live rows. A page returningdata.length=2withtotal_rows=3is the contract for “1 dead in this segment”.topic_cachecells (fr_t/to_t/fr_tm/to_tm) are not refreshed. If the deleted instance was the min/max t/tm of its file, the cell rollup can lie. Cheap to fix on next cold reload. Expensive to fix incrementally. Deferred until a consumer needs it.With
zero_payload, the row is marked deleted BEFORE the wipe. A wipe that fails answers-1with the row already dead. A wipe cut part way (the file size limit, a full disk) is logged as “Cannot zero the payload, short write: the file size limit or the disk is full”, with the byteswrittenandexpected, not with anerrno.
tranger2_set_rt_key_deleted_callback()¶
Registers a key-delete callback on a handle returned by
tranger2_open_rt_mem(), tranger2_open_rt_disk() or
tranger2_open_iterator(). The callback fires when
tranger2_delete_key() runs against a key
this subscriber tracks:
On the master, directly from
tranger2_delete_key().On
rt_by_diskfollowers, from the inotify watcher ondisks/<rt_id>/when the master mirrors the deletion.
The subscriber’s filter is honoured, the same one its records go through:
a specific key matches that one, an rkey (in match_cond) the keys its
regular expression matches, and neither of them every deletion. Up to
7.25.20 only key was looked at, and a feed opened with an rkey was told
every key deleted:
json_t *rt = tranger2_open_rt_mem(
tranger, "devices", "", json_pack("{s:s}", "rkey", "^DVES_"),
on_record, "dves-card", "", NULL
);
tranger2_set_rt_key_deleted_callback(rt, on_key_deleted, NULL);
tranger2_delete_key(tranger, "devices", "OTHER_1"); // on_key_deleted not called
tranger2_delete_key(tranger, "devices", "DVES_000000"); // called onceAdditive: existing handles default to “no callback”, behavior
unchanged. Passing cb=NULL clears any previously registered callback.
typedef int (*tranger2_key_deleted_callback_t)(
json_t *tranger,
json_t *topic,
const char *key, // the deleted key
json_t *list, // iterator / rt_mem / rt_disk entry, don't own
void *user_data
);
int tranger2_set_rt_key_deleted_callback(
json_t *list,
tranger2_key_deleted_callback_t cb,
void *user_data
);Parameters
| Key | Type | Description |
|---|---|---|
list | json_t * | A handle previously returned by tranger2_open_rt_mem, tranger2_open_rt_disk or tranger2_open_iterator. |
cb | tranger2_key_deleted_callback_t | Function to invoke, or NULL to clear. |
user_data | void * | Opaque pointer echoed back to the callback. |
Returns
Returns 0 on success, -1 if list is NULL.
Example
A follower feed that closes itself when the key it follows is deleted:
PRIVATE int on_key_deleted(
json_t *tranger, json_t *topic, const char *key, json_t *list, void *user_data)
{
tranger2_close_rt_disk(tranger, list); // allowed: see the notes
return 0;
}
/* The loop is the PARAMETER: a "yev_loop" key in the config is overwritten */
json_t *tranger = tranger2_startup(0, jn_tranger, yev_loop);
json_t *rt = tranger2_open_rt_disk(
tranger, "devices", "DVES_000000", NULL, on_record, "live-card", "", NULL
);
tranger2_set_rt_key_deleted_callback(rt, on_key_deleted, NULL);Notes
The callback fires once per feed for each delete. An rt_disk feed that owns an inotify watcher hears the delete only from its own directory, and the master-side fan-out skips it. Until 2026-09-15 a feed was fired once for every inotify event of every feed of the topic (six times in a three-feed follower), and not at all for a key that had no record since the feed opened.
The callback can close its own feed (tranger2_close_rt_disk()). The
watcher is stopped at the end of the batch it is handling, and the rest
of that batch is dropped.
The feed’s watcher exists only when the tranger was started with a loop:
pass it as the third argument of tranger2_startup(). Without a loop an
rt_disk has no watcher, and it is fired directly by the master’s delete
path, like an rt_mem.
tranger2_delete_topic()¶
The tranger2_delete_topic() function deletes a topic from the TimeRanger database, effectively removing all associated records and metadata.
int tranger2_delete_topic(
json_t *tranger,
const char *topic_name
);Parameters
| Key | Type | Description |
|---|---|---|
tranger | json_t * | A pointer to the TimeRanger database instance. |
topic_name | const char * | The name of the topic to be deleted. |
Returns
Returns 0 on success, or a negative value if an error occurs.
Notes
Deleting a topic is irreversible. Make sure that the topic is no longer needed before calling tranger2_delete_topic().
Only a master can delete: on a replica (master=0) it returns -1 with “Only master can delete”, and the directory stays. The topic name follows the topic name rule: tranger2_delete_topic(tranger, "../other_db/users") returns -1 and deletes nothing.
tranger2_dict_topic_desc_cols()¶
tranger2_dict_topic_desc_cols() retrieves the column descriptions of a specified topic in dictionary format.
json_t *tranger2_dict_topic_desc_cols(
json_t *tranger,
const char *topic_name
);Parameters
| Key | Type | Description |
|---|---|---|
tranger | json_t * | A reference to the TimeRanger database instance. |
topic_name | const char * | The name of the topic whose column descriptions are to be retrieved. |
Returns
A JSON object containing the column descriptions of the specified topic. The caller must decrement the reference count of the returned JSON object.
Notes
This function is similar to tranger2_list_topic_desc_cols(), but returns the data in dictionary format instead of a list.
tranger2_get_iterator_by_id()¶
Retrieve an iterator by its identifier. If the iterator exists, it is returned. Otherwise, NULL is returned.
json_t *tranger2_get_iterator_by_id(
json_t *tranger,
const char *topic_name,
const char *iterator_id,
const char *creator
);Parameters
| Key | Type | Description |
|---|---|---|
tranger | json_t * | Pointer to the TimeRanger database instance. |
topic_name | const char * | Name of the topic associated with the iterator. |
iterator_id | const char * | Unique identifier of the iterator to retrieve. |
creator | const char * | Identifier of the entity that created the iterator. NULL or empty matches only creatorless iterators. To reach one registered with a creator, pass that same creator. |
Returns
Returns a pointer to the iterator JSON object if found. Otherwise, returns NULL.
Notes
This function does not produce any error messages if the iterator is not found.
The creator filters the match: pass the same creator used at open. An empty
creator matches only entries that were themselves opened without a creator.
It is one hash lookup (since 7.25.0): the topic keeps
iterators_by_id — {creator: {id: iterator}} — beside its iterators
array. It used to walk the array, and tranger2_open_iterator()
calls it to refuse a duplicate, so opening N iterators on a topic was O(N²).
It also opens the topic when it is closed (it goes through tranger2_topic()).
To ask whether a handle you registered is still alive, check
tranger2_topic_is_open() first, then look the handle up by its id: a topic
closed and opened again has the same name and none of the old handles.
// Is my iterator "it1" still there? Never trust a pointer kept from the open.
json_t *it = tranger2_topic_is_open(tranger, "frames")?
tranger2_get_iterator_by_id(tranger, "frames", "it1", "my_service") : NULL;tranger2_get_rt_disk_by_id()¶
Retrieve a real-time disk instance by its identifier. If the specified real-time disk exists, it returns the corresponding JSON object. Otherwise, it returns NULL.
json_t *tranger2_get_rt_disk_by_id(
json_t *tranger,
const char *topic_name,
const char *rt_id,
const char *creator
);Parameters
| Key | Type | Description |
|---|---|---|
tranger | json_t * | Pointer to the TimeRanger database instance. |
topic_name | const char * | Name of the topic associated with the real-time disk. |
rt_id | const char * | Identifier of the real-time disk to retrieve. |
creator | const char * | Creator identifier used to filter the real-time disk instances. NULL or empty matches only creatorless feeds. To reach one registered with a creator, pass that same creator. |
Returns
Returns a JSON object representing the real-time disk instance if found. Otherwise, returns NULL.
Notes
This function does not produce any error messages if the real-time disk is not found.
The creator filters the match: pass the same creator used at open. An empty
creator matches only entries that were themselves opened without a creator.
tranger2_get_rt_mem_by_id()¶
Retrieve a real-time memory instance by its identifier. If the specified real-time memory instance exists, it is returned. Otherwise, NULL is returned.
json_t *tranger2_get_rt_mem_by_id(
json_t *tranger,
const char *topic_name,
const char *rt_id,
const char *creator
);Parameters
| Key | Type | Description |
|---|---|---|
tranger | json_t * | Pointer to the TimeRanger database instance. |
topic_name | const char * | Name of the topic associated with the real-time memory instance. |
rt_id | const char * | Identifier of the real-time memory instance to retrieve. |
creator | const char * | Creator identifier used to filter the real-time memory instance. NULL or empty matches only creatorless feeds. To reach one registered with a creator, pass that same creator. |
Returns
Returns a pointer to the real-time memory instance as a json_t * if found, otherwise returns NULL.
Notes
This function does not produce any internal logging or error messages if the requested real-time memory instance is not found.
The creator filters the match: pass the same creator used at open. An empty
creator matches only entries that were themselves opened without a creator.
tranger2_iterator_get_page()¶
Retrieves a page of records from an iterator in the TimeRanger database. The function returns a JSON object containing the total number of rows, the number of pages based on the specified limit, and the list of retrieved records.
json_t *tranger2_iterator_get_page(
json_t *tranger,
json_t *iterator,
json_int_t from_rowid, // based 1
size_t limit,
BOOL backward
);Parameters
| Key | Type | Description |
|---|---|---|
tranger | json_t * | A pointer to the TimeRanger database instance. |
iterator | json_t * | A pointer to the iterator from which records will be retrieved. |
from_rowid | json_int_t | The starting position of the page, 1-based. See the note below: it is a position among the rows the iterator RETURNS, which is a global rowid only when the iterator does not filter. |
limit | size_t | The maximum number of records to retrieve in the page. |
backward | BOOL | If TRUE, records are retrieved in reverse order. Otherwise, they are retrieved in forward order. |
Returns
A JSON object containing the retrieved records, total row count, and pagination details. The caller owns the returned JSON object and must manage its memory.
Notes
This function is useful for paginating through records in an iterator. The from_rowid parameter determines the starting point, and the limit parameter controls the number of records retrieved per page.
A filtered iterator (one opened with any match_cond condition) pages over
its row index, so total_rows, pages and the page contents count only the
records that MATCH — and from_rowid is a position among those, not a global
rowid. An unfiltered iterator has no index, and its positions are the global
rowids. See tranger2_open_iterator().
An unfiltered iterator pages the key as it is: its segments are taken
from the cache again when the key’s cache moved since they were taken (its
rows, its number of files, its last file), so the rows appended since the open
are readable and total_rows / pages (the live count) can be paged to the
end. A filtered iterator keeps the index and the segments of its open.
A delete of the key drops them: an unfiltered iterator takes its segments again at its next page (a key written again with the same number of rows and files, spread another way, is not the same key), and a filtered one pages an empty index -- the rows it indexed are gone. A page that finds the key gone from disk behind its back (a replica whose master deleted it) logs “Key gone from disk while its iterator was open” and stops.
json_t *it = tranger2_open_iterator(tranger, "topic", "K", NULL, NULL, "pager", "", NULL, NULL);
// K: 2 rows on day 1, 1 on day 2
tranger2_delete_key(tranger, "topic", "K");
// K written again: 1 row on day 1, 2 on day 2
json_t *page = tranger2_iterator_get_page(tranger, it, 1, 100, FALSE);
// page["data"]: the 3 new rows (7.25.4: 1 row, and a CRITICAL)json_t *it = tranger2_open_iterator(tranger, "topic", "key", NULL, NULL, "it1", "", NULL, NULL);
/* ... 3 records appended to "key" since ... */
json_t *page = tranger2_iterator_get_page(tranger, it, 1, 100, FALSE);
/* page["data"] holds the 3 new rows too; page["total_rows"] counts them */A purge of the old files of the key
(tranger2_delete_key_files()) moves no
position. The positions of an unfiltered iterator count from the base the key
had when it opened (position N is g_rowid = base + N), and the ones of the
rows purged since are returned empty. A filtered iterator keeps its index, which
holds g_rowids, and skips the rows purged. Counted from the base of NOW, every
position moved by the rows purged, and a client paging forward skipped live
rows.
// K: 2 rows a day, days 1-3 (g_rowids 1-6), pages of 2
json_t *it = tranger2_open_iterator(tranger, "topic", "K", json_object(), NULL, "pager", "", NULL, NULL);
tranger2_iterator_get_page(tranger, it, 1, 2, FALSE); // A1:1 A2:2
tranger2_delete_key_files(tranger, "topic", "K", day2); // the 2 rows of day 1
tranger2_iterator_get_page(tranger, it, 3, 2, FALSE); // B1:3 B2:4 -- the next page, nothing skipped
tranger2_iterator_get_page(tranger, it, 1, 2, FALSE); // empty: purged
tranger2_iterator_size(it); // 6: its positionstranger2_iterator_size()¶
tranger2_iterator_size() returns the number of records in the specified iterator.
For a filtered iterator, the size of its index. For an unfiltered one, its
page positions: on a key purged since the open
(tranger2_delete_key_files()) they include the
positions of the rows purged, which are returned empty.
size_t tranger2_iterator_size(
json_t *iterator
);Parameters
| Key | Type | Description |
|---|---|---|
iterator | json_t * | The iterator whose size is to be determined. |
Returns
The number of records the iterator will return.
Notes
A filtered iterator answers with its index: the exact number of records
matching its match_cond. An unfiltered one sums its segments — the same
thing when nothing is filtered out.
tranger2_list_keys()¶
Returns a JSON array with the key names of a topic, read from its in-memory
cache.
json_t *tranger2_list_keys(
json_t *tranger,
const char *topic_name
);Parameters
| Key | Type | Description |
|---|---|---|
tranger | json_t * | A pointer to the TimeRanger database instance. |
topic_name | const char * | The name of the topic from which to retrieve the keys. |
Returns
A JSON array of key-name strings. The caller owns the returned array and must free it when no longer needed. Never NULL — a missing topic yields an empty array.
Notes
The keys are from the topic’s in-memory cache, not a disk scan. Slow for
thousands of keys (allocates one string per key).
tranger2_list_topic_desc_cols()¶
Returns a JSON array containing the column descriptions of a topic in the TimeRanger database. The returned JSON object must be decremented after use.
json_t *tranger2_list_topic_desc_cols(
json_t *tranger,
const char *topic_name
);Parameters
| Key | Type | Description |
|---|---|---|
tranger | json_t * | A JSON object representing the TimeRanger database instance. |
topic_name | const char * | The name of the topic whose column descriptions are to be retrieved. |
Returns
A JSON array containing the column descriptions of the specified topic. The caller must decrement the reference count of the returned JSON object after use.
Notes
This function was previously known as tranger_list_topic_desc().
tranger2_list_topics()¶
Returns a JSON array with the names of the topics currently opened in memory
(the tranger["topics"] registry) — not a disk scan.
json_t *tranger2_list_topics(
json_t *tranger
);Parameters
| Key | Type | Description |
|---|---|---|
tranger | json_t * | The tranger database instance. |
Returns
A JSON array with the names of the currently-opened topics. The caller owns the returned array and must free it. Never NULL — an empty array when no topic is opened.
Notes
This lists only topics already opened in memory. To list every topic present on
disk — including ones not yet opened — use
tranger2_list_topic_names().
tranger2_open_iterator()¶
Opens an iterator for traversing records in a topic within the TimeRanger database. The iterator allows filtering records based on specified conditions and supports real-time data loading.
json_t *tranger2_open_iterator(
json_t *tranger,
const char *topic_name,
const char *key, // required
json_t *match_cond, // owned
tranger2_load_record_callback_t load_record_callback, // called on LOADING and APPENDING, optional
const char *iterator_id, // iterator id, optional, if empty will be the key
const char *creator, // creator
json_t *data, // JSON array, if not empty, fills it with the LOADING data, not owned
json_t *extra // owned, user data, this json will be added to the return iterator
);Parameters
| Key | Type | Description |
|---|---|---|
tranger | json_t * | Pointer to the TimeRanger database instance. |
topic_name | const char * | Name of the topic to iterate over. |
key | const char * | Specific key to filter records. Required. |
match_cond | json_t * | JSON object containing filtering conditions. Owned by the function. |
load_record_callback | tranger2_load_record_callback_t | Callback function invoked during data loading and appending. Optional. |
iterator_id | const char * | Unique identifier for the iterator. If empty, the key is used. |
creator | const char * | Identifier of the entity creating the iterator. |
data | json_t * | JSON array to store loaded data. Not owned by the function. |
extra | json_t * | Additional user data attached to the iterator. Owned by the function. |
Returns
Returns a JSON object representing the iterator. The topic owns it: close it
with tranger2_close_iterator().
NULL (logged, and match_cond and extra are consumed all the same) when the
topic cannot be opened, key is empty, an iterator with this iterator_id
and creator is already open, or the index of a filtered paging iterator
cannot be built.
Notes
The iterator supports real-time data loading and filtering based on various conditions. Use tranger2_close_iterator() to release resources when done.
A load that stops half way says so. When the LOADING (a callback, or
data) meets a row whose metadata or whose CONTENT cannot be read, it stops
there, logs it, and leaves "load_failed": true in the iterator. The rows
before it, in the load’s direction, were handed (forward: the oldest,
backward: the newest). What the callback saw, and what data holds, is then
NOT the whole history: a caller that asks the history a question (is any
record of this key frozen by a snapshot?) must read the flag before it takes a
“no”. Up to 7.25.4 a record whose content was not readable was handed to the
callback as NULL and the load went on. An only_md load reads no content,
and no content fails it.
After a restart too. The cache of a topic is built from disk when it is
opened. A .md2 file of a key that it cannot count flags the key: one that
cannot be opened or read. It logs “md2 file of the key unreadable when its cache was built: every load of
the key says load_failed” once, at the open, and then every iterator of the
key -- paging ones too -- logs “The history of the key is not whole: a md2
file of it could not be read when its cache was built” and says
load_failed. A loading stops where the first flagged file is in its
direction, the place a running tranger stops when the damage happens behind
its back. Key A with the files of days 1, 2 and 3, and the .md2 of day 2
made unreadable (mode 0000) while the yuno was down:
forward load -> the rows of day 1, then load_failed
backward load -> the rows of day 3, then load_failedA key directory that cannot be LISTED is not an empty key. When the
cache build cannot open keys/<key>/ (EMFILE, EACCES, ENOMEM), cannot
READ it (readdir() fails: EIO, ESTALE, and 7.25.4 took that as
the end of the directory, and the key was read without the files not listed
yet), or has no memory for an entry of the listing, none of the key’s files is
known, so the
key is flagged whole: it logs “key directory cannot be listed when its cache
was built: every load of the key says load_failed” once, and every iterator of
the key logs “The history of the key is not whole: its directory could not be
listed when its cache was built” and says load_failed with no row handed. A
keyless tranger2_open_list() names the key in
load_failed_keys, so treedb refuses a create of that id. An append into the
key lists it again first: while the directory still cannot be listed the
append is refused (“Cannot append record, its key cannot be listed: its row
would follow rows no cell counts”). Once it can, the key’s cache is built from
the disk, the flag goes (“key directory listed again”, an INFO) and the append
goes on. Up to 7.25.4 the key loaded as an EMPTY key with load_failed false,
and an append counted its row as the first of its file.
keys/A of mode 0000 at the open:
iterator of A -> no row, load_failed
keyless list -> the rows of B, load_failed, load_failed_keys ["A"]
append into A -> -1 (refused)
keys/A of mode 02770 again:
append into A -> 0, the key listed again
iterator of A -> every row of A, the new one last, load_failed falseWhen keys/ itself cannot be listed, no key can be flagged by name: the topic
is not opened (see tranger2_open_topic()).
A flag goes when its cause goes, on a replica too. A flag is not kept until the topic is opened again:
A load tries the flags again first.
tranger2_open_iterator()(and sotranger2_open_list()) of a flagged key lists the key again when its directory changed since it was flagged (its mode or its entries:ctime) or failed to open then and opens now, and counts a flagged.md2again when it changed since it was flagged (its mode or its content:ctimeor size) or failed to open then and opens now. A flag whose directory or file looks as it did is not tried: a directory that cannot be read is not listed, a damaged file not read, and their failure not logged, at every load. What still cannot be read stays flagged, and the load saysload_failedwith nothing more in the log. So a directory whosereaddir()recovers without changing stays flagged until it changes, until an append of the master or a notification of a replica lists it (both list it whatever it looks like), or until the topic is opened again (tests/c/timeranger2/test_unlisted_relist_once):keys/A opens, its readdir() fails (EIO) at the open -> A flagged, logged once three loads of A -> load_failed, nothing more in the log readdir() works again, keys/A as it was -> a load: load_failed, not listed chmod keys/A (its ctime moves) -> a load: INFO key directory listed again, every row, load_failed falseA replica’s notification lists the key again. When the master appends to a key that the replica was not able to list, the notification of the new row (an
rt_diskfeed) lists the key first. The rows of the notified file, which nobody was handed, are published to the feeds with their rowids in the WHOLE key. The rows of the key’s other files are history, for a load. While the directory still cannot be listed, nothing is read (“New records of the key not read: its directory cannot be listed”).A master’s append counts a flagged file, and lists an unlisted key, before it writes (see above).
replica opens, keys/A of mode 0000 -> A flagged unlisted, feed "rtALL" open
chmod 02770 keys/A; the master appends A (its 4th row)
replica, at the notification -> INFO key directory listed again,
rtALL gets the row, rowid 4
replica: iterator of A -> A@1 A@2 A@3 A@4, load_failed falseIn 7.25.4 a key whose directory listing failed loaded as an empty key (see above), and the notification counted the notified file alone as the whole key (the feed got rowid 1 for the key’s 4th row).
A .md2 of 0 rows gets no cell and flags nothing. With an EMPTY .json it
loses nothing. With a .json that is NOT empty it is what an append that was
never acknowledged leaves (the content is written first, the md2 row after,
and the row was never written: the md2 failed, or the process died between
the two). That is not damage: the file is ignored, as until 7.25.4, with a
warning that names it, and the key loads from its other files:
WARNING: md2 file of the key with no rows and a content file that is not empty:
an append that was never acknowledged, the file is ignored
topic_directory=<store>/devices key=A file_id=2000-01-02 content_size=32
forward load -> the rows of days 1 and 3, load_failed falseThe next
append into that day’s file names its content with a row of its own, and the
warning goes. A .md2 cut to 0 bytes behind the yuno’s back has the same
shape, and loses the rows of its file the same way it did until 7.25.4: a
.json much larger than one record is the sign. Put the pair back from a
backup.
A .md2 whose last row is torn is not damage either. Its size is not a
whole number of 32-byte rows: a power cut came during the write of a row. The
md2 row is the commit point of an append, and an append is acknowledged only
after the whole row is written, so a torn row is an append that was never
acknowledged. A MASTER cuts the .md2 back to its whole rows (to
floor(size / 32) * 32 bytes) when it first reads the file, logs one warning,
and the key loads whole. The next append goes into the same file:
WARNING: md2 file of the key ends in a part of a row: an append that was never
acknowledged was cut back
topic=devices key=A file_id=2000-01-02
path=<store>/devices/keys/A/2000-01-02.md2 old_size=109 new_size=96
forward load -> the rows of days 1, 2 (3 rows) and 3, load_failed falseThe cut removes fewer than 32 bytes, all after the last whole row. The
.json is left as it is: its bytes after the last row belong to no row, and
the next append writes at its end. If the cut itself fails, that is damage:
the file is flagged as above (“Cannot cut back a md2 file that ends in a part
of a row: the file is damaged”). A torn first row (a .md2 of fewer than 32
bytes) is cut to 0 bytes, and the file is then a .md2 of 0 rows, ignored
with the warning above.
Not every tail on no row boundary is a torn row. 7.25.4 did not cut a torn
row back: the next appends wrote their rows after the torn bytes, on no row
boundary, and those rows were acknowledged. In such a file the last 32 bytes
are a whole row, and a cut removes the end of it. So the tail is cut only when
it is a torn row after good rows. A row is good when its content is WHOLE:
inside the .json, one json value and one NUL byte after it, as an append
writes it (or only zero bytes, for an instance deleted with its content
zeroed, see tranger2_delete_instance()). A NUL
inside a string is written as the escape \u0000, so it is part of the value
and is not a NUL byte. A content that the process that checks it has not the
memory to parse is not a content that is not whole: the largest memory block
(gbmem_get_maximum_block()) is set by MEM_MAX_BLOCK for each yuno, and the
yuno that wrote it can have a larger block. A content larger than that block
is read in parts: a NUL before its last byte says that it is not whole, with
no block of its size, and when its only NUL is its last byte the check cannot
tell. A content smaller than the block can still need more to parse (a long
string, a long array or object: jansson doubles its buffers and tables). When
its parse fails for memory, the check cannot tell either. The tail is not cut
on a content that the check cannot tell. The two rules, which must both hold:
The last 32 bytes of the
.md2, read as a row, are NOT a row of their own: their content does not end exactly at the end of the.json(__offset__ + __size__equal to its size), and it is not whole.The last whole row is good, and so is the whole row before it, whose content ends at or before the start of the content of the last one.
Rule 1 stops the cut of every file that 7.25.4 left whose last row still
names its content as 7.25.4 wrote it, also when the .json has content after
the last row (an append killed between its content and its md2 row, and 7.25.4
did not cut that content back): the last row is a real row, and its content
is whole wherever the .json ends. When the process that checks has not the
memory to parse that content, the check cannot tell, and that is enough to
not cut. Only a content damaged since (its bytes changed) lets such a file
reach rule 2. For a torn row, the last 32 bytes
are the end of the last whole row and the start of the torn one: read as a
row, their __offset__ and __size__ are bytes moved out of their fields.
They name a whole record of the .json, to the byte, only by a coincidence
of 64-bit values, and then the file is flagged, not cut. Rule 2 stops a cut
when the rows before the tail are not good. Example: a topic with no tkey
(its __tm__ is 0, as in a treedb topic) and records of 256 bytes, so every
__offset__ is a multiple of 256. There, a row moved by one byte names a
range INSIDE the .json (the real one divided by 256). A test of the range
only takes it for a good row. It is not whole: it does not start where a
record starts.
When a rule fails, the file is NOT cut: a CRITICAL names the shape and the
cause, the file is flagged as above, and the key says load_failed until
the file is repaired by hand (see
the repair in treedb). The
bytes of the file do not change:
CRITICAL: md2 file of the key ends in a whole row that is not on a row boundary:
written by 7.25.4 after a torn row; not cut, repair it by hand
cause="its content is a whole record of the content file"
topic=devices key=A file_id=2000-01-02
path=<store>/devices/keys/A/2000-01-02.md2 md2_size=173
content_size=242 row_at=141 __offset__=160 __size__=32
CRITICAL: md2 file of the key ends in a part of a row after a last whole row
that is not valid: not cut, repair it by hand
cause="its content is not a record"
topic=devices key=A file_id=2000-01-02 ... row_at=64The cause of the first is “its content ends the content file”, “its
content is a whole record of the content file” or “its content has no NUL
but the one at its end, and this process has not the memory to parse it
(MEM_MAX_BLOCK): it can be a record written by a yuno with a larger block”.
The cause of the second is one of “its content is not inside the content
file”, “its content is not a record”, “this process has not the memory to
parse its content (MEM_MAX_BLOCK): it cannot be checked”, “the whole row
before it is not a good row” or “this process has not the memory to parse
the content of the whole row before it (MEM_MAX_BLOCK): it cannot be
checked” (for these two, row_at is that row), and “its content starts
before the end of the content of the row before it”.
Example: a master with a block of 64 KiB opens a file that 7.25.4 left, whose last row names a record of 100 KiB, written by a yuno with a larger block. The file is flagged, and no row is lost. The same when the record is a string of 40 000 bytes, or an array in 20 000 bytes: it is smaller than the block, but its parse needs more (the log also has the ERROR “SIZE GREATER THAN MAX_BLOCK” of the allocation that failed):
CRITICAL: md2 file of the key ends in a whole row that is not on a row boundary:
written by 7.25.4 after a torn row; not cut, repair it by hand
cause="its content has no NUL but the one at its end, and this process
has not the memory to parse it (MEM_MAX_BLOCK): it can be a record
written by a yuno with a larger block"
topic=devices key=A file_id=2000-01-01 md2_size=159 row_at=127
__offset__=1024 __size__=102400When the check cannot RUN, a CRITICAL says so, and the file is not cut. It is one of three:
“Cannot check the torn tail of a md2 file, its content file cannot be read: not cut”: the
.jsoncannot be opened (EMFILE,ENFILE,EACCES) or read. With thepathof the.json.“Cannot check the torn tail of a md2 file, no memory to read a content: not cut”: with the
pathof the.jsonand thesizeasked for.“Cannot read a record of md2 file, read FAILED” or “Cannot read a record of md2 file, short read”: a row of the
.md2cannot be read. With thepathof the.md2, therow(end,lastorbefore last) and itsoffset.
CRITICAL: Cannot check the torn tail of a md2 file, its content file cannot be
read: not cut
topic=devices key=A file_id=2000-01-02
path=<store>/devices/keys/A/2000-01-02.json errno=13 serrno="Permission denied"At an open the file is flagged too: without the check, the whole rows can be
the moved rows of the shape 7.25.4 left. A master’s next append into the file
counts it again, and the check runs then. At an append the file is not
flagged (see tranger2_append_record()).
A replica makes the same check, and flags the file the same way: it does not read rows that are not on a row boundary.
A REPLICA never writes. It reads only the whole rows of such a file, and logs
nothing for the torn row: a replica also sees a torn row while a live master
is writing it, and counts the row when the master’s next notification of the
file comes. A file with no whole row (fewer than 32 bytes) is a .md2 of 0
rows for the replica, so when its .json is not empty the replica logs the
0-rows warning above, and does not cut. When the master opens the store, it
cuts the file back. Up to 7.25.4 the cache build logged a CRITICAL (“Cannot
read last record, md2 file corrupted”) and left the whole file out of the
key, on a master and on a replica: its acknowledged rows were missing from
every load, and nothing failed.
The flag of a file goes once a cell counts it again: an append into the file
that finds it readable (see tranger2_append_record()),
on a follower the file read whole from the disk after the master wrote it, or a
load of the key once the file changed or opens again (see A flag goes when its
cause goes below).
tranger2_delete_key() clears every flag with the key.
Up to 7.25.4 the cache build dropped an unreadable file and nothing failed: the
key read as a shorter key.
json_t *data = json_array();
json_t *it = tranger2_open_iterator(
tranger, "devices", "dev-1", NULL, NULL, "", "me", data, NULL
);
if(!it) {
// not opened, the cause is in the log
} else if(json_is_true(json_object_get(it, "load_failed"))) {
// data holds only a part of the history: do not decide on it
}
if(it) {
tranger2_close_iterator(tranger, it);
}
JSON_DECREF(data)A deleted key. A delete of the key
(tranger2_delete_key(), or a replica’s key-deleted
notice) drops what its iterators took from it: an unfiltered iterator takes its
segments again at its next page, a filtered one keeps an empty index (its rows
are gone, and an index is built only at the open).
match_cond — the conditions an iterator honors
backward
only_md (don't load jn_record on calling callbacks; honored by
the historical load AND the realtime rt_mem/rt_disk
feeds, which hand the callback NULL jn_record)
rkey (str) PCRE2 regex over keys ("" == ".*"), for a KEYLESS
list (no `key`) = the whole topic, narrowed to the keys
matching it. Honored by BOTH halves: the disk load only
visits matching keys, and the realtime feed only
dispatches appends of matching keys.
from_rowid / to_rowid
from_t / to_t t: PERSISTENCE time (when the record was appended)
from_tm / to_tm tm: MESSAGE time (the record's tkey field)
user_flag / not_user_flag / user_flag_mask_set / user_flag_mask_notsett and tm are two independent axes and their ranges combine (AND). Both
are expressed in the topic’s own unit: seconds, or milliseconds when the
topic sets sf_t_ms / sf_tm_ms.
A time bound can be given as text, read once when the iterator, the list
or the feed (rt_mem, rt_disk) opens, and written back into its match_cond
as an integer, by tranger2_time_of_json_text() --
the same rule as a text tkey:
""is no bound (0).The text of a JSON integer,
-?(0|[1-9][0-9]*)(the command parser’s grammar), is that integer:"-86400"is relative, as below,"-0"is0.A text written as another number is refused: blanks around it (
" 5","5 "), leading zeros ("007"), a+("+5"), decimals or an exponent ("10.5","1E5"), a decimal comma ("1,5",",5"), or words that are all digits ("5 5","2026 10 01","2026, 10, 01"). It is never read as a date.So is a text with such numbers among other words: a comma between two digits (
"1,5 days ago","1 2,10:00"), or two or more words of digits in a row ("5 5 ago","2026 10 01 10:00","yesterday 1 2026"). A run of digits is a date only as day month year ("01 10 2026") or after the name of a month ("Oct 1, 2026","Oct 1 26","march the 5 2026"). A number before a unit or an hour with am/pm is a count, and the run before it must be a date:"Oct 1 2026 3 pm"is one,"12 10 pm"is refused. The decimal comma of the seconds ("2026-10-01T10:00:00,500Z") and a comma that closes a date before its hour ("2026-10-01,10:00","Oct 1,3 pm") are a date’s. The details are intranger2_time_of_text().A string with a NUL inside (
"<a date>\u0000garbage") is refused: it is judged whole (tranger2_time_of_json_text()).Any other text is a date read by
approxidate_careful()and taken to the unit of its axis (an end bound takes its whole second).
A text that is no date, or a number refused, is refused with the WARNING
“match_cond refused: a time bound is no date” (why, topic_name,
bound, and the first 64 bytes of the text in value), and the open answers
NULL. Up to SDK 7.29.3 a text with a T that was no date was NOW, a text
without one was cut by strtoll() ("garbage" was no bound,
"2026-10-01 00:00:00 +0000" the time 2026, " 5" 5, "10.5" 10), and a
feed did not read a text bound at all.
The open writes the bounds it read into its match_cond, as integers in
the unit of ITS topic, and so the negative bounds it resolves for the key. A
match_cond is the open’s own: a caller that opens several topics or keys
with one condition hands each open a copy, never the same json increfed.
Only the library shares one, where it is the same topic and key:
tranger2_open_list() with a key hands its
history iterator and then its feed the same match_cond, so the feed gets
the bounds the iterator resolved for that key (a keyless list gives each
key’s iterator a copy, and its feed the negative bounds unresolved).
Up to 7.29.3 tr2search -r did, and every topic read a date in the unit of the
first one.
json_t *cond = json_pack("{s:s, s:I}",
"from_t", "2026-10-01T00:00:00+0000",
"load_record_callback", (json_int_t)(uintptr_t)load_cb
);
for(int i = 0; topics[i]; i++) {
json_t *list = tranger2_open_list(tranger, topics[i],
json_deep_copy(cond), /* each topic reads the date in its own unit */
NULL, NULL, FALSE, NULL);
/* ... */
}
JSON_DECREF(cond)json_t *it = tranger2_open_iterator(tranger, "raw_t", "r1",
json_pack("{s:s}", "from_t", "2026-10-01 00:00:00 +0000"), /* the date */
NULL, "day", "", data, NULL);
it = tranger2_open_iterator(tranger, "raw_t", "r1",
json_pack("{s:s}", "from_t", "garbage"),
NULL, "bad", "", data, NULL);
/* NULL; WARNING "match_cond refused: a time bound is no date", why "a text that is no date" */
it = tranger2_open_iterator(tranger, "raw_t", "r1",
json_pack("{s:s}", "from_t", "10.5"),
NULL, "bad", "", data, NULL);
/* NULL; the same WARNING, why "a number that is not the text of a JSON integer" */
it = tranger2_open_iterator(tranger, "raw_t", "r1",
json_pack("{s:s}", "from_t", ""),
NULL, "all", "", data, NULL);
/* every record: "" is no bound */A negative t / tm bound is relative to the key’s last record: its t,
and its tm -- the tm of the last row of the file that holds the key’s
highest t, not the highest tm of the key (which a replica and a reloaded
master do not know: they read the first and last rows of each file).
from_t = -N takes the records after last_t - N (that time excluded), and
to_t = -N takes the ones up to last_t - N. The bound is resolved once, when the
iterator opens, and written back into its match_cond. A list of several keys
gives each key’s iterator its own copy, so each key is bounded by its own last
record (before v7 the bound was relative to the TOPIC’s last record, of any
key). The realtime half of a list of several keys keeps the bound unresolved
-- there is no one last record for all of them -- so a negative from bounds
nothing there and a negative to takes no new record. On a key with no record
yet a negative from bounds nothing and a negative to takes no row. A db_history that starts with the last day of its key:
json_t *match_cond = json_pack("{s:I}",
"from_t", (json_int_t)-86400 // the last 24 h before the last record
);Up to 7.25.22 a negative bound reached the comparison raw, against an unsigned
__t__, and no record matched -- without a word. (Before v7 it already meant
“relative to the last record”, and that is the meaning restored.)
A negative from_rowid / to_rowid counts rows from the key’s end: -1 is
its last live row. It is resolved the same way, once, when the iterator opens,
and written back into its match_cond as the absolute g_rowid. So a filtered
iterator opened with from_rowid = -4 indexes the last 4 rows of the key at its
open, and pages those 4 rows when the key grows and when its old files are
purged later (tranger2_delete_key_files()). A
new open takes the last 4 rows of the key as it is then:
json_t *it = tranger2_open_iterator(tranger, "tracks", key,
json_pack("{s:I}", "from_rowid", (json_int_t)-4), // rows 6..9 of a key of 9 rows
NULL, "last4", "", NULL, NULL
);
/* 4 more rows appended, then the old files purged: its pages still give 6..9 */
json_t *page = tranger2_iterator_get_page(tranger, it, 1, 10, FALSE);In 7.27.5 and 7.27.6 the bound was resolved again when a purge took the segments again: with rows appended since the open, the pages came back empty with “indexed rowid without segment”.
A replica whose numbering of the key moves to the master’s (the master purged
rows it had not counted, a md2 file it flagged unreadable: the WARNING “The
numbering of the key moved to the master’s”) moves by the same delta the
index of its filtered iterators AND the absolute rowid bounds of their
match_cond, positive ones included: a pager opened with from_rowid=4, to_rowid=6 over rows C2 D1 D2 of the replica pages C2 D1 D2 after the
purge, numbered 6..8. In 7.27.5 and 7.27.6 a positive to_rowid stayed
behind the index, and the page held one row, with “indexed rowid without
segment”.
The two axes are not ordered the same way, and the scan knows it:
Rows are in
torder, except in a file that holds a late record (a__t__below the times the file already had). The master marks that file<file>.unordered, and inside a marked file the scan reads every row of the selected files instead of stopping at the first row past thetrange.Nothing orders
tm. It is written by the producer (a device that sends what it buffered writes it out of order), and the files are cut byt. Atmcondition leaves no file out and ends no scan: it skips rows, one by one. Correct, and it reads every md2 row of the key, so its cost grows with the files of the key: ~15 ms for a query of one minute on one key of 30 day files x 20000 rows (performance/c/perf_timeranger2, and the match condition is parsed once per scan, not per row, where it was ~0.15 s before). Every scan (an iterator’s load, its index, its pages, a follower’s new rows) reads the md2 rows in blocks of 1024, onepread()each, never past the rows its segment counts. Up to 7.25.22, it made anlseek()and aread()per 32-byte row (~0.44 s for that query). Thefr_tm/to_tmof a file inlist-keysare read from its first and last rows unless the file was read whole (a late__t__): approximate when itstmgoes back. From 7.25.5 to 7.25.22 the master marked a file whosetmwent back and the scan trusted the others (7.4 ms there, after a migrationmark-tm-order). The markers and the command are gone, and the ones a store has are ignored.The marker goes down before the row. The master writes the marker of a record whose
tgoes back BEFORE its md2 row, so a process that dies between the two leaves a marker with no row behind it (one whole read of the file at the next load), never a row with no marker. A marker that cannot be written is logged (“Cannot mark md2 file, a reload will misread its time range”). The master still reads that file whole, and writes the marker at the next append to the file (“md2 file marked, the marker missed earlier is written”). A file whose marker cannot exist --<file_id>.unorderedlonger thanNAME_MAX, or its path<topic>/keys/<key>/<marker>ofPATH_MAXbytes or more while the.jsonand.md2fit -- is logged once for the file, at its first late record (“Cannot mark md2 file, the marker name or path is too long: a load reads it whole (logged once)”), never tried again, and read whole at every load, with no log there (a replica loads it at every append). The master and the load ask the same length rule, so the file’s range is never taken from its first and last rows. Up to 7.29.3 a path too long logged an ERROR with a stack at EVERY late append, and a reload took the file as in order (a late row’s maximum was hidden from atquery). A name too long logged an ERROR at every load. For example, a topic directory of 3950 bytes and a key of 120:keys/<key>/2000-01-01.jsonis 4092 bytes, the marker 4097. Five late appends log one ERROR, and after a reload atquery around the file’s maximum still finds it. Neither is fsync’ed, like the append itself. A marker lost all the same (a power cut, a rollback to a binary that appends without markers) is not written again: that file’strange is read from its first and last rows until it gets a marker. (tranger2_mark_tm_order(), which was able to write it again, was removed with the tm markers after 7.25.22.)A file left out by
t(atrange around a file whose rows are out of it) is a hole in the rowids the scan walks: the scan steps over it, and afrom_rowid/to_rowidthat falls in it begins at the next row the scan can read. A hole is not an error. The scan logs an ERROR only when the segments overlap the row it just read, a broken invariant: “next segment begins before the row just read” (forward) or “previous segment ends after the row just read” (backward), withcur_rowidand the segment’s first or last row (7.25.4 logged “next rowids not consecutive” for every hole, and lost the rows after it).
Both hold in both directions, and for a paged iterator too. With the rows
t=100 E1, t=50000 E2, t=200 E3 (late) in one file:
json_t *data = json_array();
json_t *it = tranger2_open_iterator(
tranger, "readings", "0000000000000000001",
json_pack("{s:I, s:I, s:b}",
"from_t", (json_int_t)150,
"to_t", (json_int_t)250,
"backward", 0
),
NULL, "range", "", data, NULL
);
// data holds E3 (up to 7.25.3: nothing, the scan stopped at E2)
tranger2_close_iterator(tranger, it);
JSON_DECREF(data)And with three day files of one key, tm 100, 5000 and 150, the middle one is
out of to_tm = 300 and the scan steps over it:
json_t *data = json_array();
json_t *it = tranger2_open_iterator(
tranger, "readings", "K",
json_pack("{s:I}", "to_tm", (json_int_t)300),
NULL, "range", "", data, NULL
);
// data holds D1 D3 (7.25.4: D1, and "next rowids not consecutive")
tranger2_close_iterator(tranger, it);
JSON_DECREF(data)Every condition is honored per record, in both modes of the iterator:
LOADING (a
load_record_callbackor bothdata): each record is matched as the history is walked.PAGING (neither): a filtered iterator builds its row index when it opens, so
tranger2_iterator_size()andtranger2_iterator_get_page()count and return exactly the matching records, andget_page’sfrom_rowidis a position among THOSE rows. An unfiltered iterator builds no index — its open stays cheap regardless of the key size, and its positions are the global rowids.
tranger2_open_list()¶
tranger2_open_list() opens a list of records in memory, optionally enabling real-time updates via memory or disk.
json_t *tranger2_open_list(
json_t *tranger,
const char *topic_name,
json_t *match_cond, // owned
json_t *extra, // owned, will be added to the returned rt
const char *rt_id,
BOOL rt_by_disk,
const char *creator
);Parameters
| Key | Type | Description |
|---|---|---|
tranger | json_t * | The TimeRanger database instance. |
topic_name | const char * | The name of the topic to open. |
match_cond | json_t * | A JSON object specifying filtering conditions for records. Owned by the function. |
extra | json_t * | Additional user data to be added to the returned real-time object. Owned by the function. |
rt_id | const char * | The real-time list identifier. |
rt_by_disk | BOOL | If TRUE, enables real-time updates via disk. Otherwise, uses memory. |
creator | const char * | The identifier of the entity creating the list. |
Returns
Returns a JSON object representing the real-time list (rt_mem or rt_disk),
or, when real-time is disabled (to_rowid set), the extra given, tagged
"list_type": "no_rt". Close either with
tranger2_close_list().
NULL (logged, and match_cond and extra are consumed) on error: no topic, no
load_record_callback, a bad rkey, a real-time feed that cannot be opened,
no extra for a no_rt list, and when the history of THE key of a list of
one key (key set) was not loaded whole (the load_failed of
tranger2_open_iterator()).
A keyless list with a key that cannot be loaded logs the key (“Cannot load the whole history of a key of the list: the records read before the failure were handed, the list goes on with the next key”), loads every other key, opens its real-time feed, and says what it lacks in the handle it returns:
{
"list_type": "rt_mem",
"load_failed": true,
"load_failed_keys": ["B"]
}The records already handed to the callback stay handed: of a failed key, the ones read before the failure (forward: its oldest, backward: its newest). A caller that answers a question from the list must read the flag, or it takes “not in the list” for “not on disk”:
json_t *list = tranger2_open_list(tranger, "devices", match_cond, extra, "", FALSE, "");
if(list && json_is_true(json_object_get(list, "load_failed"))) {
json_t *keys = json_object_get(list, "load_failed_keys");
// these keys are on disk and not in what the callback saw
}What the callers in the SDK do with it:
| Caller | With load_failed |
|---|---|
treedb (every topic, __snaps__, __graphs__) | remembers the keys, refuses a create of such an id, refuses shoot/activate of a snap when __snaps__ has any, and its asset guards fail closed (see TreeDB). |
| treedb’s snapshot guard of the assets | a walk that did not load everything fails closed: the gc and the delete of an asset refuse. |
tr_queue / tr2q_mqtt (trq_load(), tr2q_load()) | the pending messages read are in memory, but those of a row that cannot be read are not, and are not delivered. The load returns -1 and does NOT move nor save the queue’s first_rowid, so the next load, once the store is repaired, finds them (up to 7.25.4 it saved the size of the topic, and they were skipped for ever). |
| msg2db | reloads the key newest first, up to the damage. The id is marked incomplete (msg2db_id_incomplete()) and an ERROR names it with served=N. See tr_msg2db. |
Until 7.25.4 such a list was handed over silently, with the key missing.
Notes
Loading all records can introduce delays in application startup. Use filtering conditions in match_cond to optimize performance.
The history is loaded with one-shot iterators of their own creator
(__tranger2_open_list__): an iterator the caller keeps open on a key does not
make the load “already exist”.
json_t *list = tranger2_open_list(
tranger, "devices",
json_pack("{s:I, s:I}",
"to_rowid", (json_int_t)1000000, // one-shot, no realtime
"load_record_callback", (json_int_t)(uintptr_t)load_cb
),
json_object(), // extra: IS the returned handle of a no_rt list
"", FALSE, "me"
);
if(!list) {
// refused (logged)
} else {
if(json_is_true(json_object_get(list, "load_failed"))) {
// some key's history could not be read: what load_cb got is not the
// whole topic, and "load_failed_keys" names those keys
}
tranger2_close_list(tranger, list);
}tranger2_open_rt_disk()¶
Opens a real-time disk-based iterator for monitoring changes in a topic. The function allows tracking new records appended to the topic by monitoring disk events.
json_t *tranger2_open_rt_disk(
json_t *tranger,
const char *topic_name,
const char *key, // if empty receives all keys, else only this key
json_t *match_cond, // owned
tranger2_load_record_callback_t load_record_callback, // called on append new record on disk
const char *rt_id, // rt disk id, REQUIRED
const char *creator,
json_t *extra // owned, user data, this json will be added to the return iterator
);Parameters
| Key | Type | Description |
|---|---|---|
tranger | json_t * | Pointer to the TimeRanger database instance. |
topic_name | const char * | Name of the topic to monitor. |
key | const char * | Specific key to monitor. If empty, all keys are monitored. |
match_cond | json_t * | JSON object containing conditions for filtering records. Owned by the function. |
load_record_callback | tranger2_load_record_callback_t | Callback function invoked when a new record is appended to the disk. |
rt_id | const char * | Unique identifier for the real-time disk iterator. Required. |
creator | const char * | Identifier of the entity creating the iterator. |
extra | json_t * | Additional user data to be associated with the iterator. Owned by the function. |
Returns
Returns a JSON object representing the real-time disk iterator. The caller does not own the returned object.
Notes
This function is used when monitoring real-time changes in a topic via disk events. The iterator remains active until explicitly closed using tranger2_close_rt_disk().
The rowid the callback receives is the key’s global g_rowid — the one that
never resets — not the row’s position inside the .md2 file it lives in (that
one is i_rowid, and it restarts at 1 with every new file). A consumer can
therefore dedupe or page by it across a file rotation.
Several feeds can be open on the same key (a per-key one and a whole-topic one,
say). Each is served exactly once per record: the feed keeps a watermark of the
last row it was given, per key and per file -- one mark per FILE, so a batch
that touches two files of a key (a late record and a current one, or a rotation
while the follower is busy) serves each feed both (before 7.25.0, the second feed lost both) -- and the master hard-links each new
.md2 into the directory of every feed that wants the key. A feed’s watermark
dies with its key, so a key re-created later does not inherit it.
The path of that link, disks/<rt_id>/<key>/<file>, is longer than the
master’s keys/<key>/<file>, so a key the master stores can be too long for a
feed. Its records do not reach that feed, and the master logs it ONCE for the
key and the feed, at its first record (“Path too long, the records of this
key do not reach this feed (logged once)”). An unreleased development version
logged an ERROR with a stack for every record and every feed. Up to 7.29.3, the
path was cut in silence and the link was made from the cut path. For example,
with a feed directory of 3962 bytes, a key of 200 bytes (no room for
<key>) or of 125 bytes (no room for /2000-01-01.md2 after it): five
records log one ERROR for each key.
tranger2_open_rt_mem()¶
Opens a real-time memory stream for a given topic in tranger. This function enables real-time message processing for the specified key and applies filtering conditions from match_cond. The callback tranger2_load_record_callback_t is invoked when new records are appended.
json_t *tranger2_open_rt_mem(
json_t *tranger,
const char *topic_name,
const char *key, // if empty receives all keys, else only this key
json_t *match_cond, // owned
tranger2_load_record_callback_t load_record_callback, // called on append new record on mem
const char *list_id, // rt list id, optional (internally will use the pointer of rt)
const char *creator,
json_t *extra // owned, user data, this json will be added to the return iterator
);Parameters
| Key | Type | Description |
|---|---|---|
tranger | json_t * | Pointer to the TimeRanger database instance. |
topic_name | const char * | Name of the topic for which real-time memory streaming is enabled. |
key | const char * | Specific key to monitor. If empty, all keys are monitored. |
match_cond | json_t * | JSON object containing filtering conditions. Owned by the function. |
load_record_callback | tranger2_load_record_callback_t | Callback function invoked when a new record is appended in memory. |
list_id | const char * | Optional identifier for the real-time list. If empty, the internal pointer is used. |
creator | const char * | Identifier of the entity creating the real-time memory stream. |
extra | json_t * | Additional user data stored in the returned iterator. Owned by the function. |
Returns
Returns a JSON object representing the real-time memory stream. The caller does not own the returned object.
Notes
This function is valid when the Yuno instance is the master writing real-time messages from tranger2_append_record().
tranger2_open_topic()¶
The tranger2_open_topic() function opens a topic in the TimeRanger database. If the topic is already open, it returns the existing topic JSON object.
json_t *tranger2_open_topic(
json_t *tranger,
const char *topic_name,
BOOL verbose
);Parameters
| Key | Type | Description |
|---|---|---|
tranger | json_t * | A JSON object representing the TimeRanger database. |
topic_name | const char * | The name of the topic to open. |
verbose | BOOL | If TRUE, additional debug information is printed. |
Returns
A JSON object representing the opened topic. The returned object is not owned by the caller and must not be modified or freed.
Notes
This function is idempotent. This means that calling it multiple times with the same topic_name will return the same JSON object without creating a new instance.
It returns NULL, never a critical, when the name is refused by the topic name rule, and when the directory exists but is not a topic (it has no topic_desc.json): “Not a topic: topic_desc.json not found”, logged only with verbose. A peer can send any name, and with on_critical_error=2 a critical is an exit(0) of the yuno.
It also returns NULL when the keys/ directory of the topic exists and cannot be listed (EMFILE, EACCES, ENOMEM, or a readdir() that fails): “Cannot list the keys of the topic” (with errno), then “Cannot open topic: its keys cannot be listed”. On a filesystem whose readdir() gives no d_type (XFS with ftype=0, NFS, FUSE, overlay) each key is asked with lstat(), as the entry type tells it: a symbolic link is not a key (up to 7.25.4 stat() followed it, and made a key of another key). An lstat() that fails with EACCES (keys/ can be read and not searched) takes the entry as a key, as the entry type does, and the listing of that key says the EACCES and flags it. One that fails with anything else but ENOENT (a key removed meanwhile) fails the listing the same way: “Cannot list the keys of the topic, lstat() FAILED”. Up to 7.25.4 that key was taken as “not a directory” and left out with no log, and the topic opened without it. The topic is not left open: opened with no keys, a key nobody read looked like a key that does not exist, and a treedb accepted a create of its id (so up to 7.25.4). The next open, or the next tranger2_topic(), tries again:
json_t *topic = tranger2_open_topic(tranger, "devices", TRUE);
if(!topic) {
// Error already logged: not a topic, a refused name, or keys/ cannot be listed.
// Nothing of the topic is in memory; try again later.
return -1;
}A keys/ that does not exist (ENOENT) is still a topic with no key, as before.
tranger2_print_md0_record()¶
Prints metadata of a record, including row ID, time, message time, and key, into a buffer.
void tranger2_print_md0_record(
char *bf,
int bfsize,
const char *key,
json_int_t rowid,
const md2_record_ex_t *md_record_ex,
BOOL print_local_time
);Parameters
| Key | Type | Description |
|---|---|---|
bf | char * | Buffer to store the formatted metadata string. |
bfsize | int | Size of the buffer to prevent overflow. |
key | const char * | Key associated with the record. |
md_record_ex | const md2_record_ex_t * | Pointer to the record metadata structure. |
print_local_time | BOOL | Flag indicating whether to print time in local format. |
Returns
This function does not return a value.
Notes
The function formats the metadata into the provided buffer. This makes sure of it does not exceed bfsize. It prints row ID, time, message time, and key.
tranger2_print_md1_record()¶
Prints metadata information of a record, including row ID, user flag, system flag, timestamps, and key, into a buffer.
void tranger2_print_md1_record(
char *bf,
int bfsize,
const char *key,
json_int_t rowid,
const md2_record_ex_t *md_record_ex,
BOOL print_local_time
);Parameters
| Key | Type | Description |
|---|---|---|
bf | char * | Buffer where the formatted metadata string will be stored. |
bfsize | int | Size of the buffer to make sure that safe writing. |
key | const char * | Key associated with the record. |
md_record_ex | const md2_record_ex_t * | Pointer to the record metadata structure containing the information to be printed. |
print_local_time | BOOL | Flag indicating whether to print timestamps in local time (if TRUE) or UTC (if FALSE). |
Returns
This function does not return a value.
Notes
The function formats and writes metadata details into the provided buffer. This makes sure that the output does not exceed bfsize bytes.
tranger2_print_md2_record()¶
Prints detailed metadata of a record, including row ID, offset, size, timestamp, and file path, into the provided buffer.
void tranger2_print_md2_record(
char *bf,
int bfsize,
json_t *tranger,
json_t *topic,
const char *key,
json_int_t rowid,
const md2_record_ex_t *md_record_ex,
BOOL print_local_time
);Parameters
| Key | Type | Description |
|---|---|---|
bf | char * | Buffer where the formatted metadata string will be stored. |
bfsize | int | Size of the buffer to make sure that safe writing. |
tranger | json_t * | Reference to the TimeRanger database instance. |
topic | json_t * | JSON object representing the topic associated with the record. |
key | const char * | Key identifying the record within the topic. |
md_record_ex | const md2_record_ex_t * | Pointer to the extended metadata structure of the record. |
print_local_time | BOOL | Flag indicating whether to print timestamps in local time. |
Returns
This function does not return a value.
Notes
The function formats metadata details into the provided buffer. This makes sure that the output remains within the specified buffer size.
tranger2_print_record_filename()¶
Formats and stores the filename of a record in a buffer, using metadata from tranger2_print_record_filename().
void tranger2_print_record_filename(
char *bf,
int bfsize,
json_t *tranger,
json_t *topic,
const md2_record_ex_t *md_record_ex,
BOOL print_local_time
);Parameters
| Key | Type | Description |
|---|---|---|
bf | char * | Pointer to the buffer where the formatted filename will be stored. |
bfsize | int | Size of the buffer in bytes. |
tranger | json_t * | Pointer to the TimeRanger database instance. |
topic | json_t * | Pointer to the topic associated with the record. |
md_record_ex | const md2_record_ex_t * | Pointer to the record metadata structure. |
print_local_time | BOOL | Flag indicating whether to format timestamps in local time. |
Returns
This function does not return a value.
Notes
The buffer bf must be large enough to store the formatted filename. The function retrieves metadata from tranger2_print_record_filename() to construct the filename.
tranger2_read_record_content()¶
Reads the content of a record from a given topic in the TimeRanger database. The function retrieves the record’s data based on its metadata.
json_t *tranger2_read_record_content(
json_t *tranger,
json_t *topic,
const char *key,
md2_record_ex_t *md_record_ex
);Parameters
| Key | Type | Description |
|---|---|---|
tranger | json_t * | Pointer to the TimeRanger database instance. |
topic | json_t * | Pointer to the topic from which the record will be read. |
key | const char * | The key identifying the record to be read. |
md_record_ex | md2_record_ex_t * | Pointer to the metadata structure of the record to be retrieved. |
Returns
Returns a json_t * object containing the record’s content. The caller owns the returned JSON object and must free it when no longer needed.
Notes
This function is useful when only metadata was loaded and the full record content needs to be retrieved.
tranger2_read_user_flag()¶
The tranger2_read_user_flag() function retrieves the user flag associated with a specific record in a given topic.
uint16_t tranger2_read_user_flag(
json_t *tranger,
const char *topic_name,
const char *key,
uint64_t __t__,
uint64_t rowid
);Parameters
| Key | Type | Description |
|---|---|---|
tranger | json_t * | Pointer to the TimeRanger database instance. |
topic_name | const char * | Name of the topic containing the record. |
rowid | uint64_t | Unique identifier of the record whose user flag is to be retrieved. |
Returns
Returns the uint16_t user flag associated with the specified record.
Notes
The function is used in writing mode to check the user flag of a record before modifying it.
tranger2_set_trace_level()¶
Sets the trace level of the tranger instance, controlling the verbosity of logging and debugging output.
void tranger2_set_trace_level(
json_t *tranger,
int trace_level
);Parameters
| Key | Type | Description |
|---|---|---|
tranger | json_t * | A pointer to the tranger instance whose trace level is to be modified. |
trace_level | int | The new trace level to be set, determining the verbosity of logging. |
Returns
This function does not return a value.
Notes
Higher trace levels typically enable more detailed logging, which can be useful for debugging.
tranger2_set_system_flag()¶
Sets or clears specific bits in the system flag of a record — the metadata
band reserved for the framework (for example the immutable-record bit), distinct from
the user flag set by tranger2_set_user_flag().
int tranger2_set_system_flag(
json_t *tranger,
const char *topic_name,
const char *key,
uint64_t __t__,
uint64_t rowid,
uint16_t mask,
BOOL set
);Parameters
| Key | Type | Description |
|---|---|---|
tranger | json_t * | Pointer to the TimeRanger database instance. |
topic_name | const char * | Name of the topic containing the record. |
key | const char * | Record key — with __t__ and rowid it identifies the record metadata. |
__t__ | uint64_t | Record time. |
rowid | uint64_t | Real rowid in the file (not the topic-global rowid). |
mask | uint16_t | Bitmask specifying which bits in the system flag to modify. |
set | BOOL | If TRUE, the bits in mask are set. If FALSE, they are cleared. |
Returns
Returns 0 on success, or a negative error code on failure.
Notes
Only the bits in mask are touched. The system flag is framework-owned. User
code normally goes through higher-level treedb helpers rather than setting system
bits directly.
Master only: on a replica, or on a master that lost its lock (see
tranger2_stop()), it writes nothing, logs “Only master
can write” and returns -1. Up to 7.25.4 it asked nothing: the write of the read-only md2 fd failed,
which is a CRITICAL, and with the default on_critical_error the process
exited(0).
tranger2_set_user_flag()¶
Sets or clears specific bits in the user flag of a record in a topic within the TimeRanger database.
int tranger2_set_user_flag(
json_t *tranger,
const char *topic_name,
const char *key,
uint64_t __t__,
uint64_t rowid,
uint16_t mask,
BOOL set
);Parameters
| Key | Type | Description |
|---|---|---|
tranger | json_t * | Pointer to the TimeRanger database instance. |
topic_name | const char * | Name of the topic containing the record. |
rowid | uint64_t | Unique identifier of the record whose user flag is modified. |
mask | uint32_t | Bitmask specifying which bits in the user flag must be modified. |
set | BOOL | If TRUE, the bits specified in mask are set. If FALSE, they are cleared. |
Returns
Returns 0 on success, or a negative error code on failure.
Notes
This function modifies only the bits specified in mask. This leaves other bits in the user flag unchanged.
// Mark row 3 of key "dev-1" (its __t__ is 1731601280) as pending, keep the other bits
if(tranger2_set_user_flag(tranger, "messages", "dev-1", 1731601280, 3, 0x0001, TRUE) < 0) {
// not the master, or the row is not there: logged
}Master only: on a replica, or on a master that lost its lock (see
tranger2_stop()), it writes nothing, logs “Only master
can write” and returns -1. Up to 7.25.4 it asked nothing: the write of the read-only md2 fd failed,
which is a CRITICAL, and with the default on_critical_error the process
exited(0).
tranger2_shutdown()¶
The tranger2_shutdown() function stops the TimeRanger database, releasing all allocated memory.
int tranger2_shutdown(
json_t *tranger
);Parameters
| Key | Type | Description |
|---|---|---|
tranger | json_t * | A pointer to the TimeRanger database instance to be stop. |
Returns
Returns 0 on success, or a negative value if an error occurs.
Notes
This function must be called when the database is no longer needed to free resources.
tranger2_startup()¶
Initializes the TimeRanger 2 database, setting up its internal structures and preparing it for use. The function requires a hgobj instance and a JSON configuration object that defines database parameters.
json_t *tranger2_startup(
hgobj gobj,
json_t *jn_tranger,
yev_loop_h yev_loop
);Parameters
| Key | Type | Description |
|---|---|---|
gobj | hgobj | A handle to the GObj framework instance managing the database. |
jn_tranger | json_t * | A JSON object containing database configuration parameters. See tranger2_json_desc for details. |
yev_loop | yev_loop_h | An event loop handle for asynchronous operations. |
Returns
A JSON object representing the initialized TimeRanger 2 database instance, or NULL on failure (logged).
Notes
The returned JSON object must be properly managed and eventually passed to tranger2_stop() or tranger2_shutdown() to release resources.
A path or a path/database that does not fit in PATH_MAX is refused: NULL, and the ERROR of build_path(). Up to 7.29.3 the store was made in the directory that fitted, the one above the store asked for.
A master that finds __timeranger2__.json EMPTY makes it again, as if it were not there: the write of a first open did not end (the disk was full, or the process died between the create and the write). It says it with a WARNING, “timeranger2.json is empty (its first write did not end): made again”, then the INFO “Creating timeranger2.json”. A replica ("master": false) still refuses the empty file (“Cannot load json file, bad json”): it cannot write. Up to 7.30.18 a master refused it too, at every try, and the store never opened again, even when the disk had space again. To see it on a node:
ls -l /yuneta/store/myapp/events/__timeranger2__.json
# -rw-rw---- 1 yuneta yuneta 0 ... <- empty: the next open of a master makes it again
# -r--r----- 1 yuneta yuneta 0 ... <- empty in mode 0440: made again tooThe file is saved and then set to mode 0440. On a filesystem with delayed allocation (ext4), a power cut can keep the mode and lose the data, so an empty file can be 0440. The master makes it writable again (mode rpermission) before it writes it. Up to 7.30.19 that write failed with “Cannot create json file” (EACCES) for any user but root, and the yuno exited.
json_t *tranger = tranger2_startup(gobj, json_pack("{s:s, s:s, s:b}",
"path", "/yuneta/store/myapp",
"database", "events",
"master", 1
), yev_loop);
if(!tranger) {
return -1; // Error already logged
}tranger2_stop()¶
The tranger2_stop() function closes the TimeRanger database. This makes sure that all topics and file descriptors are properly closed.
int tranger2_stop(
json_t *tranger
);Parameters
| Key | Type | Description |
|---|---|---|
tranger | json_t * | A pointer to the TimeRanger database instance to be closed. |
Returns
Returns 0 on success, or a negative value if an error occurs.
Notes
tranger2_shutdown() calls it when nobody did. Call it
yourself when the event loop needs a turn between the two, to finish the
watchers the topics close.
What the stop does:
It closes every topic, and the fds of
fd_opened_files: the__timeranger2__.jsonlock of a master is one of them, so the stop gives the single-master lock back. Each closed fd is marked-1: a second stop, or the shutdown, never closes that number again (by then it can belong to somebody else).A tranger stays usable after the stop. The first call that opens a topic again, or that WRITES (
tranger2_create_topic(),_delete_topic(),_backup_topic(),_write_topic_var(),_write_topic_cols(),_append_record(),_delete_key(),_delete_instance(),_write_user_flag(),_set_user_flag(),_set_system_flag()), revives it: the shutdown then closes what the revival opened, and a master takes its lock again BEFORE anything is written. If another process took the store in the meantime (flock()answersEWOULDBLOCK), it is the startup’s single-master conflict: the revival logs a CRITICAL aton_critical_error, “Master lock NOT retaken after a stop: another process holds it, go on as not master”. With a yuno’s default ("on_critical_error": 2, exit(0)) the process exits there and stays down, as a second instance does at the startup. Any OTHER failure to take the lock says nothing of another master and is an ERROR, not a critical, and the process goes on: “Master lock NOT retaken after a stop: cannot open the lock file, go on as not master” (EMFILE,ENOENT...) or “... flock() FAILED, go on as not master” (ENOLCK,EINTR...). A tranger that did not take its lock back (and did not exit) goes on as a replica: it reads, every write is refused, and its json says"master": false, "master_lost": true. It never takes the lock again, not even once it is free: its memory did not follow what the other master wrote. Stop it and start it again to make it the master. Readmasteragain after a restart: it is what the tranger holds now. A lookup of a topic that is already open revives nothing.A TRANSIENT failure demotes for good too. An
EMFILE(too many open files for a moment), anEINTRor anENOLCKat the revive is logged as an ERROR, not a critical: the process does NOT exit, it goes on as a replica, and it stays one after the cause is gone. Every write it is asked for is refused from then on (each with its own error), and nothing else says so. Nothing restarts it: the operator does. Watch for it:The log line “Master lock NOT retaken after a stop” (any of its three endings), at ERROR or CRITICAL.
The
masterattribute of theC_TRANGERservice, which answers what the tranger IS (false once demoted). Read it withview-attrsWITHOUTattribute=: that form goes through the gclass’mt_reading, but the one-attribute form answers the stored configuration.ycommand -c 'command-yuno id=<id> service=__yuno__ command=view-attrs gobj=<tranger service>' # "master": false on a yuno configured as the master = demoted: restart itin C,
master_losttrue in the tranger’s json.
Restart the yuno (
kill-yuno+run-yuno) once the cause is found: the new process takes the lock at its startup.
This is the life of the tranger of a C_TRANGER that is stopped and started
again: the tranger is built in mt_create, stopped in mt_stop, and shut down
only in mt_destroy.
json_t *tranger = tranger2_startup(0, jn_tranger, yev_loop); // master, lock taken
tranger2_open_topic(tranger, "devices", TRUE);
tranger2_stop(tranger); // topics closed, lock given back
// (C_TRANGER mt_stop)
tranger2_topic(tranger, "devices"); // opens it again: revived, lock retaken
// (C_TRANGER started again)
tranger2_shutdown(tranger); // closes what the revival openedAnd when another process took the store while it was stopped:
tranger2_stop(a); // a gives the lock back
json_t *b = tranger2_startup(0, jn_tranger_b, 0); // b takes the store
tranger2_create_topic(a, "devices", "id", "tm", 0, 0, 0, 0);
// a cannot take the lock: nothing written, "devices" opened read-only
kw_get_bool(0, a, "master", 0, 0); // FALSE
kw_get_bool(0, a, "master_lost", 0, 0); // TRUEtranger2_str2system_flag()¶
Converts a formatted string containing system flag representations into a system_flag2_t integer. The input string can use delimiters such as ‘|’, ’ ', or ‘,’.
system_flag2_t tranger2_str2system_flag(
const char *system_flag
);Parameters
| Key | Type | Description |
|---|---|---|
system_flag | const char * | A string containing system flag values separated by ‘|’, ’ ', or ‘,’. |
Returns
Returns the corresponding system_flag2_t integer representing the parsed system flags.
Notes
This function is useful for converting human-readable flag representations into their corresponding bitmask values.
tranger2_topic()¶
Retrieve a topic by its name from the TimeRanger database. If the topic is not already opened, tranger2_open_topic() is called to open it.
json_t *tranger2_topic(
json_t *tranger,
const char *topic_name
);Parameters
| Key | Type | Description |
|---|---|---|
tranger | json_t * | Pointer to the TimeRanger database instance. |
topic_name | const char * | Name of the topic to retrieve. |
Returns
A JSON object representing the topic. The returned object is not owned by the caller and must not be modified or freed.
Notes
If the topic exists on disk but has not been opened yet, tranger2_open_topic() is automatically invoked.
tranger2_topic_desc()¶
tranger2_topic_desc() retrieves the description of a specified topic from the TimeRanger database.
json_t *tranger2_topic_desc(
json_t *tranger,
const char *topic_name
);Parameters
| Key | Type | Description |
|---|---|---|
tranger | json_t * | A reference to the TimeRanger database instance. |
topic_name | const char * | The name of the topic whose description is to be retrieved. |
Returns
A JSON object containing the topic description. The caller must decrement the reference count of the returned JSON object.
The description is a projection of the topic, not the topic itself:
| Key | Description |
|---|---|
topic_name | The name of the topic. |
pkey | The column that is the primary key. |
pkey2s | The columns that are secondary keys, if the topic declares any. |
tkey | The column that carries the time, if the topic declares one. |
system_flag | The system flag the topic was created with. |
topic_version | The version of the topic schema. |
system_topic | true when the topic cannot be deleted. |
main_topic | true on the topic the tree of a treedb hangs from, as the schema marks it (a topic hooked to itself, one per treedb, stamped by treedb_open_db()). |
cols | The column descriptors, as a list. |
A key the topic does not carry is omitted rather than returned empty, so a
topic with no secondary key has no pkey2s at all.
pkey2s matters to any reader that must NAME a record and not only identify
it. A topic whose id column is flagged rowid, uuid or qualified keys its
records by a value that is not the name: a rowid and a UUID mean nothing to a
person, and a qualified id (<parent id>.<name>) names every ancestor together
with the record. treedb_system_schema keys topics and cols with qualified
ids. In all three the name lives in the secondary key. A viewer given only
pkey can do nothing but print the address.
Notes
The returned JSON object must be properly decremented using json_decref() to avoid memory leaks.
tranger2_topic_key_size()¶
Retrieves the number of records associated with a specific key in a given topic
within the TimeRanger database. If key is empty, the whole-topic size is
returned instead.
uint64_t tranger2_topic_key_size(
json_t *tranger,
const char *topic_name,
const char *key
);Parameters
| Key | Type | Description |
|---|---|---|
tranger | json_t * | Pointer to the TimeRanger database instance. |
topic_name | const char * | Name of the topic containing the key. |
key | const char * | The key whose record count is to be retrieved. If empty, the whole-topic size is returned (sum over every key). |
Returns
Returns the number of records associated with the specified key, or the
whole-topic total when key is empty.
Notes
Counts are from the in-memory cache. A missing topic returns 0. When key is
empty the call delegates to tranger2_topic_size().
tranger2_topic_key_range()¶
Returns the time span of one key of a topic, on both axes, read from the in-memory cache totals (maintained on load and on every append) — so a client can bound a time picker to what the key holds without reading a single record.
json_t *tranger2_topic_key_range( // return is yours
json_t *tranger,
const char *topic_name,
const char *key
);Parameters
| Key | Type | Description |
|---|---|---|
tranger | json_t * | Pointer to the TimeRanger database instance. |
topic_name | const char * | Name of the topic containing the key. |
key | const char * | The key whose span is to be retrieved. |
Returns
A JSON object, yours to decref:
{"fr_t": t, "to_t": t, "fr_tm": tm, "to_tm": tm, "rows": n}rows is the LIVE count, the rows that can be read. A key whose old files were
deleted (tranger2_delete_key_files()) also has
"purged": n, its base: the rows of the files deleted. Its live g_rowids are
purged+1 .. purged+rows, so its LAST g_rowid is purged + rows, not
rows. A key never purged has no purged. On a replica, "renumbered": n
says how far the g_rowids of the key moved when it took a base of the master
that counted rows it had not (a file it had flagged unreadable): whoever keeps
positions of the key outside its iterators moves them by it.
NULL if the topic (logged) or the key (silent) is unknown.
Example
/*
* The last g_rowid of a key, to page it by g_rowid: a key of 1823411 rows
* purged and 4000 live answers {"rows": 4000, "purged": 1823411, ...}
*/
json_t *range = tranger2_topic_key_range(tranger, "raw_events", "dev1");
if(range) {
json_int_t last_rowid = kw_get_int(gobj, range, "purged", 0, 0) +
kw_get_int(gobj, range, "rows", 0, 0); // 1827411
JSON_DECREF(range)
}Notes
t (persistence) and tm (message time) are independent, so the two spans
differ whenever data was backfilled. Both are in the topic’s own unit: seconds,
or milliseconds when the topic sets sf_t_ms / sf_tm_ms (read system_flag
from the topic desc).
tranger2_name_is_identifier()¶
TRUE if name can name a topic, a treedb or a msg2db. It is the name rule
of everything that is a DIRECTORY, kept in one place (since SDK 7.26.9). Such a
name is a directory (<database>/<topic>/), a key of the in-memory json and a
segment of the kw paths built from it, so it must not be empty, "." or
"..", nor hold / or the path delimiter `. A leading . is allowed.
Silent: the caller says what the name was for. A column, a hook or an fkey is
only a key, and follows the looser
treedb_col_name_is_identifier().
BOOL tranger2_name_is_identifier(
const char *name
);Parameters
| Key | Type | Description |
|---|---|---|
name | const char * | The name to check. |
Returns
TRUE if the name is usable, FALSE otherwise.
Example
tranger2_name_is_identifier("users"); // TRUE
tranger2_name_is_identifier(".hidden"); // TRUE: a leading '.' is allowed
tranger2_name_is_identifier("users`id"); // FALSE: the path delimiter
tranger2_name_is_identifier("kids/all"); // FALSE: a path separator
tranger2_name_is_identifier(".."); // FALSEtranger2_reserved_name_fault()¶
Why name cannot name a NEW topic from outside (a command), or
NULL. Beside its topics, the directory of a store holds what the libraries
keep there. A topic with such a name is a directory over that file or
folder, now or later, when the library writes it (it is possible that the store does not exist
yet). So these names are refused:
| Name | Why | What the store keeps there |
|---|---|---|
a leading . | “a name starting with ‘.’ is a hidden file of the store” | .blobs (the files of a treedb), .<treedb>.treedb_schema.json.tmp (apply-schema) |
a leading __ | “a name starting with ‘__’ is of the system” | __timeranger2__.json, the system topics of a treedb (__snaps__, __graphs__, __assets__, __icons__) |
a .json suffix | “a name ending in ‘.json’ is a file of the store” | __timeranger2__.json, <treedb>.treedb_schema.json, <msg2db>.msg2db_schema.json |
a .bak suffix | “a name ending in ‘.bak’ is the backup of a topic” | the backup of tranger2_backup_topic() |
The name rule (tranger2_name_is_identifier())
is not asked here: ask it too. The function logs nothing. The caller says
what the name was for. A topic that a C caller creates is not asked this:
the system topics of a treedb, and the queues of the MQTT broker, named
<client_id>-IN / <client_id>-OUT after a client id the client chose (a
client .foo has the queues .foo-IN and .foo-OUT), are made by C code. C_TRANGER and C_TREEDB ask it of the new topics of a
peer (since SDK 7.27.0).
const char *tranger2_reserved_name_fault(
const char *name
);Parameters
| Key | Type | Description |
|---|---|---|
name | const char * | The name of the new topic. An empty or NULL name answers NULL: the name rule refuses it. |
Returns
A static string with the cause, or NULL when a topic can take the name.
Example
tranger2_reserved_name_fault("users"); // NULL
tranger2_reserved_name_fault(".blobs"); // "a name starting with '.' is a hidden file of the store"
tranger2_reserved_name_fault("__snaps__"); // "a name starting with '__' is of the system"
tranger2_reserved_name_fault("x.json"); // "a name ending in '.json' is a file of the store"
tranger2_reserved_name_fault("users.bak"); // "a name ending in '.bak' is the backup of a topic"
tranger2_reserved_name_fault("a/b"); // NULL: tranger2_name_is_identifier() refuses it
/* a new topic that a peer names */
const char *why = tranger2_reserved_name_fault(topic_name);
if(why) {
gobj_log_warning(gobj, 0,
"function", "%s", __FUNCTION__,
"msgset", "%s", MSGSET_PARAMETER,
"msg", "%s", "topic name of the store, refused",
"why", "%s", why,
"topic_name", "%s", topic_name,
NULL
);
return -1;
}tranger2_topic_var_fault()¶
Why jn_var cannot be the topic_var.json of a topic from
outside (a command), or NULL.
tranger2_create_topic() merges jn_var into the
topic, and a new topic_version in it writes the topic’s topic_cols.json
and topic_var.json again. A NULL or a json null is no var. Anything
else must be a dict that:
has none of the fields that the tranger keeps itself:
topic_name,pkey,tkey,system_flag,cols,directory,wr_fd_files,rd_fd_files,rd_fd_life,lists,disks,iterators,iterators_by_id,cache,delete_seq,delete_seq_max,deletes_applied,delete_ref_keys,long_key_refs,long_key_hashes,fs_event_master,filename_mask,xpermission,rpermission. An open does not read them from the var.has no
system_topic: it marks a topic that nothing deletes, a topic of the system.has a
topic_versionthat is an integer from 0 toTRANGER2_VERSION_MAX, when it has one. A string there was an ERROR with a stack at the next change of the topic’s version.has a
last_rowid_id(the counter of a rowid key) that is an integer, 0 or more, when it has one.
The function logs nothing. The caller says it. C_TRANGER’s create-topic
asks it of a peer’s jn_var (since SDK 7.27.0).
const char *tranger2_topic_var_fault(
json_t *jn_var
);Parameters
| Key | Type | Description |
|---|---|---|
jn_var | json_t * | The var of the topic, not owned. NULL or a json null: no var. |
Returns
A static string with the cause, or NULL when the var can be used.
Example
tranger2_topic_var_fault(NULL); // NULL: no var
json_t *v = json_pack("{s:i}", "topic_version", 2);
tranger2_topic_var_fault(v); // NULL
json_decref(v);
v = json_pack("{s:s}", "topic_version", "2");
tranger2_topic_var_fault(v); // "the topic_version of jn_var is not an integer from 0 to 2147483647"
json_decref(v);
v = json_pack("{s:b}", "system_topic", 1);
tranger2_topic_var_fault(v); // "jn_var marks a system topic, which nothing deletes"
json_decref(v);
v = json_pack("{s:s}", "directory", "/tmp");
tranger2_topic_var_fault(v); // "jn_var has a field the tranger keeps itself"
json_decref(v);
v = json_pack("{s:i}", "last_rowid_id", -1);
tranger2_topic_var_fault(v); // "the last_rowid_id of jn_var is not an integer, 0 or more"
json_decref(v);
v = json_pack("[i]", 2);
tranger2_topic_var_fault(v); // "jn_var is not a json object"
json_decref(v);From a peer, through C_TRANGER:
command-yuno id=<id> service=tranger command=create-topic topic_name=pp jn_var='{"system_topic":true}'
# -> -1 <role^name>: cannot create topic 'pp': jn_var marks a system topic, which nothing deletestranger2_topic_is_open()¶
TRUE if the topic is currently open in this tranger. Silent — a closed topic is
a legitimate answer, not an error.
BOOL tranger2_topic_is_open(
json_t *tranger,
const char *topic_name
);Parameters
| Key | Type | Description |
|---|---|---|
tranger | json_t * | Pointer to the TimeRanger database instance. |
topic_name | const char * | Name of the topic to check. |
Returns
TRUE if the topic is open, FALSE otherwise.
Notes
tranger2_topic_name()¶
Retrieves the topic name from the given json_t * topic object.
const char *tranger2_topic_name(
json_t *topic
);Parameters
| Key | Type | Description |
|---|---|---|
topic | json_t * | A JSON object representing the topic. |
Returns
A pointer to a string containing the topic name. The returned string must not be modified or freed by the caller.
Notes
If the topic parameter is NULL or invalid, the behavior is undefined.
tranger2_topic_size()¶
tranger2_topic_size() retrieves the total number of records present in a specified topic within the TimeRanger database.
uint64_t tranger2_topic_size(
json_t *tranger,
const char *topic_name
);Parameters
| Key | Type | Description |
|---|---|---|
tranger | json_t * | Pointer to the TimeRanger database instance. |
topic_name | const char * | Name of the topic whose record count is to be retrieved. |
Returns
Returns the total number of records in the specified topic as a uint64_t value.
Notes
If the topic does not exist, the function can return 0.
tranger2_write_topic_cols()¶
The tranger2_write_topic_cols() function updates the column definitions of a specified topic in the TimeRanger database.
int tranger2_write_topic_cols(
json_t *tranger,
const char *topic_name,
json_t *jn_cols // owned
);Parameters
| Key | Type | Description |
|---|---|---|
tranger | json_t * | A pointer to the TimeRanger database instance. |
topic_name | const char * | The name of the topic whose column definitions are updated. |
jn_cols | json_t * | A JSON object containing the new column definitions. The ownership of this object is transferred to tranger2_write_topic_cols(). |
Returns
Returns 0 on success, or a negative error code on failure.
Notes
Master-only. Replaces the topic’s columns wholesale — both the in-memory
topic["cols"] and the on-disk topic_cols.json (not a merge). A columns change
must bump the topic’s topic_version (and schema_version for structural
changes), otherwise the persisted topic_cols.json masks the new schema on the
next reload.
The file is replaced, never written in place: topic_cols.json.new is removed
if a process left it, created O_EXCL|O_NOFOLLOW with the tranger’s
rpermission, written, fsync’ed, renamed over the old file, and the directory
fsync’ed: it survives a power cut too (it is written when a topic is created,
re-versioned or re-ordered, never per record, and up to 7.25.4 it was not
fsync’ed: a topic_version change was able to leave the new
version on disk over cols lost with the power). The memory takes the new
columns only when the file did. Otherwise, the call logs (“Cannot replace
topic_cols.json, ...”) and returns -1 with the old file and the old columns in
place. Up to 7.25.4 it wrote in place,
put the columns in memory before writing, and returned 0 whatever happened.
tranger2_write_topic_var() replaces topic_var.json the same way.
json_t *cols = json_pack("{s:s, s:s, s:s}", "id", "", "content", "", "extra", "");
if(tranger2_write_topic_cols(tranger, "readings", cols) < 0) { // cols consumed either way
// not written: topic_cols.json and topic["cols"] are the old ones
}One change does not wait for the bump. When the incoming schema holds the same
columns saying the same things and only their order differs,
tranger2_create_topic() rewrites the file: the
freeze is there so that a change to what a column declares cannot arrive
unannounced, and an order announces nothing new.
tranger2_write_topic_var()¶
The tranger2_write_topic_var() function updates the variable metadata of a specified topic in the TimeRanger database.
int tranger2_write_topic_var(
json_t *tranger,
const char *topic_name,
json_t *jn_topic_var
);Parameters
| Key | Type | Description |
|---|---|---|
tranger | json_t * | A pointer to the TimeRanger database instance. |
topic_name | const char * | The name of the topic whose variable metadata is to be updated. |
jn_topic_var | json_t * | A JSON object containing the new variable metadata for the topic. The ownership of this object is transferred to tranger2_write_topic_var(). |
Returns
Returns 0 on success, or a negative value on failure.
Notes
Master-only. Merges jn_topic_var into the existing topic_var.json (update,
not replace) and persists it. The in-memory topic is updated too, except the
immutable descriptor fields, and only when the file was written.
The file is never rewritten in place: the merge is written to
topic_var.json.new and renamed over topic_var.json, so a process that dies
half way leaves the old file or the new one, never an empty one. treedb calls
this on every create of a rowid-key node (the last_rowid_id counter lives
here), so it does not fsync(): it is safe against the death of the process,
not against a power cut. A topic_version change
(tranger2_create_topic()) does fsync.
A write that fails leaves the old topic_var.json as it was and answers -1.
A write cut part way (the file size limit, a full disk) is logged as “Cannot
replace topic_var.json, short write: the file size limit or the disk is full”,
without an errno: a short write does not set one.
tranger2_write_topic_var(tranger, "items", json_pack("{s:I}", "last_rowid_id", (json_int_t)42));
// topic_var.json: {"topic_version": 3, "last_rowid_id": 42} (a new inode)tranger2_write_user_flag()¶
The tranger2_write_user_flag() function updates the user flag of a specific record identified by rowid in the given topic_name within the TimeRanger database.
int tranger2_write_user_flag(
json_t *tranger,
const char *topic_name,
const char *key,
uint64_t __t__,
uint64_t rowid,
uint16_t user_flag
);Parameters
| Key | Type | Description |
|---|---|---|
tranger | json_t * | A pointer to the TimeRanger database instance. |
topic_name | const char * | The name of the topic containing the record. |
rowid | uint64_t | The unique row identifier of the record to update. |
user_flag | uint32_t | The new user flag value to be assigned to the record. |
Returns
Returns 0 on success, or a negative value on failure.
Notes
This function modifies the user flag of an existing record but does not alter other record attributes. Use tranger2_set_user_flag() if you need to update the flag using a mask.
Master only: on a replica, or on a master that lost its lock (see
tranger2_stop()), it writes nothing, logs “Only master
can write” and returns -1. Up to 7.25.4 it asked nothing: the write of the read-only md2 fd failed,
which is a CRITICAL, and with the default on_critical_error the process
exited(0).
tranger2_list_topic_names()¶
tranger2_list_topic_names() returns a JSON array of topic names by scanning the tranger database directory on disk. Unlike tranger2_list_topics(), which reads from the in-memory topic registry, this function reads subdirectory names from the filesystem.
json_t *tranger2_list_topic_names(
json_t *tranger
);Parameters
| Key | Type | Description |
|---|---|---|
tranger | json_t * | The tranger database handle (JSON object). |
Returns
A new JSON array of strings, each being a topic name found as a subdirectory in the tranger database directory. The caller owns the returned array and must call json_decref() on it. Hidden entries (names starting with .) are excluded.
NULL when the directory cannot be listed (opendir() or readdir() fails): logged, “Cannot list the topics of the store” with errno, and the cause is left in gobj_log_last_message(). A store that cannot be listed is not a store with no topic: up to 7.25.4 it answered [] with no log, and list-queues and clean-queues of the MQTT broker answered an empty list with 0. They answer -1 now.
Notes
This function operates on disk, not in memory. It can return topic names that are not currently open, or miss topics that exist only in memory. Use tranger2_list_topics() to get the in-memory list instead.
Example
json_t *names = tranger2_list_topic_names(tranger);
if(!names) {
// Error already logged: the store cannot be listed, NOT "no topic"
return -1;
}
size_t idx; json_t *jn_name;
json_array_foreach(names, idx, jn_name) {
printf("%s\n", json_string_value(jn_name));
}
JSON_DECREF(names)tranger2_topic_path()¶
tranger2_topic_path() writes the filesystem path of a topic into the provided buffer. The path is constructed by appending the topic name to the tranger database directory.
int tranger2_topic_path(
char *bf,
size_t bfsize,
json_t *tranger,
const char *topic_name
);Parameters
| Key | Type | Description |
|---|---|---|
bf | char * | Output buffer where the topic path will be written. |
bfsize | size_t | Size of the output buffer in bytes. |
tranger | json_t * | The tranger database handle (JSON object). |
topic_name | const char * | Name of the topic. |
Returns
Returns 0 on success.
Notes
The resulting path has the format <directory>/<topic_name>, where <directory> is the tranger database directory stored in the tranger handle.