Message routing, multiplexing, and persistent queuing.
Source: kernel/c/root-linux/src/c_channel.c, c_iogate.c,
c_qiogate.c, c_mqiogate.c
C_CHANNEL¶
Base communication channel — wraps a transport + protocol stack into a single open/closed abstraction with message-rate statistics.
| Property | Value |
|---|---|
| States | ST_STOPPED, ST_CLOSED, ST_OPENED |
| Input events | EV_ON_MESSAGE, EV_SEND_MESSAGE, EV_ON_OPEN, EV_ON_CLOSE, EV_DROP |
| Output events | EV_ON_MESSAGE, EV_ON_OPEN, EV_ON_CLOSE |
Key attributes¶
| Attribute | Type | Description |
|---|---|---|
opened | bool | Channel state (read-only). |
subscriber | pointer | Gobj receiving output events. |
txMsgsec / rxMsgsec | integer | Current message rate per second (stats). |
maxtxMsgsec / maxrxMsgsec | integer | Peak message rates (stats). |
Message rates¶
The rates of C_CHANNEL and C_IOGATE are computed when their stats are
read: the messages since the previous computation, divided by the time
elapsed in milliseconds. A read less than a second after the previous one
answers the last rates and does not move the window. So the rate is exact
over any interval of a second or more, and two readers share one window
(each sees the rate since the other’s read). Up to 7.25.21 the time was taken
in whole seconds (a read after 1.9 s divided by 1: +90 %), and every read
moved the window, so the messages of a read under a second went into no
rate.
What the design does not do, by choice (there is no timer: a gate nobody reads costs nothing):
The window is as wide as the reader’s interval: a reader every 60 s gets the average of those 60 s, and a burst of one second inside them is diluted.
The maxima (
maxtxMsgsec/maxrxMsgsec) move only when a window closes, so they are maxima of the readers’ windows, not of any second.The messages before the first read go into no rate: the first read opens the window.
A rate per second whatever the readers do needs a tick of its own, as the
control center keeps (its timeout attr).
A C_CHANNEL read by its parent (a gate) answers its BARE counters, which
the gate merges into its own stats.
C_IOGATE¶
I/O gate — routes messages between channels and upper-layer destinations. Supports one-at-a-time (rotated) or broadcast delivery.
| Property | Value |
|---|---|
| States | ST_STOPPED, ST_IDLE |
| Input events | EV_ON_MESSAGE, EV_SEND_MESSAGE, EV_ON_IEV_MESSAGE, EV_SEND_IEV, EV_ON_OPEN, EV_ON_CLOSE, EV_DROP, EV_STOPPED |
| Output events | EV_ON_MESSAGE, EV_ON_OPEN, EV_ON_CLOSE |
Key attributes¶
| Attribute | Type | Description |
|---|---|---|
persistent_channels | bool | Keep channels alive across reconnections. |
send_type | integer | 0 = rotated one-at-a-time, 1 = broadcast to all. A kw can choose for itself with __send_type__. |
Send to all¶
With send_type 1 every open channel gets the message. A message in a
gbuffer is read out by the channel that sends it (C_PROT_TCP4H appends it
to its frame), so every open channel but the last gets a COPY of what is
left to read -- with its secret flag, label and address -- and the last one
takes the original gbuffer. So a sender that kept a reference to that
gbuffer finds it read out after a send to all, as after a send to one. A
kw without a gbuffer is shared (kw_incref()). Up to 7.25.20 the gbuffer was
shared: the first channel sent the message, and the others an empty frame,
which a C_PROT_TCP4H peer drops the connection on (“frame_length cannot be
0”).
gbuffer_t *gbuf = gbuffer_create(len, len);
gbuffer_append(gbuf, data, len);
gobj_send_event(gate, EV_SEND_MESSAGE,
json_pack("{s:I, s:i}",
"gbuffer", (json_int_t)(uintptr_t)gbuf, // the kw owns it
"__send_type__", 1 // to every open channel
),
gobj
);tests/c/c_tcp (test8).
__send_type__ is a key of the yuno, like __temp__: the kw of a send is
built by the yuno’s own code, never taken from a peer (see
iev_create_from_gbuffer()), so the frontier does
not delete it.
The channel of a message¶
What the gate publishes (EV_ON_MESSAGE, EV_ON_OPEN, EV_ON_CLOSE, and
the event of an EV_ON_IEV_MESSAGE) carries the channel it came in by, in
__temp__: channel, its name, and channel_gobj, its pointer. A send
(EV_SEND_MESSAGE, EV_SEND_IEV) whose kw carries them goes to that
channel, so an answer goes back by the channel of its request:
PRIVATE int ac_on_message(hgobj gobj, gobj_event_t event, json_t *kw, hgobj src)
{
// kw: {"gbuffer": ..., "__temp__": {"channel": "input-1", "channel_gobj": ...}}
json_t *kw_answer = json_pack("{s:I, s:O}",
"gbuffer", (json_int_t)(uintptr_t)gbuf_answer,
"__temp__", json_object_get(kw, "__temp__")
);
gobj_send_event(src, EV_SEND_MESSAGE, kw_answer, gobj); // to input-1
KW_DECREF(kw)
return 0;
}__temp__ is local to the yuno: a peer’s is dropped when its frame is
decoded, and it is left out of every frame sent (7.30.27+, see
iev_create_from_gbuffer()). So a __temp__
that is not a dict was put by code of this yuno: the gate replaces it, with
one ERROR, “temp of a message is not a dict, replaced” (7.30.27+. Up to
7.30.26, it was two ERRORs and the value stayed, and the subscriber read it as
the channel).
The event of an EV_ON_IEV_MESSAGE is one a peer sent, and the gate
publishes it as its own. So a peer can never name an event the transport
publishes itself (EV_ON_OPEN, EV_ON_CLOSE, EV_ON_MESSAGE, ...), nor
an event a treedb publishes (EV_TREEDB_NODE_CREATED, _UPDATED,
_DELETED, _LINKED, _UNLINKED: the gate’s subscriber is often subscribed
to its own treedb too, and cannot tell the two apart): the session refuses
it before it gets here (7.30.27+, see
C_IEVENT_SRV, “A frame the gate cannot route”).
And an event of a peer that a subscriber of the gate does not take in its
current state (no action for it there, no mt_inject_event) is refused by
the gate: the frame is dropped, the channel stays, and the subscriber never
sees it. The subscriber is often the yuno’s main gclass, subscribed to every
event of the gate, and a peer can name any event some gclass has as public.
Said with a WARNING at most once per 10 s per channel, suppressed the ones
not said since. The rest are said, once, when the channel closes or the gate
stops (7.30.27+). Up to 7.30.26 each such frame was an “Event NOT DEFINED in
state” in the subscriber, an ERROR with a stack: 200 frames were 200 stacks
and over 1 MB of log. The same event sent by the yuno’s own code is still that
loud error. The check is
gobj_find_subscriber_refusing_event().
WARNING "event of a peer the yuno does not take now, refused",
"event": "EV_TRANGER_RECORD_ADDED", "subscriber": "C_AGENT^agent",
"state": "ST_IDLE", "channel": "input-3", "peername": "10.0.0.7:51200",
"username": "operator@example.com", "suppressed": 0
INFO "Events of a peer refused, not said", "why": "the channel closed",
"channel": "input-3", "not_said": 199A send with no channel to go to (none named, none open) is refused with
“No channel FOUND to send”, an ERROR said once every 1000 in a row, per
gate, its repeated the ones not said since the last one. A send to all
channels (__send_type__ 1) that finds none goes through the same brake. A
drop (EV_DROP) of a channel name the gate does not have (“channel NOT
FOUND”) has a brake of its own, so a drop never hides the first send of an
outage, nor counts as one of its sends. The run ends when a send finds a
channel (for the drops, when a drop finds its channel), or when the gate
stops: then ONE INFO says it (“Sends with no channel ended”, “Drops of a
channel not found ended”), with the whole run and the ones the brake did not
say, and the count starts again, so a new outage is said at once (7.30.27+.
Up to 7.30.26, every send said it, the send to all with a stack, and nothing
said the end of the run):
ERROR: {"msg": "No channel FOUND to send", "event": "EV_SEND_MESSAGE", "repeated": 0, ...}
ERROR: {"msg": "No channel FOUND to send", "event": "EV_SEND_MESSAGE", "repeated": 999, ...}
INFO: {"msg": "Sends with no channel ended", "why": "a channel found to send", "without_channel": 1500, "not_said": 499, ...}The messages that the gate failed to send are counted in its stats,
txMsgsNoChannel (a drop is not a message).
tests/c/c_iogate_channel.
Commands¶
| Command | Description |
|---|---|
view-channels | List managed channels and their state. |
enable-channel / disable-channel | Enable or disable a channel. |
trace-on-channel / trace-off-channel | Toggle tracing on a channel. |
reset-stats-channel | Reset channel statistics. |
Stats¶
stats-yuno on a C_IOGATE service answers the usual envelope, with the
gate’s counters and rates under data. Up to 7.25.21 its mt_stats returned
the bare counters and the command answered -1 with no comment (the same
for C_QIOGATE).
ycommand -c 'stats-yuno id=<id> service=__input_side__'
# {"result": 0, "data": {"txMsgs": 1519, "rxMsgs": 1520, "txMsgsec": 12, "rxMsgsec": 12, ..., "txMsgsNoChannel": 0}}From C, a parent reads it the same way:
json_t *jn_stats = gobj_stats(gobj_gate, NULL, 0, gobj);
json_t *jn_data = kw_get_dict(gobj, jn_stats, "data", 0, 0); // the counters
json_int_t txMsgs = kw_get_int(gobj, jn_data, "txMsgs", 0, 0);
JSON_DECREF(jn_stats)C_QIOGATE¶
Queued I/O gate — extends C_IOGATE with persistent message queuing backed by timeranger. Provides ack-based delivery with automatic retry.
| Property | Value |
|---|---|
| States | ST_STOPPED, ST_IDLE |
| Input events | same as C_IOGATE + EV_TIMEOUT |
| Output events | same as C_IOGATE |
Key attributes¶
| Attribute | Type | Description |
|---|---|---|
tranger_path | string | Path for the timeranger queue storage. |
tranger_database | string | Database name. |
topic_name | string | Queue topic name. |
max_pending_acks | integer | Maximum messages in flight without ack. |
timeout_ack | integer | Ack timeout in seconds. |
timeout_poll | integer | Poll of the queue, in milliseconds (default 1000): it resends what waits for its ack and checks the backup. 0 or less is raised to 1000 ms with a WARNING, at start and when code writes it later (“timeout_poll 0 or less never polls, the floor is used”): up to 7.26.9 such a gate never resent anything. Example: 'kw': {'timeout_poll': 2000}. |
backup_queue_size | integer | Backup retention size. |
alert_queue_size | integer | Queue size step for alerts (default 2000, 0: no alert): at each step the queue climbs, the gate logs an ERROR (“ALERT Queuing N msgs”) and sends an email through the emailsender service of its yuno, at most one a minute for each gate. A step is alerted again only after the queue went half a step below it: with 1000, an alert at 10000, none while the queue swings 9999 <-> 10000, and a new one at 10000 only after it fell below 9500. Example: 'kw': {'alert_queue_size': 1000}. Up to 7.30.3 the test was “the size is a multiple of it”, so a queue that swung around a multiple alerted at every message that landed on it, and one gate’s email silenced the others’ for a minute. |
disable_alert | boolean | No alert at all, neither the ERROR nor the email (default false). For the gates of a test, which fill their queue on purpose. Example: 'kw': {'alert_queue_size': 1000, 'disable_alert': true}. |
min_free_disk_percentage | integer | The gate refuses new messages while the disk of tranger_path has less free space than this, in percent (default 10). 0 or less: it refuses them only when the queue cannot save one. Example: 'kw': {'min_free_disk_percentage': 5}. |
msgs_in_queue | integer (stats) | Messages in the queue, not acked yet: a gauge, read live. |
pending_acks | integer (stats) | Messages sent and waiting for their ack: a gauge, read live. |
dropped_msgs | integer (stats) | Messages refused because the queue failed to save them, or failed to open, since the yuno started. |
exitOnError | integer | What a queue that cannot be opened at the start does to the yuno (default -1). Not 0: one ERROR, “Queue cannot be opened: exiting”, and the yuno exits with this code (-1: its watcher launches it again). 0: the gate degrades and the yuno goes on (see A queue that cannot be opened). Example: 'kw': {'exitOnError': 0}. |
queue_ready | bool (stats) | The queue is open. False while stopped, and when it failed to open at the start with exitOnError 0 (see A queue that cannot be opened). |
msgs_in_queue and pending_acks are stats attributes, so they are in the
stats of the gate itself and in those of a C_MQIOGATE above it, one entry
per child (up to 7.25.21 only the gate’s own mt_stats said them, which a
C_MQIOGATE never asks: the size of a persistent link’s queue was not readable
through it).
ycommand -c 'stats-yuno id=<id> service=__output_side__'
# {"output-qiogate-sensors": {"msgs_in_queue": 12, "pending_acks": 3, "dropped_msgs": 0, ...}, ...}A full disk¶
A gate whose queue cannot save a message refuses it and goes on. The
EV_SEND_MESSAGE returns -1 to its sender, and dropped_msgs counts it.
What is already in the queue is still delivered. The gate refuses new
messages:
While the disk of
tranger_pathhas less free space thanmin_free_disk_percentage(checked every second): so no record is half written on a full disk.After a save that failed, until the next check (one second later).
It logs one ERROR when it starts to refuse (“Queue refuses new messages”,
with the cause and the free_percent), and one INFO when it accepts them
again (“Queue accepts new messages again”, with the count dropped). The
poll of the gate does the check also when no message arrives, so the gate
recovers by itself when disk space is free again. Up to 7.28.3, a message that the
queue failed to save aborted the yuno (CRITICAL “Message NOT SAVED in the
queue”). The yuno started again, loaded its queue and aborted again on the
next message, in a loop, for as long as the disk stayed full.
A sender sees the refusal in the return of the event:
if(gobj_send_event(gobj_output_side, EV_SEND_MESSAGE, kw, gobj) < 0) {
// the queue refused it: see dropped_msgs and the gate's ERROR
}A queue that cannot be opened¶
The queue is the role of a qiogate, so by default a gate whose queue cannot
be opened at its start (a tranger_path that cannot be made, a disk, a
permission, a bad config) ends the yuno, at the open: the causes logged
by timeranger2, one ERROR, “Queue cannot be opened: exiting” (with path,
queue, topic_name, exitOnError), and the yuno exits with exitOnError
(-1). Its watcher launches it again, from 2 s up to each 60 s, so the gate
opens by itself when the cause is gone, the same cadence as the degraded
gate below. Up to 7.30.17 the yuno went on without its queue and aborted at
its first message (CRITICAL “trq_msgs NULL”). 7.30.18 degraded every gate.
With exitOnError 0 (a queue that is secondary for its yuno), the gate
goes on without it: queue_ready
false, and every message refused (-1 to its sender, counted in
dropped_msgs, no log per message). It says it ONCE, “Queue cannot be
opened: the gate refuses messages, trying again” (with path, queue,
topic_name), after the causes logged by timeranger2, and tries to open the
queue again at each timeout of its timer, the wait doubling from 1 s up to
60 s. Each try writes the causes of timeranger2 again. When it opens: one
INFO, “Queue opened: the gate accepts messages”. What the queue held is
loaded, and sent if the bottom side is open. An empty tranger_path,
tranger_database or topic_name is not tried again (they are read only):
“Queue cannot be opened, its config is bad: the gate refuses messages”. A
stop takes the tries back.
A log watcher alerts on the ERROR, or on queue_ready false:
ycommand -c 'stats-yuno id=<id> service=__output_side__'
# {"output-qiogate-sensors": {"queue_ready": false, "dropped_msgs": 40, ...}, ...}{
"name": "output-qiogate-sensors",
"gclass": "C_QIOGATE",
"kw": {
"tranger_path": "/yuneta/store/queues/gate_sensors",
"tranger_database": "sensors",
"topic_name": "msgs",
"exitOnError": 0
}
}tests/c/c_qiogate_stats (exitOnError 0) and
tests/c/c_tcp_s_addr_in_use’s test_exit_on_error (the default exits).
What the queue keeps¶
The queue keeps JSON: what the gate saves is a copy of the message, its
metadata (__*__ keys) taken out unless with_metadata. A binary field at
its top level (a gbuffer) is saved as its content, in the serialized form
of kw_serialize() (__gbuffer___), and the copy keeps no reference of it. Up to 7.30.17 the copy increfed the gbuffer and nobody gave
it back -- one leaked per message -- and its address was saved as a number.
json_t *kw = json_pack("{s:s, s:I}", "id", "1", "gbuffer", (json_int_t)(uintptr_t)gbuf);
gobj_send_event(qiogate, EV_SEND_MESSAGE, kw, gobj); // queued as {"id": "1", "__gbuffer___": ...}With with_metadata the __*__ keys are kept, all but __temp__ (7.30.27+):
it is local, the channel a message came in by and a pointer of this process
(C_IOGATE), and the queue sends what it keeps to the peer. Up to 7.30.26 it
was queued and sent.
json_t *kw = json_pack("{s:s, s:s, s:{s:I}}",
"id", "1", "__md_x__", "kept", "__temp__", "channel_gobj", (json_int_t)(uintptr_t)channel);
gobj_send_event(qiogate, EV_SEND_MESSAGE, kw, gobj); // queued as {"id": "1", "__md_x__": "kept"}Commands¶
| Command | Description |
|---|---|
queue_mark_pending | Mark the messages of a range (from-rowid, to-rowid) as pending on disk: a load sends them again. |
queue_mark_notpending | Mark them as not pending: a load does not send them. |
Both need the yuno paused, and act on the OPEN queue: the range is read into
a list of its own, the live queue and its messages in memory are not touched,
and a mark is taken at the next load (the next start of the gate). On a gate
whose queue is not open (queue_ready false) they answer -1, “the queue
is not open (queue_ready false): nothing marked”. Up to 7.30.17 they opened
the queue again over the open one (“tranger NOT NULL”, the live store shut
down) and closed it after them. This left the gate with no queue. On a
queue that failed to open, they crashed the yuno.
ycommand -c 'command-yuno id=<id> service=<the C_QIOGATE> command=queue_mark_pending from-rowid=1200 to-rowid=1300'
# 101 messages marked as PENDINGC_MQIOGATE¶
Multiple-queue I/O gate. It sends each message to one of its C_QIOGATE
children, chosen from a key of the message, or to all of them. Each child is
a persistent queue towards one destination, so an outage of one destination
does not stop the others. This is the entry gateway that spreads the keys of
a treedb over several nodes (see the key).
Its children must all be C_QIOGATE.
| Property | Value |
|---|---|
| States | ST_STOPPED, ST_IDLE |
| Input events | same as C_IOGATE |
| Output events | same as C_IOGATE |
Key attributes¶
| Attribute | Type | Description |
|---|---|---|
method | string | lastdigits (default): send to ONE child. broadcast: send to every child. |
digits | integer | How many characters at the end of the key value choose the child. Default 1. |
key | string | Field of the message kw that holds the key. Default id. It must be a string. |
How lastdigits chooses the child¶
Take the last
digitscharacters ofkw[key].Read them as a decimal number if they are all digits. Otherwise read them as hexadecimal.
The child is that number modulo the number of children (child 0 is the first).
With two children and digits: 1, id: "dev-0042" goes to output-0
(2 % 2 = 0) and id: "dev-0043" goes to output-1. With id: "a1b2c3" the
last character 3 is used.
The same key always goes to the same child while the number of children stays the same. If you add or remove a child, keys move to other children. Plan the number of children, or move the data of the keys with them.
A key that is not a string, or is empty, is logged as an error and goes to the first child.
Example¶
Two destinations, one queue each (the queue kw is the C_QIOGATE one,
shortened here):
{
"name": "__output_side__",
"gclass": "C_MQIOGATE",
"kw": {
"method": "lastdigits",
"digits": 1,
"key": "id"
},
"children": [
{
"name": "output-0",
"gclass": "C_QIOGATE",
"kw": {"topic_name": "gate_events", "tkey": "tm"},
"children": [
{"name": "output-0", "gclass": "C_IOGATE", "children": [
{"name": "output-0", "gclass": "C_CHANNEL", "children": [
{"name": "output-0", "gclass": "C_PROT_TCP4H", "children": [
{"name": "output-0", "gclass": "C_TCP",
"kw": {"url": "tcp://node-0:2000"}}
]}
]}
]}
]
},
{
"name": "output-1",
"gclass": "C_QIOGATE",
"kw": {"topic_name": "gate_events", "tkey": "tm"},
"children": [
{"name": "output-1", "gclass": "C_IOGATE", "children": [
{"name": "output-1", "gclass": "C_CHANNEL", "children": [
{"name": "output-1", "gclass": "C_PROT_TCP4H", "children": [
{"name": "output-1", "gclass": "C_TCP",
"kw": {"url": "tcp://node-1:2000"}}
]}
]}
]}
]
}
]
}Commands¶
| Command | Description |
|---|---|
view-channels | List child queue gates and their state. |