Skip to article frontmatterSkip to article content
Site not loading correctly?

This may be due to an incorrect BASE_URL configuration. See the MyST Documentation for reference.

Gateway GClasses

Message routing, multiplexing, and persistent queuing.

Source: kernel/c/root-linux/src/c_channel.c, c_iogate.c, c_qiogate.c, c_mqiogate.c


C_CHANNEL

Base communication channel — wraps a transport + protocol stack into a single open/closed abstraction with message-rate statistics.

PropertyValue
StatesST_STOPPED, ST_CLOSED, ST_OPENED
Input eventsEV_ON_MESSAGE, EV_SEND_MESSAGE, EV_ON_OPEN, EV_ON_CLOSE, EV_DROP
Output eventsEV_ON_MESSAGE, EV_ON_OPEN, EV_ON_CLOSE

Key attributes

AttributeTypeDescription
openedboolChannel state (read-only).
subscriberpointerGobj receiving output events.
txMsgsec / rxMsgsecintegerCurrent message rate per second (stats).
maxtxMsgsec / maxrxMsgsecintegerPeak message rates (stats).

Message rates

The rates of C_CHANNEL and C_IOGATE are computed when their stats are read: the messages since the previous computation, divided by the time elapsed in milliseconds. A read less than a second after the previous one answers the last rates and does not move the window. So the rate is exact over any interval of a second or more, and two readers share one window (each sees the rate since the other’s read). Up to 7.25.21 the time was taken in whole seconds (a read after 1.9 s divided by 1: +90 %), and every read moved the window, so the messages of a read under a second went into no rate.

What the design does not do, by choice (there is no timer: a gate nobody reads costs nothing):

A rate per second whatever the readers do needs a tick of its own, as the control center keeps (its timeout attr).

A C_CHANNEL read by its parent (a gate) answers its BARE counters, which the gate merges into its own stats.


C_IOGATE

I/O gate — routes messages between channels and upper-layer destinations. Supports one-at-a-time (rotated) or broadcast delivery.

PropertyValue
StatesST_STOPPED, ST_IDLE
Input eventsEV_ON_MESSAGE, EV_SEND_MESSAGE, EV_ON_IEV_MESSAGE, EV_SEND_IEV, EV_ON_OPEN, EV_ON_CLOSE, EV_DROP, EV_STOPPED
Output eventsEV_ON_MESSAGE, EV_ON_OPEN, EV_ON_CLOSE

Key attributes

AttributeTypeDescription
persistent_channelsboolKeep channels alive across reconnections.
send_typeinteger0 = rotated one-at-a-time, 1 = broadcast to all. A kw can choose for itself with __send_type__.

Send to all

With send_type 1 every open channel gets the message. A message in a gbuffer is read out by the channel that sends it (C_PROT_TCP4H appends it to its frame), so every open channel but the last gets a COPY of what is left to read -- with its secret flag, label and address -- and the last one takes the original gbuffer. So a sender that kept a reference to that gbuffer finds it read out after a send to all, as after a send to one. A kw without a gbuffer is shared (kw_incref()). Up to 7.25.20 the gbuffer was shared: the first channel sent the message, and the others an empty frame, which a C_PROT_TCP4H peer drops the connection on (“frame_length cannot be 0”).

gbuffer_t *gbuf = gbuffer_create(len, len);
gbuffer_append(gbuf, data, len);
gobj_send_event(gate, EV_SEND_MESSAGE,
    json_pack("{s:I, s:i}",
        "gbuffer", (json_int_t)(uintptr_t)gbuf,     // the kw owns it
        "__send_type__", 1                          // to every open channel
    ),
    gobj
);

tests/c/c_tcp (test8).

__send_type__ is a key of the yuno, like __temp__: the kw of a send is built by the yuno’s own code, never taken from a peer (see iev_create_from_gbuffer()), so the frontier does not delete it.

The channel of a message

What the gate publishes (EV_ON_MESSAGE, EV_ON_OPEN, EV_ON_CLOSE, and the event of an EV_ON_IEV_MESSAGE) carries the channel it came in by, in __temp__: channel, its name, and channel_gobj, its pointer. A send (EV_SEND_MESSAGE, EV_SEND_IEV) whose kw carries them goes to that channel, so an answer goes back by the channel of its request:

PRIVATE int ac_on_message(hgobj gobj, gobj_event_t event, json_t *kw, hgobj src)
{
    // kw: {"gbuffer": ..., "__temp__": {"channel": "input-1", "channel_gobj": ...}}
    json_t *kw_answer = json_pack("{s:I, s:O}",
        "gbuffer", (json_int_t)(uintptr_t)gbuf_answer,
        "__temp__", json_object_get(kw, "__temp__")
    );
    gobj_send_event(src, EV_SEND_MESSAGE, kw_answer, gobj);   // to input-1
    KW_DECREF(kw)
    return 0;
}

__temp__ is local to the yuno: a peer’s is dropped when its frame is decoded, and it is left out of every frame sent (7.30.27+, see iev_create_from_gbuffer()). So a __temp__ that is not a dict was put by code of this yuno: the gate replaces it, with one ERROR, “temp of a message is not a dict, replaced” (7.30.27+. Up to 7.30.26, it was two ERRORs and the value stayed, and the subscriber read it as the channel).

The event of an EV_ON_IEV_MESSAGE is one a peer sent, and the gate publishes it as its own. So a peer can never name an event the transport publishes itself (EV_ON_OPEN, EV_ON_CLOSE, EV_ON_MESSAGE, ...), nor an event a treedb publishes (EV_TREEDB_NODE_CREATED, _UPDATED, _DELETED, _LINKED, _UNLINKED: the gate’s subscriber is often subscribed to its own treedb too, and cannot tell the two apart): the session refuses it before it gets here (7.30.27+, see C_IEVENT_SRV, “A frame the gate cannot route”).

And an event of a peer that a subscriber of the gate does not take in its current state (no action for it there, no mt_inject_event) is refused by the gate: the frame is dropped, the channel stays, and the subscriber never sees it. The subscriber is often the yuno’s main gclass, subscribed to every event of the gate, and a peer can name any event some gclass has as public. Said with a WARNING at most once per 10 s per channel, suppressed the ones not said since. The rest are said, once, when the channel closes or the gate stops (7.30.27+). Up to 7.30.26 each such frame was an “Event NOT DEFINED in state” in the subscriber, an ERROR with a stack: 200 frames were 200 stacks and over 1 MB of log. The same event sent by the yuno’s own code is still that loud error. The check is gobj_find_subscriber_refusing_event().

WARNING "event of a peer the yuno does not take now, refused",
    "event": "EV_TRANGER_RECORD_ADDED", "subscriber": "C_AGENT^agent",
    "state": "ST_IDLE", "channel": "input-3", "peername": "10.0.0.7:51200",
    "username": "operator@example.com", "suppressed": 0
INFO "Events of a peer refused, not said", "why": "the channel closed",
    "channel": "input-3", "not_said": 199

A send with no channel to go to (none named, none open) is refused with “No channel FOUND to send”, an ERROR said once every 1000 in a row, per gate, its repeated the ones not said since the last one. A send to all channels (__send_type__ 1) that finds none goes through the same brake. A drop (EV_DROP) of a channel name the gate does not have (“channel NOT FOUND”) has a brake of its own, so a drop never hides the first send of an outage, nor counts as one of its sends. The run ends when a send finds a channel (for the drops, when a drop finds its channel), or when the gate stops: then ONE INFO says it (“Sends with no channel ended”, “Drops of a channel not found ended”), with the whole run and the ones the brake did not say, and the count starts again, so a new outage is said at once (7.30.27+. Up to 7.30.26, every send said it, the send to all with a stack, and nothing said the end of the run):

ERROR: {"msg": "No channel FOUND to send", "event": "EV_SEND_MESSAGE", "repeated": 0, ...}
ERROR: {"msg": "No channel FOUND to send", "event": "EV_SEND_MESSAGE", "repeated": 999, ...}
INFO:  {"msg": "Sends with no channel ended", "why": "a channel found to send", "without_channel": 1500, "not_said": 499, ...}

The messages that the gate failed to send are counted in its stats, txMsgsNoChannel (a drop is not a message).

tests/c/c_iogate_channel.

Commands

CommandDescription
view-channelsList managed channels and their state.
enable-channel / disable-channelEnable or disable a channel.
trace-on-channel / trace-off-channelToggle tracing on a channel.
reset-stats-channelReset channel statistics.

Stats

stats-yuno on a C_IOGATE service answers the usual envelope, with the gate’s counters and rates under data. Up to 7.25.21 its mt_stats returned the bare counters and the command answered -1 with no comment (the same for C_QIOGATE).

ycommand -c 'stats-yuno id=<id> service=__input_side__'
# {"result": 0, "data": {"txMsgs": 1519, "rxMsgs": 1520, "txMsgsec": 12, "rxMsgsec": 12, ..., "txMsgsNoChannel": 0}}

From C, a parent reads it the same way:

json_t *jn_stats = gobj_stats(gobj_gate, NULL, 0, gobj);
json_t *jn_data = kw_get_dict(gobj, jn_stats, "data", 0, 0);   // the counters
json_int_t txMsgs = kw_get_int(gobj, jn_data, "txMsgs", 0, 0);
JSON_DECREF(jn_stats)

C_QIOGATE

Queued I/O gate — extends C_IOGATE with persistent message queuing backed by timeranger. Provides ack-based delivery with automatic retry.

PropertyValue
StatesST_STOPPED, ST_IDLE
Input eventssame as C_IOGATE + EV_TIMEOUT
Output eventssame as C_IOGATE

Key attributes

AttributeTypeDescription
tranger_pathstringPath for the timeranger queue storage.
tranger_databasestringDatabase name.
topic_namestringQueue topic name.
max_pending_acksintegerMaximum messages in flight without ack.
timeout_ackintegerAck timeout in seconds.
timeout_pollintegerPoll of the queue, in milliseconds (default 1000): it resends what waits for its ack and checks the backup. 0 or less is raised to 1000 ms with a WARNING, at start and when code writes it later (“timeout_poll 0 or less never polls, the floor is used”): up to 7.26.9 such a gate never resent anything. Example: 'kw': {'timeout_poll': 2000}.
backup_queue_sizeintegerBackup retention size.
alert_queue_sizeintegerQueue size step for alerts (default 2000, 0: no alert): at each step the queue climbs, the gate logs an ERROR (“ALERT Queuing N msgs”) and sends an email through the emailsender service of its yuno, at most one a minute for each gate. A step is alerted again only after the queue went half a step below it: with 1000, an alert at 10000, none while the queue swings 9999 <-> 10000, and a new one at 10000 only after it fell below 9500. Example: 'kw': {'alert_queue_size': 1000}. Up to 7.30.3 the test was “the size is a multiple of it”, so a queue that swung around a multiple alerted at every message that landed on it, and one gate’s email silenced the others’ for a minute.
disable_alertbooleanNo alert at all, neither the ERROR nor the email (default false). For the gates of a test, which fill their queue on purpose. Example: 'kw': {'alert_queue_size': 1000, 'disable_alert': true}.
min_free_disk_percentageintegerThe gate refuses new messages while the disk of tranger_path has less free space than this, in percent (default 10). 0 or less: it refuses them only when the queue cannot save one. Example: 'kw': {'min_free_disk_percentage': 5}.
msgs_in_queueinteger (stats)Messages in the queue, not acked yet: a gauge, read live.
pending_acksinteger (stats)Messages sent and waiting for their ack: a gauge, read live.
dropped_msgsinteger (stats)Messages refused because the queue failed to save them, or failed to open, since the yuno started.
exitOnErrorintegerWhat a queue that cannot be opened at the start does to the yuno (default -1). Not 0: one ERROR, “Queue cannot be opened: exiting”, and the yuno exits with this code (-1: its watcher launches it again). 0: the gate degrades and the yuno goes on (see A queue that cannot be opened). Example: 'kw': {'exitOnError': 0}.
queue_readybool (stats)The queue is open. False while stopped, and when it failed to open at the start with exitOnError 0 (see A queue that cannot be opened).

msgs_in_queue and pending_acks are stats attributes, so they are in the stats of the gate itself and in those of a C_MQIOGATE above it, one entry per child (up to 7.25.21 only the gate’s own mt_stats said them, which a C_MQIOGATE never asks: the size of a persistent link’s queue was not readable through it).

ycommand -c 'stats-yuno id=<id> service=__output_side__'
# {"output-qiogate-sensors": {"msgs_in_queue": 12, "pending_acks": 3, "dropped_msgs": 0, ...}, ...}

A full disk

A gate whose queue cannot save a message refuses it and goes on. The EV_SEND_MESSAGE returns -1 to its sender, and dropped_msgs counts it. What is already in the queue is still delivered. The gate refuses new messages:

It logs one ERROR when it starts to refuse (“Queue refuses new messages”, with the cause and the free_percent), and one INFO when it accepts them again (“Queue accepts new messages again”, with the count dropped). The poll of the gate does the check also when no message arrives, so the gate recovers by itself when disk space is free again. Up to 7.28.3, a message that the queue failed to save aborted the yuno (CRITICAL “Message NOT SAVED in the queue”). The yuno started again, loaded its queue and aborted again on the next message, in a loop, for as long as the disk stayed full.

A sender sees the refusal in the return of the event:

if(gobj_send_event(gobj_output_side, EV_SEND_MESSAGE, kw, gobj) < 0) {
    // the queue refused it: see dropped_msgs and the gate's ERROR
}

A queue that cannot be opened

The queue is the role of a qiogate, so by default a gate whose queue cannot be opened at its start (a tranger_path that cannot be made, a disk, a permission, a bad config) ends the yuno, at the open: the causes logged by timeranger2, one ERROR, “Queue cannot be opened: exiting” (with path, queue, topic_name, exitOnError), and the yuno exits with exitOnError (-1). Its watcher launches it again, from 2 s up to each 60 s, so the gate opens by itself when the cause is gone, the same cadence as the degraded gate below. Up to 7.30.17 the yuno went on without its queue and aborted at its first message (CRITICAL “trq_msgs NULL”). 7.30.18 degraded every gate.

With exitOnError 0 (a queue that is secondary for its yuno), the gate goes on without it: queue_ready false, and every message refused (-1 to its sender, counted in dropped_msgs, no log per message). It says it ONCE, “Queue cannot be opened: the gate refuses messages, trying again” (with path, queue, topic_name), after the causes logged by timeranger2, and tries to open the queue again at each timeout of its timer, the wait doubling from 1 s up to 60 s. Each try writes the causes of timeranger2 again. When it opens: one INFO, “Queue opened: the gate accepts messages”. What the queue held is loaded, and sent if the bottom side is open. An empty tranger_path, tranger_database or topic_name is not tried again (they are read only): “Queue cannot be opened, its config is bad: the gate refuses messages”. A stop takes the tries back.

A log watcher alerts on the ERROR, or on queue_ready false:

ycommand -c 'stats-yuno id=<id> service=__output_side__'
# {"output-qiogate-sensors": {"queue_ready": false, "dropped_msgs": 40, ...}, ...}
{
    "name": "output-qiogate-sensors",
    "gclass": "C_QIOGATE",
    "kw": {
        "tranger_path": "/yuneta/store/queues/gate_sensors",
        "tranger_database": "sensors",
        "topic_name": "msgs",
        "exitOnError": 0
    }
}

tests/c/c_qiogate_stats (exitOnError 0) and tests/c/c_tcp_s_addr_in_use’s test_exit_on_error (the default exits).

What the queue keeps

The queue keeps JSON: what the gate saves is a copy of the message, its metadata (__*__ keys) taken out unless with_metadata. A binary field at its top level (a gbuffer) is saved as its content, in the serialized form of kw_serialize() (__gbuffer___), and the copy keeps no reference of it. Up to 7.30.17 the copy increfed the gbuffer and nobody gave it back -- one leaked per message -- and its address was saved as a number.

json_t *kw = json_pack("{s:s, s:I}", "id", "1", "gbuffer", (json_int_t)(uintptr_t)gbuf);
gobj_send_event(qiogate, EV_SEND_MESSAGE, kw, gobj);   // queued as {"id": "1", "__gbuffer___": ...}

With with_metadata the __*__ keys are kept, all but __temp__ (7.30.27+): it is local, the channel a message came in by and a pointer of this process (C_IOGATE), and the queue sends what it keeps to the peer. Up to 7.30.26 it was queued and sent.

json_t *kw = json_pack("{s:s, s:s, s:{s:I}}",
    "id", "1", "__md_x__", "kept", "__temp__", "channel_gobj", (json_int_t)(uintptr_t)channel);
gobj_send_event(qiogate, EV_SEND_MESSAGE, kw, gobj);   // queued as {"id": "1", "__md_x__": "kept"}

Commands

CommandDescription
queue_mark_pendingMark the messages of a range (from-rowid, to-rowid) as pending on disk: a load sends them again.
queue_mark_notpendingMark them as not pending: a load does not send them.

Both need the yuno paused, and act on the OPEN queue: the range is read into a list of its own, the live queue and its messages in memory are not touched, and a mark is taken at the next load (the next start of the gate). On a gate whose queue is not open (queue_ready false) they answer -1, “the queue is not open (queue_ready false): nothing marked”. Up to 7.30.17 they opened the queue again over the open one (“tranger NOT NULL”, the live store shut down) and closed it after them. This left the gate with no queue. On a queue that failed to open, they crashed the yuno.

ycommand -c 'command-yuno id=<id> service=<the C_QIOGATE> command=queue_mark_pending from-rowid=1200 to-rowid=1300'
# 101 messages marked as PENDING

C_MQIOGATE

Multiple-queue I/O gate. It sends each message to one of its C_QIOGATE children, chosen from a key of the message, or to all of them. Each child is a persistent queue towards one destination, so an outage of one destination does not stop the others. This is the entry gateway that spreads the keys of a treedb over several nodes (see the key).

Its children must all be C_QIOGATE.

PropertyValue
StatesST_STOPPED, ST_IDLE
Input eventssame as C_IOGATE
Output eventssame as C_IOGATE

Key attributes

AttributeTypeDescription
methodstringlastdigits (default): send to ONE child. broadcast: send to every child.
digitsintegerHow many characters at the end of the key value choose the child. Default 1.
keystringField of the message kw that holds the key. Default id. It must be a string.

How lastdigits chooses the child

  1. Take the last digits characters of kw[key].

  2. Read them as a decimal number if they are all digits. Otherwise read them as hexadecimal.

  3. The child is that number modulo the number of children (child 0 is the first).

With two children and digits: 1, id: "dev-0042" goes to output-0 (2 % 2 = 0) and id: "dev-0043" goes to output-1. With id: "a1b2c3" the last character 3 is used.

The same key always goes to the same child while the number of children stays the same. If you add or remove a child, keys move to other children. Plan the number of children, or move the data of the keys with them.

A key that is not a string, or is empty, is logged as an error and goes to the first child.

Example

Two destinations, one queue each (the queue kw is the C_QIOGATE one, shortened here):

{
    "name": "__output_side__",
    "gclass": "C_MQIOGATE",
    "kw": {
        "method": "lastdigits",
        "digits": 1,
        "key": "id"
    },
    "children": [
        {
            "name": "output-0",
            "gclass": "C_QIOGATE",
            "kw": {"topic_name": "gate_events", "tkey": "tm"},
            "children": [
                {"name": "output-0", "gclass": "C_IOGATE", "children": [
                    {"name": "output-0", "gclass": "C_CHANNEL", "children": [
                        {"name": "output-0", "gclass": "C_PROT_TCP4H", "children": [
                            {"name": "output-0", "gclass": "C_TCP",
                             "kw": {"url": "tcp://node-0:2000"}}
                        ]}
                    ]}
                ]}
            ]
        },
        {
            "name": "output-1",
            "gclass": "C_QIOGATE",
            "kw": {"topic_name": "gate_events", "tkey": "tm"},
            "children": [
                {"name": "output-1", "gclass": "C_IOGATE", "children": [
                    {"name": "output-1", "gclass": "C_CHANNEL", "children": [
                        {"name": "output-1", "gclass": "C_PROT_TCP4H", "children": [
                            {"name": "output-1", "gclass": "C_TCP",
                             "kw": {"url": "tcp://node-1:2000"}}
                        ]}
                    ]}
                ]}
            ]
        }
    ]
}

Commands

CommandDescription
view-channelsList child queue gates and their state.