logcenter

Centralized log collector for a node. Every yuno ships its log records to logcenter over UDP (the peer of the kernel’s to_udp log handler, default udp://127.0.0.1:1992). logcenter counts them by priority, writes them to size/disk-guarded rotatory files, aggregates a summary report, watches free disk/memory, and can e-mail the report or even restart the node on a queue alarm.

Architecture¶

C_LOGCENTER            <- counting, rotation, summary, monitoring, self-heal
    > C_GSS_UDP_S      <- UDP server (udp://127.0.0.1:1992), receives log records
    > C_TIMER          <- periodic disk/memory monitor

Summary e-mails are sent by publishing EV_SEND_EMAIL to the emailsender service (see emailsender).

Wire format¶

Each UDP datagram from the to_udp handler is:

<priority:1 char><sequence:8 hex><message bytes...><crc:8 hex>

logcenter verifies the trailing CRC, tracks the sequence (gaps are only logged under the debug trace), strips both, then files the message. If the message body is JSON ({...}) it is also folded into the summary tree.

A record longer than one datagram (1500 bytes) is sent in PIECES, a NUL after the last one. C_GSS_UDP_S joins the pieces per PEER, "ip:port" of the sending yuno, so the long records of two yunos that arrive interleaved come out whole. Up to 7.25.4 every peer shared one channel, and such records were joined into one corrupt record -- one real cause of “json_load_callback() FAILED” on logcenter, written off until then as UDP noise.

pause-yuno and play-yuno of logcenter stop and start its UDP server, which opens a new socket and a new read on it. Up to 7.25.4 the read went on with the number of the socket the pause had closed -- the log file opened by the play just before could have it -- and logcenter stopped receiving with nothing logged. See C_UDP_S (Stop and start again).

What it does with each record¶

1. Counts it by syslog priority — Alert, Critical, Error, Warning, Info, Debug, Audit, Monitor.

2. Writes it to a rotatory log file under the yuno’s logs/ directory, named from log_filename (mask DD/MM/CCYY-W-ZZZ). Rotation is bounded by max_rotatoryfile_size and pauses when free disk drops below min_free_disk. Each NEW file (a new day, or a size rotation) sends the summary e-mail and then resets the counters and the summary trees. With send_summary_disabled it does neither: no e-mail, and the counters go on adding up until reset-counters or a restart (they are not persistent). The same file opened again after a write that failed (a quota, no inodes, EIO) or after an rm is not a new file: no e-mail. When the new file of a day cannot be opened at once, its e-mail goes at the first open that works, once. A log_filename with no date letter (logcenter.log) is one file for ever: it is appended to at every start, never emptied by a date. See rotatory.

3. Aggregates JSON records into a summary grouped by msgset → msg → count (separately for errors, warnings, infos).

Configuration¶

Inspect the effective config at runtime with ycommand command-yuno id=<id> service=__yuno__ command=view-config.

Commands¶

The summary report (from display-summary / send-summary) looks like:

{
    "Global Counters": { "Alert": 0, "Critical": 0, "Error": 2, "Warning": 39, "Info": 1185, "Debug": 1937, "Audit": 0, "Monitor": 0 },
    "Node Owner": "artgins",
    "Global Errors":   { "App": { "email NOT sent": 1 }, "Liburing": { "getaddrinfo() FAILED": 1 } },
    "Global Warnings": { "Info": { "No authz db, authz only to local access": 1 } },
    "Global Infos":    { "Connect Disconnect": { "Connected": 295, "Disconnected": 386 } }
}

Monitoring & self-heal¶

• Resource watch: on its timer logcenter checks free disk and free RAM (each re-armed for 24 h after a check). When either falls to/below min_free_disk / min_free_mem it logs a warning and e-mails a report.

• Queue-alarm restart: when a record carries a queue_size field and restart_on_alarm is set with queue_size ≥ queue_restart_limit, it runs restart_yuneta_command via system(), throttled so a new restart cannot fire until timeout_restart_yuneta has elapsed. This is a last-resort watchdog for a node whose queues are runaway. It is off by default.

Debugging¶

Enable with ycommand command-yuno id=<id> service=__yuno__ command=set-gclass-trace gclass=C_LOGCENTER set=1 level=debug.