Debugging a yuno

This document covers what you do when a yuno does not behave correctly. It explains how to enable the traces that show you what happens, where the output goes, and how to follow one message through several yunos. It also explains the part that the centralized log aggregator (logcenter) plays.

This document is the companion to YUNO_LIFECYCLE.md. That one covers how the agent manages yunos. This one covers how to look inside them.

1. Mental model¶

Three observation layers are independent. A confusion between them is the first source of frustration:

You configure four destinations per yuno, with daemon_log_handlers in the yuno config JSON:

                        ┌──────────────────────────┐
       severity logs    │     file handler         │  → /yuneta/logs/<yuno>/<mask>.log
       + traces  ──────►│     (rotatory, ~8 MB)    │
                        └──────────────────────────┘
                        ┌──────────────────────────┐
                        │     udp handler          │  → udp://host:port
                        │     (default :1992)      │  → typically the logcenter yuno
                        └──────────────────────────┘
                        ┌──────────────────────────┐
                        │     stdout (console mode)│  → terminal when not daemonised
                        └──────────────────────────┘
                        ┌──────────────────────────┐
                        │     remote_log over      │
                        │     ievent / websocket   │  → SPA "dev panel" (live viewer)
                        └──────────────────────────┘

One log line can go to all four destinations at the same time. No destination is “the” log. They are different sinks.

2. Severity levels (gobj_log_*)¶

These are the calls every gclass uses to record events. They are not traces — they fire regardless of trace settings. The six public ones are defined in kernel/c/gobj-c/src/glogger.c:

glogger.c declares two more channels that are not syslog channels:

• LOG_AUDIT (8) — the framework writes these lines without the standard header. They record the command audit. A tool that filters on the standard timestamp format skips them, so read the audit file directly.

• LOG_MONITOR (9) — the monitoring tools use this channel.

Per-yuno HARD RULE (see CLAUDE.md): every error-return path calls gobj_log_error or carries an // Error already logged comment. If you cannot find the error in the log, that yuno has a bug. The log did not lose it.

3. Trace categories¶

A trace is the running commentary that the framework can emit. It is off by default. It is noisy, so enable it only when you need it, and disable it when you finish.

3.1 Global trace levels¶

Defined in s_global_trace_level[16] at kernel/c/gobj-c/src/gobj.c:

These are global bits. When you enable one, it affects every gobj in the yuno.

3.2 Per-gclass trace levels¶

Each gclass declares its own up-to-16 levels in s_user_trace_level[16]. Example: c_tcp_s.c

enum {
    TRACE_LISTEN        = 0x0001,
    TRACE_NOT_ACCEPTED  = 0x0002,
    TRACE_ACCEPTED      = 0x0004,
    TRACE_TLS           = 0x0008,
};
PRIVATE const trace_level_t s_user_trace_level[16] = {
    {"listen",          "Trace listen"},
    {"not-accepted",    "Trace not accepted connections"},
    {"accepted",        "Trace accepted connections"},
    {"tls",             "Trace tls"},
    {0, 0},
};

The names are gclass-specific. Common ones across runtime gclasses:

• C_TCP, C_TCP_S: traffic, connect, tls, listen, accepted, not-accepted

• C_PROT_HTTP_SR, C_PROT_HTTP_CL: traffic

• C_IEVENT_SRV, C_IEVENT_CLI: ievents, ievents2 (the second dumps full kw)

• C_WEBSOCKET: gclass-specific debug for HTTP-upgrade handshakes

• C_TIMER, C_TIMER0: their own tick / periodic

To see what a gclass offers, run get-gclass-trace gclass=<X> (see §4).

3.3 Per-gobj trace levels¶

These levels are the same as the per-gclass levels, but they are scoped to one gobj instance. They are useful when you have ten TCP connections and you want the trace of one connection. API: gobj_set_gobj_trace() at kernel/c/gobj-c/src/gobj.c:11256.

3.4 The no_trace parallel system¶

For every “set trace” command there is a “set no-trace” counterpart. The framework subtracts the no-trace mask from the effective trace mask. So you can enable a noisy level globally, then silence it on specific gclasses or gobjs. Functions: gobj_set_global_no_trace() at gobj.c:11396, gobj_set_gclass_no_trace() at gobj.c:11617, gobj_set_gobj_no_trace() at gobj.c:11746.

3.5 Deep trace mode¶

gobj_set_deep_tracing(level) enables all traces, and the masks do not apply. There is no ycommand for it. It is available only in the C API, and the framework uses it internally for emergency dumps. Do not use it unless you can accept the volume.

4. Turning traces on and off¶

All commands go to the yuno itself, addressed to its __yuno__ service. Handlers in kernel/c/root-linux/src/c_yuno.c:

# discover what a gclass offers
ycommand -c 'command-yuno id=<yuno> service=__yuno__ command=get-gclass-trace gclass=C_TCP_S'

# enable / disable
ycommand -c 'command-yuno id=<yuno> service=__yuno__ command=set-global-trace level=machine set=1'
ycommand -c 'command-yuno id=<yuno> service=__yuno__ command=set-gclass-trace gclass=C_TCP_S level=traffic set=1'
ycommand -c 'command-yuno id=<yuno> service=__yuno__ command=set-gobj-trace gobj=<short_name> level=machine set=1'

# silence (no_trace)
ycommand -c 'command-yuno id=<yuno> service=__yuno__ command=set-global-no-trace level=ev_kw set=1'
ycommand -c 'command-yuno id=<yuno> service=__yuno__ command=set-gclass-no-trace gclass=C_TIMER level=periodic set=1'

# inspect current state
ycommand -c 'command-yuno id=<yuno> service=__yuno__ command=get-global-trace'
ycommand -c 'command-yuno id=<yuno> service=__yuno__ command=get-gclass-trace gclass=C_TCP_S'
ycommand -c 'command-yuno id=<yuno> service=__yuno__ command=get-gobj-trace gobj=<short_name>'

The short form in CLAUDE.md, ycommand -c 'command-yuno id=<id> service=__yuno__ command=…', is exactly this. The shorter form ycommand -c 'set-global-trace …' sends command-yuno to the yuno that is registered as the default yuno.

Persistence¶

• Global traces are persisted: c_yuno.c (save_global_trace) writes them to the yuno’s trace_levels attribute, and they are re-applied on the next start.

• Gclass and gobj traces are live-only. They die with the process.

CAUTION: a forgotten set-global-trace level=machine set=1 survives a restart, and it fills your disk. It gives no message first. Always pair the enable and the disable in the same session.

4.1 When the yuno never reaches the agent (--global-trace)¶

Every command above travels over the yuno’s control channel to the agent. So none of them work for the failure that most needs a trace: a yuno that dies, hangs or fails before that channel is ready. ycommand cannot reach it, and list-yunos reports running=false even while the process is alive.

Since 7.8.2 the levels can be armed on the command line instead:

# one level, or several — repeatable and comma-separated
auth_bff --config-file='[...]' --global-trace=machine
auth_bff --config-file='[...]' --global-trace=machine,create_delete,start_stop

# what levels exist
auth_bff --global-trace=list

The framework applies them after it registers every gclass, and before the first service starts, so they cover start up itself. An unknown level stops the yuno with a message that points at list. The yuno does not ignore it.

To reproduce a yuno that the agent launches, take its command line from running-bin id=<id> or running-keys id=<id>. You can also use the script that the agent writes at /yuneta/realms/<realm>/<yuno>/bin/<role>^<id>.sh. Then append the flag.

Two older methods, and their limits:

• kill -10 <pid> (SIGUSR1) cycles the global mask 0 → 0x00FF0000 → 0x0FFF0000 → 0xFFFF0000 → 0. It is useful on a process that already runs. It does nothing for a problem during start up.

• --verbose-log=N is not a trace switch. It only overrides the stdout log handler’s field bitmask (which fields each line prints). --verbose-log=3 prints fewer fields than the config default of 255, which is why it reads as “it does nothing”. Use --global-trace.

4.2 Two warnings that arrive without being asked for¶

Some failures below the framework cannot wait for a trace, so they report themselves:

• getaddrinfo() BLOCKED the event loop (gobj_log_warning, msgset OS, from yev_loop.c) — name resolution is synchronous and runs inside the loop. A slow resolver therefore stops every gobj in the process, not only the socket that you open. The framework emits the warning with the host and the elapsed msec when the time is more than 1 s. If you see it, the yuno is not slow. It is stopped.

• YUNETAS static_resolv: … in syslog (journalctl) — the CONFIG_FULLY_STATIC resolver writes it. That resolver is below the gobj log and cannot reach it. Three conditions emit the message: a nameserver in /etc/resolv.conf that does not answer (rate limited to one message per nameserver per 5 min), a resolution of more than 1 s, or a failed allocation.

A dead first nameserver costs ~6 s (A + AAAA timeouts) on every lookup, so a yuno that opens many channels can spend minutes in start up. The resolver caches answers since 7.8.2, which limits the cost to the first lookup. But the correction belongs in the node’s /etc/resolv.conf.

5. Reading the logs¶

5.1 File paths¶

Per-yuno log file, built by yuneta_log_file() at kernel/c/root-linux/src/yunetas_environment.c:

/yuneta/logs/<yuno_role_plus_name>/<filename_mask>

The mask is the value that you set in daemon_log_handlers.<handler>.filename_mask (see §5.4). By convention it is <role>-W.log, where a rotation counter replaces the W.

Active log discovery:

ls -lt /yuneta/logs/<yuno>/
tail -f /yuneta/logs/<yuno>/<latest>.log | grep -a "keyword"

5.2 Log line format¶

Every log record is a JSON object built in glogger.c. Fields added automatically by discover() at glogger.c:1231:

Searching is JSON-friendly:

grep -a '"priority":3' /yuneta/logs/<yuno>/<file>.log       # all errors
grep -a '"gclass":"C_TCP_S"' /yuneta/logs/<yuno>/<file>.log  # one gclass
grep -a '"msg":"Event NOT DEFINED in state"' …               # the canonical FSM bug

5.3 Rotation¶

The rotatory library rotates the file when it crosses a size threshold (default 8 MB, configurable via max_megas_rotatoryfile_size, entry_point.c). The library renames the old files, and the active filename never moves. There is no rotation by time. There is no cron. The rotation happens on the next write that crosses the threshold.

5.4 Where to configure handlers¶

In the yuno’s config JSON, under environment.daemon_log_handlers (or console_log_handlers in non-daemon mode), parsed at kernel/c/root-linux/src/entry_point.c:

"environment": {
    "daemon_log_handlers": {
        "to_file": {
            "handler_type": "file",
            "filename_mask": "mqtt_broker-W.log",
            "handler_options": 255
        },
        "to_udp": {
            "handler_type": "udp",
            "url": "udp://127.0.0.1:1992",
            "handler_options": 255
        }
    }
}

handler_options is a bitmask of LOG_HND_OPT_* (glogger.h) that selects which severities the handler accepts. 255 accepts all of them. If you clear bits, the handler drops DEBUG, INFO, AUDIT and the other severities.

To add or remove handlers at run time, use the add-log-handler and del-log-handler commands of c_yuno.c.

6. The FSM trace (machine)¶

This is the most useful trace for the behavior of a gobj. It is defined in glogger.c (trace_machine). Called from the event dispatcher in gobj.c:

• Before dispatch: a 🔜 line per event entry.

  • “Event NOT DEFINED” error: a 📛 line. This is the canonical failure in which the parent FSM does not declare the event of the child (see CLAUDE.md “CHILD vs SERVICE” section).

• After dispatch: a 🔄 line per executed event.

• State change: a 🔀🔀 line.

Two output formats, switched by the integer variable trace_machine_format:

Format 1 — short, ANSI-coloured:

🔜 EV_RX_DATA !!c_tcp:open
🔄 EV_RX_DATA !!c_tcp :open from !!service_main
🔀🔀 mach(!!c_tcp), new st(:closed), prev st(:open)

Default — verbose:

🔜 mach(!!c_tcp), st: :open, ev: EV_RX_DATA, from(!!service_main)
🔄 mach(!!c_tcp), st: :open, ev: EV_RX_DATA, from(c_tcp_s^server)
🔀🔀 mach(!!c_tcp), new st(:closed), prev st(:open)

!! before a name means that the gobj is not running at that moment. Two of them in a row are usually the bug.

Scoping the machine trace¶

The machine trace of a whole yuno gives too much output on anything larger than a toy test. You can make it narrow in two ways:

# only one gclass
ycommand -c 'command-yuno id=<yuno> service=__yuno__ command=set-gclass-trace gclass=C_TCP_S level=machine set=1'

# only one instance
ycommand -c 'command-yuno id=<yuno> service=__yuno__ command=set-gobj-trace gobj=<short_name> level=machine set=1'

Both are live-only — they vanish on restart.

The same trace in the browser¶

This chapter is about a yuno on a node. But a browser SPA runs the same kernel, ported to JavaScript. Since @yuneta/gobj-js 7.9.5 the JS runtime has this level model, with the same names and the same bits. A habit that you learn here therefore transfers, and you can read two traces side by side.

gobj_set_global_trace("machine", true);          // the big one, same as above
gobj_set_gclass_trace("C_MY_VIEW", "machine", true);
gobj_set_gobj_no_trace(noisy_src, "machine", true);   // veto, by the SOURCE

set_log_callback((level, msg) => { ... });       // the trace arrives as `debug`

There is no ycommand on that side. The switch is the call above, and the output goes to the browser console. It goes to any other destination that set_log_callback() selects, and that is how the dev panel of gobj-ui shows the machine inside the app. doc.yuneta.io/navigation runs three demos with the panel connected to that callback. Read them if you want to see the lines before you write your own code.

7. Following a message end-to-end¶

Canonical request flow on a typical Yuneta service:

   external client
         │
         ▼
  ┌─────────────┐   gclass trace 'traffic'
  │   C_TCP_S   │   gobj_trace_dump_gbuf(gobj, gbuf, …)
  └──────┬──────┘
         │
         ▼
  ┌─────────────────┐   gclass trace 'traffic'
  │ C_PROT_HTTP_SR  │
  └────────┬────────┘
           │
           ▼
  ┌─────────────────┐   gclass trace 'ievents' / 'ievents2'
  │   C_IEVENT_SRV  │   trace_inter_event2(gobj, prefix, event, kw)
  └────────┬────────┘
           │
           ▼
  ┌─────────────────┐   global trace 'machine' lights up the FSM dispatch
  │  service gclass │   gclass-specific traces fire its custom emit points
  └────────┬────────┘
           │
           ▼
  ┌─────────────────┐   global trace 'fs'
  │   timeranger2   │   record append + rowid emitted
  │   (treedb)      │
  └────────┬────────┘
           │
           ▼  outbound publish
  ┌─────────────────┐   gclass trace 'ievents' / 'ievents2'
  │   C_IEVENT_SRV  │
  └────────┬────────┘
           │
           ▼
  ┌─────────────────┐   gclass-specific trace
  │  C_WEBSOCKET    │   gobj_trace_dump frames
  └────────┬────────┘
           │
           ▼
        SPA browser

The correlation id¶

Inter-event messages between yunos carry a metadata block named __md_iev__ inside the kw. Inside it is the ievent_gate_stack — a LIFO of hops, each entry: {src_yuno, src_service, dst_yuno, dst_service, user, host, …}.

• Constant IEVENT_STACK_ID = "ievent_gate_stack" at kernel/c/root-linux/src/msg_ievent.h.

• Pushed on outgoing request, popped + reversed on incoming response, at kernel/c/root-linux/src/msg_ievent.c.

To grep the same transaction across multiple yunos’ logs:

grep -a 'ievent_gate_stack' /yuneta/logs/*/*.log | grep '<the user or src_yuno you care about>'

The framework propagates no automatic UUID for calls that are not ievents. A direct C function call has nothing to grep. The correlation is available only when the message crosses an ievent boundary.

Practical sequence to follow one HTTP request¶

YUNO=my_service_01

# 1. ingress + protocol
ycommand -c "command-yuno id=$YUNO service=__yuno__ command=set-gclass-trace gclass=C_TCP_S        level=traffic set=1"
ycommand -c "command-yuno id=$YUNO service=__yuno__ command=set-gclass-trace gclass=C_PROT_HTTP_SR level=traffic set=1"

# 2. internal FSM
ycommand -c "command-yuno id=$YUNO service=__yuno__ command=set-global-trace level=machine set=1"

# 3. broker/topic write
ycommand -c "command-yuno id=$YUNO service=__yuno__ command=set-global-trace level=fs set=1"

# 4. egress to SPA
ycommand -c "command-yuno id=$YUNO service=__yuno__ command=set-gclass-trace gclass=C_IEVENT_SRV level=ievents  set=1"
ycommand -c "command-yuno id=$YUNO service=__yuno__ command=set-gclass-trace gclass=C_IEVENT_SRV level=ievents2 set=1"

# trigger the request, capture the noise
tail -F /yuneta/logs/$YUNO/*.log > /tmp/$YUNO.trace &
# … reproduce …
kill %1

# disable everything
ycommand -c "command-yuno id=$YUNO service=__yuno__ command=set-gclass-trace gclass=C_TCP_S        level=traffic  set=0"
ycommand -c "command-yuno id=$YUNO service=__yuno__ command=set-gclass-trace gclass=C_PROT_HTTP_SR level=traffic  set=0"
ycommand -c "command-yuno id=$YUNO service=__yuno__ command=set-global-trace level=machine set=0"
ycommand -c "command-yuno id=$YUNO service=__yuno__ command=set-global-trace level=fs      set=0"
ycommand -c "command-yuno id=$YUNO service=__yuno__ command=set-gclass-trace gclass=C_IEVENT_SRV level=ievents  set=0"
ycommand -c "command-yuno id=$YUNO service=__yuno__ command=set-gclass-trace gclass=C_IEVENT_SRV level=ievents2 set=0"

8. The SPA-side “dev panel” viewer¶

A SPA built on the JS gobj framework can connect to a yuno. Then it can display live what crosses the websocket: the same lines that you see in the local log file, plus the bodies of the ievent messages.

Wire-up¶

• C side: nothing special — inter-events flow as usual through C_IEVENT_SRV → C_WEBSOCKET.

• JS side: C_IEVENT_CLI (kernel/js/gobj-js/src/c_ievent_cli.js) parses incoming inter-events in ac_on_message() and, if configured, invokes trace_ievent_callback(prefix, iev_msg, direction, size).

• The SPA installs that callback by writing the attribute: gobj_write_attr(gobj_yuno(), "trace_ievent_callback", info_traffic) (kernel/js/gobj-ui/src/yui_dev.js).

• The info_traffic() function (yui_dev.js:29) appends the message into the DOM container #developer-traffic-logger, which lives inside either:

  • the legacy C_YUI_WINDOW modal, or

  • the modern build_dev_panel() modal (yui_dev.js:452).

Both still ship. An app selects one of them from the version of its shell.

Rendering¶

• Header bar: blue, with direction icon (white = sent, yellow = received, red = error), byte size, and HH:MM:SS.SSSS timestamp.

• Body: read-only vanilla-jsoneditor (pinned ^0.23.0, see project memory note about not bumping it).

• Footer: ON/OFF traffic counter.

• Auto-scroll to bottom on each message.

Filtering on the SPA side¶

The UI has no filter. The viewer shows everything that flows through the websocket. The only two controls are on and off: the trace_inter_event boolean and the trace_ievent_callback itself. To filter, change what the yuno emits with the ycommand controls from §4.

Teardown order — the recursion gotcha¶

When the websocket closes, ac_on_close (c_ievent_cli.js:897) fires EV_ON_CLOSE. set_remote_log_functions redirects the JS log_error and log_warning calls to the DOM callback. If it is still installed, the callback captures the warning that the teardown path emits. The callback changes the DOM, the change can fire more events, and those events log again. The result is an infinite recursion.

The correction at c_ievent_cli.js is to call set_remote_log_functions(null) before anything publishes EV_ON_CLOSE. That call clears the hooks and resets them to the console (see helpers.js). The memory note “Remote-log unwire order” records the incident.

9. The logcenter yuno¶

yunos/c/logcenter/ collects the logs that every yuno on the host, or on the LAN, ships over UDP. It is not enabled by default. A yuno ships to UDP only if its config lists a udp handler.

How it listens¶

• UDP server (c_gss_udp_s) on udp://127.0.0.1:1992 by default (c_logcenter.c).

• Wire format: <priority-digit><8hex-seq><json-payload><8hex-crc>, fragmented per udp_frame_size (default 1500, log_udp_handler.c).

What it does on receipt¶

In c_logcenter.c:

• ac_on_message() parses each packet.

• Writes the JSON record to its own rotatory file W.log via write2logs() / _log_bf(). Default size cap 600 MB (max_rotatoryfile_size, in megabytes).

• Updates in-memory counters per severity / msgset / msg (do_log_stats(), c_logcenter.c:858).

What it exposes¶

Commands (c_logcenter.c):

Use it like any other yuno. Target it by yuno_role=logcenter, because the numeric id of the yuno changes with the realm. command-yuno implies the default service:

# rollup counters (Alert/Critical/Error/Warning/Info + Connect/Disconnect breakdown)
ycommand -c 'command-yuno yuno_role=logcenter command=display-summary'

# last N log lines (default ~100; can pass lines=N)
ycommand -c 'command-yuno yuno_role=logcenter command=tail lines=200'

# substring search (parameter is text=, not match=); maxcount caps the hits
ycommand -c 'command-yuno yuno_role=logcenter command=search text="EV_ON_CLOSE" maxcount=20'

# wipe the rollup counters — useful before reproducing an issue so the
# next display-summary only shows the new run
ycommand -c 'command-yuno yuno_role=logcenter command=reset-counters'

Three more commands are useful (c_logcenter.c): send-summary, enable-send-summary and disable-send-summary control the email rollup. restart-yuneta-on-queue-alarm is the auto-recovery hook for a UDP queue that floods.

Per-yuno vs centralized — when to use each¶

• Per-yuno tail when you know which yuno is misbehaving and want raw control over grep. /yuneta/logs/<yuno>/<file>.log is full fidelity.

• logcenter when you need correlation across multiple yunos, or when a yuno crashes too fast to read its own file, or for the rollup counters / email summaries.

Both can run at the same time. The file handler writes locally, and the UDP handler ships to logcenter in parallel. They are not exclusive.

10. Sharp edges¶

10.1 Traces accumulate — disable them when you finish¶

A forgotten set-global-trace level=machine set=1 survives a restart (see §4 Persistence). The logs then grow without limit. Always pair the enable and the disable in the same operational session. Before you leave, make sure that the state is correct with get-global-trace.

10.2 Persistence asymmetry¶

If you persisted a global level by mistake, clear it explicitly with set-global-trace level=<name> set=0. To delete the file does not help, because the value is in the treedb config of the yuno.

10.3 ievent_gate_stack is only on inter-event hops¶

A direct C function call between gobjs in the same yuno does not carry the stack, because there is no metadata to attach. The correlation exists only across yuno boundaries. Plan your traces for this limit.

10.4 LOG_AUDIT lines have no standard header¶

glogger.c writes the audit lines raw. A line filter that expects the timestamp prefix misses them. When you look for operator actions, read the audit file directly.

10.5 UDP can drop¶

UDP logs are not reliable. Under a burst, for example a machine trace that is fully on, the kernel buffer can overflow, and logcenter loses lines. It gives no message. The local file handler drops nothing, so trust the local file when you are not sure.

10.6 ev_kw is enormous¶

set-global-trace level=ev_kw set=1 writes the full kw JSON payload of every event to the log. It is useful on a single-shot test. It is ruinous on a busy service. If you need it narrowly, combine it with a machine trace that is scoped to one gclass.

10.7 SPA dev-panel teardown order¶

set_remote_log_functions(null) MUST come before do_disconnect / destroy_shell. See §8 and memory feedback_remote_log_unwire_order.

10.8 Deep tracing has no ycommand switch¶

gobj_set_deep_tracing() is available only in C (gobj.c). If a yuno generates traces that you cannot configure, look for a gobj_set_deep_tracing call that someone left in its mt_create.

11. Operational recipes¶

11.1 Watch what a yuno does¶

YUNO=<id>
ycommand -c "command-yuno id=$YUNO service=__yuno__ command=set-global-trace level=machine set=1"
tail -F /yuneta/logs/$YUNO/*.log | grep -a '"msg":'
# reproduce
ycommand -c "command-yuno id=$YUNO service=__yuno__ command=set-global-trace level=machine set=0"

11.2 Watch traffic on a TCP/HTTP service¶

ycommand -c "command-yuno id=$YUNO service=__yuno__ command=set-gclass-trace gclass=C_TCP_S        level=traffic set=1"
ycommand -c "command-yuno id=$YUNO service=__yuno__ command=set-gclass-trace gclass=C_PROT_HTTP_SR level=traffic set=1"
tail -F /yuneta/logs/$YUNO/*.log
# … done …
ycommand -c "command-yuno id=$YUNO service=__yuno__ command=set-gclass-trace gclass=C_TCP_S        level=traffic set=0"
ycommand -c "command-yuno id=$YUNO service=__yuno__ command=set-gclass-trace gclass=C_PROT_HTTP_SR level=traffic set=0"

11.3 Watch all logs from this host in one place¶

Enable logcenter. Add this to the config JSON of every yuno:

"daemon_log_handlers": {
    "to_udp": { "handler_type": "udp", "url": "udp://127.0.0.1:1992", "handler_options": 255 }
}

Then:

ycommand -c 'command-yuno yuno_role=logcenter command=tail lines=500'
ycommand -c 'command-yuno yuno_role=logcenter command=search text="<keyword>" maxcount=50'
ycommand -c 'command-yuno yuno_role=logcenter command=display-summary'
ycommand -c 'command-yuno yuno_role=logcenter command=reset-counters'   # wipe the rollup

11.4 Follow one request end-to-end¶

See §7. The set of commands is set-gclass-trace ... traffic, set-global-trace ... machine, set-global-trace ... fs and set-gclass-trace C_IEVENT_SRV ... ievents2. Do not forget to disable everything afterwards.

11.5 Capture an FSM bug in one gobj only¶

ycommand -c "command-yuno id=$YUNO service=__yuno__ command=set-gobj-trace gobj=<short_name> level=machine set=1"
ycommand -c "command-yuno id=$YUNO service=__yuno__ command=set-gobj-trace gobj=<short_name> level=ev_kw   set=1"
tail -F /yuneta/logs/$YUNO/*.log | grep -a '<short_name>'
# … done …
ycommand -c "command-yuno id=$YUNO service=__yuno__ command=set-gobj-trace gobj=<short_name> level=machine set=0"
ycommand -c "command-yuno id=$YUNO service=__yuno__ command=set-gobj-trace gobj=<short_name> level=ev_kw   set=0"

These traces are live-only. If you forget the disable step, a restart of the yuno clears them.

11.6 Spot the canonical “Event NOT DEFINED in state” error¶

That single string is the most common FSM failure. The parent FSM did not declare an event that a child publishes (see CLAUDE.md “CHILD vs SERVICE”). The framework logs it at LOG_ERR, and the trace settings do not change that, so:

grep -a '"msg":"Event NOT DEFINED in state"' /yuneta/logs/*/*.log

This command works on any host, and it needs no trace.

12. Code pointers¶