ksqsf/latency-profiler

latency-profiler · 23 KiB · Text Raw

EMACS HOLISTIC LATENCY PROFILER

Status: design

Initial scope: POSIX termcap (`-nw`) sessions whose terminal input and output
have file descriptors.  GUI event delivery, the Windows console, MS-DOS,
terminal-emulator rendering, compositors, and physical presentation timing are
deliberately excluded.

1. Objective

The profiler answers one question: after work becomes observable to Emacs,
where does time pass before Emacs has emitted the resulting terminal output?

The primary unit is an interaction, not a function sample.  A source becomes
observable when terminal input is reported readable or read, a timer becomes
due, process output is reported readable, or a file notification is queued.
An execution interaction begins when Emacs dispatches that source.  It ends
when synchronous dispatch finishes and the immediately associated redisplay is
completed, skipped, or superseded.  If that redisplay emits terminal bytes, the
interaction also records the final successful stdio flush.  Deferred work is a
new interaction linked to the interaction that scheduled it; it does not keep
the original interaction open indefinitely.

This operational boundary is intentionally narrower than "all causally
related work", which is not generally decidable in Emacs Lisp.  The primary
output is a latency distribution and a latency budget for each interaction
class.  CPU throughput is supporting evidence.

The profiler must:

* run continuously and silently while explicitly enabled;
* cover TTY input, the command loop, timers and process events, GC, redisplay,
  and TTY output before adding function attribution;
* retain every interaction and its aggregate semantic spans;
* optionally write an append-only, compact, process-crash-recoverable file;
* state the uncertainty and availability of every clock boundary;
* account for its own overhead and all lost records; and
* avoid recording keys, buffer contents, file names, process output, or other
  user data by default.

This instrumentation can identify work whose removal would yield a large
latency reduction.  It cannot promise that such work exists.  It measures only
the portion controlled by Emacs: terminal-input readiness through a successful
`fflush`.  That means bytes have left the C library for the kernel; it does not
mean a terminal, PTY peer, emulator, SSH client, or display has consumed or
rendered them.  `tcdrain` may be used only by an explicit diagnostic run,
because waiting for it changes the latency being measured.

2. Terms and clocks

All internal timestamps are unsigned 64-bit nanoseconds from one
profiler-private monotonic clock selected when profiling starts.  Prefer
`CLOCK_MONOTONIC_RAW` where available, then `CLOCK_MONOTONIC`, with a
platform-specific monotonic fallback.  Profiling does not change the clocks
used by normal Emacs timekeeping.  The file records the clock identity,
advertised resolution, measured read cost, and whether the clock includes
suspend time.  Wall-clock time appears only in session metadata.  An
unavailable boundary is absent, never zero.

TTY input normally has no per-byte kernel timestamp.  An optional native
observer polls the input descriptor without reading it and records the first
readable observation while the main thread may be busy.  The main thread marks
the descriptor empty only after its existing nonblocking drain finds no input;
the observer then rearms.  Closing or replacing a terminal descriptor must
first detach it from the observer, and the observer never changes descriptor
flags or consumes input.

A read batch records the last known-empty time, first observed-readable time,
and read begin/end.  Input in that batch arrived no earlier than the known-empty
time and no later than read completion.  The first-readable observation is a
tighter upper bound only when the decisive source range ends with the first
byte after an empty state; it says nothing about later bytes received while the
descriptor remained readable.  If the observer is unavailable, the main
thread's poll/read observation is a lower-fidelity fallback.

Each queued TTY byte carries a sidecar provenance token: batch ID and byte
offset.  Decoding combines tokens into a source range; function-key
translation and replay propagate that range.  Fabricated, unread, macro, and
synthetic events have explicit synthetic or unknown provenance.  The trace
never stores the bytes themselves.

Each timestamp has one of these qualities: observer-ready, main-poll-ready,
read, internal, write, flush, or inferred.  An input arrival interval carries
its width as uncertainty.  A derived latency therefore has lower and upper
bounds when exact input arrival is unavailable.

3. Causal model

Every dispatched root of work receives a monotonically increasing 64-bit
`cause_id`.  Root kinds are TTY command, synthetic command, timer, process
data, process sentinel, file notification, terminal resize, thread wakeup, and
internal maintenance.  Nested synchronous work keeps the current `cause_id`
and a `parent_span_id`.  A nested root also records `parent_cause_id`.
Deferred work records `scheduled_by_cause_id` and receives a new `cause_id`
when it eventually runs.

A key sequence can contain user think time.  It therefore records the source
range of both the first physical event and the decisive final physical event.
Command latency begins at the decisive event's arrival interval; prefix dwell
time is reported separately.  A command without physical provenance has no
input-to-command latency.

For commands, `read_char` associates the next requested redisplay with the
command that just finished.  If Emacs deliberately skips that redisplay
because input was already pending, the interaction is `superseded` and has no
input-to-flush latency.  Timer and process roots similarly own redisplay done
before the event loop waits again.  A redisplay invoked synchronously inside
any root already belongs to that root.

Each redisplay attempt gets a `redisplay_id` and names its serving cause.
The first nonempty write to the TTY stdio stream starts an `output_epoch`; a
successful stdio flush completes the greatest written epoch.  One flush may
therefore complete several coalesced attempts.  This attributes emitted work,
not arbitrary buffer mutations: an interaction with no bytes written has
completion latency but no input-to-flush latency.

4. Numbers recorded for every interaction

The following are raw observations.  Durations are derived on reload so a
future reader can correct clock mappings without rewriting the trace.

| Field | Meaning |
|-------|---------|
| `cause_id` | Stable identifier for this interaction. |
| `parent_cause_id`, `scheduled_by_cause_id` | Optional synchronous and deferred causal links. |
| `root_kind` | TTY command, synthetic command, timer, process, file notification, resize, thread wakeup, or maintenance. |
| `terminal_id`, `frame_id`, `thread_id` | TTY execution context; opaque session-local identifiers. |
| `first_source`, `decisive_source` | Batch and byte ranges for the first and decisive physical events. |
| `t_input_earliest`, `t_input_latest` | Decisive-input arrival bounds; latest uses first-readable only when the decisive source is the first byte after empty. |
| `t_input_ready` | First non-consuming readable observation for the batch, when available. |
| `t_read_begin`, `t_read_end` | TTY read interval for the decisive batch. |
| `t_queue_insert`, `t_queue_remove` | Queue interval for the decisive raw event. |
| `t_key_decode_done` | Completion of key-sequence decoding and keymap lookup. |
| `t_dispatch_begin`, `t_root_work_end` | Operational execution interval for the root. |
| `t_pre_command_begin`, `t_pre_command_end` | Pre-command hook interval. |
| `t_command_begin`, `t_command_end` | `command-execute` interval, including abnormal exit status. |
| `t_post_command_begin`, `t_post_command_end` | Post-command hook interval. |
| `first_redisplay_id`, `last_redisplay_id`, `final_output_epoch` | Associated presentation range, or zero. |
| `t_redisplay_begin`, `t_redisplay_end` | C redisplay interval serving the cause. |
| `t_output_begin`, `t_output_end` | Encoding and writes for the relevant output epoch. |
| `t_flush_begin`, `t_flush_end` | Successful stdio `fflush` interval for the output epoch. |
| `thread_cpu_begin`, `thread_cpu_end` | Main-thread CPU clock at interaction boundaries. |
| `status` | Normal, quit, error, throw, superseded, dropped, or still open. |
| `validity_mask` | Which optional timestamps and counters are present. |
| `input_uncertainty_ns` | `t_input_latest - t_input_earliest`. |

The reader derives at least these numbers:

| Metric | Definition |
|--------|------------|
| `batch_ready_wait_ns` | `t_read_begin - t_input_ready`; time the batch waited after its first readable observation. |
| `input_queue_ns` | `t_queue_remove - t_queue_insert`. |
| `key_decode_ns` | `t_key_decode_done - t_queue_remove`. |
| `pre_command_ns` | `t_pre_command_end - t_pre_command_begin`. |
| `command_ns` | `t_command_end - t_command_begin`. |
| `post_command_ns` | `t_post_command_end - t_post_command_begin`. |
| `redisplay_wait_ns` | `t_redisplay_begin - t_root_work_end`. |
| `redisplay_ns` | `t_redisplay_end - t_redisplay_begin`. |
| `tty_emit_ns` | `t_flush_end - t_output_begin`; encoding through stdio flush, not terminal rendering. |
| `input_to_command_min_ns` | `t_command_end - t_input_latest`. |
| `input_to_command_max_ns` | `t_command_end - t_input_earliest`. |
| `input_to_flush_min_ns` | `t_flush_end - t_input_latest`. |
| `input_to_flush_max_ns` | `t_flush_end - t_input_earliest`. |
| `main_cpu_ns` | `thread_cpu_end - thread_cpu_begin`. |
| `main_off_cpu_ns` | Interaction wall time minus main-thread CPU time; inclusive of nested roots. |
| `prefix_dwell_min_ns` | `max(0, decisive_earliest - first_latest)`; excluded from command latency. |
| `prefix_dwell_max_ns` | `decisive_latest - first_earliest`; excluded from command latency. |

Reports show count, minimum, median, p90, p95, p99, maximum, arithmetic mean,
total time, and input-arrival uncertainty.  They add p99.9 only when the sample
count supports it.  Means are never the only displayed latency statistic.
Interactions without terminal output and superseded interactions are separate
populations.  Session comparison adds confidence intervals only after a stable
replay workload exists.

5. Work and state recorded per interaction

The first implementation records only counters already available at the hook
or cheaply accumulated inside the measured loop:

* input queue depth at insertion and removal, read-batch byte count, decoded
  event count, key-sequence length, prefix depth, and extra reads while
  decoding;
* selected-buffer size and modification-tick delta, selected-window rows and
  columns, and window count;
* GC count and GC wall time during the interaction;
* redisplay attempts, preemptions and retry reasons, frames and windows
  considered and updated, rows generated and reused, glyphs emitted, terminal
  bytes requested, stdio calls, and flush failures; and
* profiler records emitted and dropped, ring high-water mark, writer CPU time,
  clock-read time, and measured observer cost.

Partial writes, `EAGAIN`, and kernel output-queue depth are not claimed while
TTY output uses stdio, which hides those details.  Syscall interposition,
per-object allocation accounting, hardware counters, every wait kind, and
every redisplay subphase are added only in a targeted diagnostic build after a
coarser span identifies a need.  Existing OS tools are preferred for hardware
and scheduler counters.

`SESSION` records the Emacs revision, executable and dump build IDs, configure
and optimization settings, GC settings, OS and CPU identity, the profiler
clock, TTY type/capabilities/size/coding/baud, local-device versus PTY, and
profiler settings.  It records whether an SSH-related variable is present, not
its value, and never records the terminal device path.

Symbol and feature names can reveal installed packages.  The initial file
contains no Lisp symbol or feature names.  A later command-grouping or sampling
phase may add them behind an explicit option.  Startup profiling,
loaded-feature inventories, and file-name hashes are outside the initial
subsystem.

6. Spans

Manual spans cover the semantic boundaries needed for the first report:

* TTY readiness observation, read, queue insert/remove, key decoding, and
  command loop;
* pre-command, command, and post-command;
* GC as one span;
* the central event poll; and
* redisplay decision, desired-matrix construction, matrix comparison, terminal
  encoding/write, and flush.

The profiler does not claim to observe blocking hidden in arbitrary Lisp,
modules, libraries, or syscalls.  Add a wait span at an Emacs wrapper only when
the initial waterfall leaves material unknown off-CPU time.  Timer, process,
file-notification, and resize spans arrive with their root kinds in the second
phase.

A completed span stores `span_id`, `parent_span_id`, `cause_id`, kind,
start time, wall duration, thread CPU duration, thread ID, status, detail ID,
and the counter deltas relevant to that kind.  Spans open during a crash are
reconstructed from begin records and marked incomplete.

7. Function attribution

Function attribution is not part of the first implementation.  Phase timing
comes first; controlled reruns use the existing Lisp profiler and an external
OS profiler such as `perf` for function-level evidence.  This reuses working
machinery and avoids adding a second signal source before semantic boundaries
have proved useful.

If those reruns cannot explain a repeatable slow phase, a later phase may add
one randomized on-CPU sampler.  It may reuse the timer setup in
`src/profiler.c`, but not its Lisp-object hash-table signal path.  Samples go
to a fixed numeric ring and carry the current `cause_id` and `span_id`.  A
mixed C/Lisp stack requires a separately validated shadow stack across
interpreted, byte-code, native-compiled, primitive, module, and nonlocal-exit
gateways.  The signal path performs no allocation, hashing, Lisp call, symbol
lookup, locking, or file I/O.

Off-CPU time initially comes from wall time minus thread CPU time and known
wait spans.  A second wall-clock sampling signal is justified only if material
unknown off-CPU time remains.  Compiler-wide C/Lisp function tracing is out of
scope: it changes the workload too much and duplicates targeted spans and
existing external tools.

8. C and Lisp symbols

The initial file contains no function identities.  If command grouping or
mixed-stack sampling is added later, signal records contain only
module-relative PCs and fixed Lisp frame IDs.  Symbolization happens off the
signal path using executable/shared-object build IDs and opt-in Lisp names.
Unknown frames remain explicit module-relative addresses.  A prefix-tree stack
dictionary is useful only after sample volume demonstrates that it saves more
space than it costs in writer complexity.

9. Collection and overhead control

The main thread and the optional input observer each write to a preallocated
single-producer ring.  Producers never allocate, lock, wait, symbolize, or do
file I/O.  A native writer thread drains the rings without taking the Lisp
lock, forms chunks, and writes with `writev`.  Its CPU time is reported.

Because the first phase stores aggregate interactions and spans rather than
function samples or per-glyph events, it retains every record and needs no
flight recorder.  Ring, chunk, and rotation sizes are configuration values
chosen by a burst test, not architectural constants.  The default file has no
compression and no forced `fsync`; a process crash can lose a partial current
chunk, while a machine or filesystem failure can lose more.

Any loss emits a `LOSS` record with stream, first/last time, records, bytes,
and reason.  A producer increments an out-of-ring drop counter before it
continues, so a full ring cannot hide the loss record it prevents.  Silent
loss invalidates a profile.

The acceptance budget for continuous mode is less than 1% additional CPU,
less than 1% median command latency, and less than 250 microseconds added to
p99 command latency on the reference workload.  These are gates, not presumed
facts.  The input observer is measured separately and disabled automatically
when its platform implementation cannot meet the gate.

10. File format (`.emlat`)

Integers are little-endian in fixed headers.  Record integers use unsigned or
signed LEB128.  Timestamps are deltas from the first timestamp in their block.
IDs start at one; zero means absent.  Strings are UTF-8 byte strings.  No Lisp
reader syntax or dumped Lisp object appears in the file.  IDs are stable only
within one rotated session.  Cross-session comparison uses explicit metadata,
never coincidentally equal numeric IDs.

10.1 File header

The fixed 64-byte header is:

| Offset | Size | Field |
|--------|------|-------|
| `0` | `8` | Magic bytes `EMLAT\0\r\n`. |
| `8` | `2` | Format major version. |
| `10` | `2` | Format minor version. |
| `12` | `4` | Header flags. |
| `16` | `16` | Random session UUID. |
| `32` | `8` | Rotated-file sequence number. |
| `40` | `8` | Realtime start in Unix nanoseconds. |
| `48` | `8` | Monotonic start tick in nanoseconds. |
| `56` | `4` | CRC32C of bytes `0..55`. |
| `60` | `4` | Reserved zero. |

10.2 Chunks

The remainder is a sequence of independently valid chunks, padded with zeros
to an eight-byte boundary.  A 40-byte chunk header contains:

| Offset | Size | Field |
|--------|------|-------|
| `0` | `4` | Chunk magic `CHNK`. |
| `4` | `2` | Chunk type. |
| `6` | `2` | Type-specific version. |
| `8` | `4` | Flags, including compression. |
| `12` | `4` | Header size, initially `40`. |
| `16` | `4` | Stored payload bytes. |
| `20` | `4` | Uncompressed payload bytes. |
| `24` | `8` | Monotonic chunk sequence. |
| `32` | `4` | Payload CRC32C after decompression. |
| `36` | `4` | Header CRC32C with this field zeroed. |

The core format requires no compression library.  Delta encoding, varints, and
dictionaries provide the baseline compactness.  If compression is added, a
numeric codec ID lives in the chunk flags so a reader can identify or skip the
chunk without first decoding `SESSION`.  Unsupported compression is reported
as loss.

Chunk types are:

| Type | Payload |
|------|---------|
| `SESSION` | Build/configuration, clocks, OS, CPU, TTY capabilities, profiler settings, privacy policy, and feature flags. |
| `STRING` | ID, byte length, bytes. |
| `INPUT` | Read-batch bounds and byte-provenance ranges, without byte contents. |
| `INTERACTION` | Cause links, output epochs, validity mask, raw timestamps, status, and fixed counters. |
| `SPAN` | Completed or open semantic spans and their counter deltas. |
| `LOSS` | Dropped/corrupt/unsupported data intervals. |

Dictionary definitions precede their first use.  Each record block begins with
a base timestamp, thread ID when homogeneous, record count, and schema
version.  Records then use timestamp deltas and varint IDs.  Unknown chunk
types and newer type-specific versions are skipped by chunk length and
reported, rather than guessed or misdecoded.  Sampling can add `MODULE`,
`SYMBOL`, `STACK`, and `SAMPLE` chunk types in a later format-minor version.

The writer constructs a complete chunk in memory before appending its header
and payload.  On reload, scanning stops at the first incomplete header,
impossible size, or failed CRC; every preceding chunk remains usable.  An
index can be added later if sequential scans become measurably slow.  Rotation
never reuses IDs within a session.

11. Embedded interface and reports

Profiling starts after a termcap terminal exists, through a Lisp primitive or
`--latency-profile[=FILE]`.  Lisp primitives start, stop, rotate, and query
status.  Dump loading and earlier startup are outside the initial clock.

The first loader scans chunks sequentially and materializes summaries, not one
Lisp object per record.  Memory mapping and lazy iterators are added only if
profile size makes that necessary.  The first reports are:

1. command-latency distributions, split into flushed, no-output, and
   superseded interactions;
2. a waterfall for one selected tail interaction, including input uncertainty;
3. redisplay work and emitted TTY bytes for that interaction; and
4. profiler overhead, loss, and unavailable-boundary diagnostics.

The report distinguishes wall time, main-thread CPU time, measured wait time,
unknown off-CPU time, and TTY emission time.  Function rankings and session
comparisons wait for the later attribution and replay phases.

12. Initial implementation map

The semantic recorder belongs in a small native `latency` core; it does not
modify `src/profiler.c` or install a sampling signal.  The existing Lisp CPU
and memory profiler remains unchanged.  Only a later mixed-stack sampler may
share timer setup with `src/profiler.c`.

The first semantic hooks belong at these existing funnels:

* `wait_reading_process_output` for poll start/end, ready descriptor classes,
  fallback readiness, and main-thread idle time;
* `tty_read_avail_input`, `kbd_buffer_store_event`,
  `kbd_buffer_get_event`, `read_decoded_event_from_main_queue`, and
  `read_key_sequence` for TTY read batches, queue latency, provenance folding,
  and key decoding;
* `command_loop_1` around pre-command hooks, `command-execute`, post-command
  hooks, and root completion;
* the redisplay decision in `read_char`, which is where the command's following
  redisplay is actually performed or skipped;
* `garbage_collect` for GC attribution;
* `redisplay_internal`, `redisplay_window`, `display_line`, `update_frame`,
  and `update_window` for redisplay decisions and work volume;
* one small TTY-output wrapper used by the glyph `fwrite` sites and `cmputc`,
  plus one flush wrapper used by `tty_update_end`, `flush_terminal`, and the
  other normal redisplay flush paths, for byte counts and stdio-flush latency.

Timer callbacks, process filters/sentinels, file notifications, and resize
roots are the next phase after the command path meets its overhead budget.
The input observer follows only if main-thread input intervals are too wide to
answer the latency question.  Mixed-stack sampling follows only if semantic
spans plus controlled `profiler.el`/OS-profiler reruns leave a repeatable tail
unexplained.

The native core exposes inline, allocation-free producer operations.  Lisp
report code is loaded only when a profile is opened.

13. Validation

Before optimization work uses these files, the first phase needs:

* round-trip tests for each initial record and dictionary type;
* truncated, corrupted, unknown-version, and rotated-file tests;
* fuzzing of the loader with strict allocation and record-count limits;
* synthetic CPU, sleep, GC, redisplay, error, quit, and nonlocal-exit commands
  with known durations and causal relationships;
* tests that force ring overflow and verify explicit `LOSS` records;
* PTY tests that control read batching, queueing, redisplay, terminal output,
  backpressure, and stdio-flush timing;
* privacy tests that scan default traces for injected keys, buffer text, file
  names, process output, terminal paths, and non-dumped symbol names; and
* A/B runs of an identical replay workload to enforce the continuous-mode
  budget.  The observer gets a separate A/B gate when added.

Only after these checks should profile results choose optimization targets.
The first target is the largest repeatable component of p95/p99
input-to-flush latency, not the hottest function in an average CPU profile.

1	EMACS HOLISTIC LATENCY PROFILER
2
3	Status: design
4
5	Initial scope: POSIX termcap (`-nw`) sessions whose terminal input and output
6	have file descriptors. GUI event delivery, the Windows console, MS-DOS,
7	terminal-emulator rendering, compositors, and physical presentation timing are
8	deliberately excluded.
9
10	1. Objective
11
12	The profiler answers one question: after work becomes observable to Emacs,
13	where does time pass before Emacs has emitted the resulting terminal output?
14
15	The primary unit is an interaction, not a function sample. A source becomes
16	observable when terminal input is reported readable or read, a timer becomes
17	due, process output is reported readable, or a file notification is queued.
18	An execution interaction begins when Emacs dispatches that source. It ends
19	when synchronous dispatch finishes and the immediately associated redisplay is
20	completed, skipped, or superseded. If that redisplay emits terminal bytes, the
21	interaction also records the final successful stdio flush. Deferred work is a
22	new interaction linked to the interaction that scheduled it; it does not keep
23	the original interaction open indefinitely.
24
25	This operational boundary is intentionally narrower than "all causally
26	related work", which is not generally decidable in Emacs Lisp. The primary
27	output is a latency distribution and a latency budget for each interaction
28	class. CPU throughput is supporting evidence.
29
30	The profiler must:
31
32	* run continuously and silently while explicitly enabled;
33	* cover TTY input, the command loop, timers and process events, GC, redisplay,
34	and TTY output before adding function attribution;
35	* retain every interaction and its aggregate semantic spans;
36	* optionally write an append-only, compact, process-crash-recoverable file;
37	* state the uncertainty and availability of every clock boundary;
38	* account for its own overhead and all lost records; and
39	* avoid recording keys, buffer contents, file names, process output, or other
40	user data by default.
41
42	This instrumentation can identify work whose removal would yield a large
43	latency reduction. It cannot promise that such work exists. It measures only
44	the portion controlled by Emacs: terminal-input readiness through a successful
45	`fflush`. That means bytes have left the C library for the kernel; it does not
46	mean a terminal, PTY peer, emulator, SSH client, or display has consumed or
47	rendered them. `tcdrain` may be used only by an explicit diagnostic run,
48	because waiting for it changes the latency being measured.
49
50	2. Terms and clocks
51
52	All internal timestamps are unsigned 64-bit nanoseconds from one
53	profiler-private monotonic clock selected when profiling starts. Prefer
54	`CLOCK_MONOTONIC_RAW` where available, then `CLOCK_MONOTONIC`, with a
55	platform-specific monotonic fallback. Profiling does not change the clocks
56	used by normal Emacs timekeeping. The file records the clock identity,
57	advertised resolution, measured read cost, and whether the clock includes
58	suspend time. Wall-clock time appears only in session metadata. An
59	unavailable boundary is absent, never zero.
60
61	TTY input normally has no per-byte kernel timestamp. An optional native
62	observer polls the input descriptor without reading it and records the first
63	readable observation while the main thread may be busy. The main thread marks
64	the descriptor empty only after its existing nonblocking drain finds no input;
65	the observer then rearms. Closing or replacing a terminal descriptor must
66	first detach it from the observer, and the observer never changes descriptor
67	flags or consumes input.
68
69	A read batch records the last known-empty time, first observed-readable time,
70	and read begin/end. Input in that batch arrived no earlier than the known-empty
71	time and no later than read completion. The first-readable observation is a
72	tighter upper bound only when the decisive source range ends with the first
73	byte after an empty state; it says nothing about later bytes received while the
74	descriptor remained readable. If the observer is unavailable, the main
75	thread's poll/read observation is a lower-fidelity fallback.
76
77	Each queued TTY byte carries a sidecar provenance token: batch ID and byte
78	offset. Decoding combines tokens into a source range; function-key
79	translation and replay propagate that range. Fabricated, unread, macro, and
80	synthetic events have explicit synthetic or unknown provenance. The trace
81	never stores the bytes themselves.
82
83	Each timestamp has one of these qualities: observer-ready, main-poll-ready,
84	read, internal, write, flush, or inferred. An input arrival interval carries
85	its width as uncertainty. A derived latency therefore has lower and upper
86	bounds when exact input arrival is unavailable.
87
88	3. Causal model
89
90	Every dispatched root of work receives a monotonically increasing 64-bit
91	`cause_id`. Root kinds are TTY command, synthetic command, timer, process
92	data, process sentinel, file notification, terminal resize, thread wakeup, and
93	internal maintenance. Nested synchronous work keeps the current `cause_id`
94	and a `parent_span_id`. A nested root also records `parent_cause_id`.
95	Deferred work records `scheduled_by_cause_id` and receives a new `cause_id`
96	when it eventually runs.
97
98	A key sequence can contain user think time. It therefore records the source
99	range of both the first physical event and the decisive final physical event.
100	Command latency begins at the decisive event's arrival interval; prefix dwell
101	time is reported separately. A command without physical provenance has no
102	input-to-command latency.
103
104	For commands, `read_char` associates the next requested redisplay with the
105	command that just finished. If Emacs deliberately skips that redisplay
106	because input was already pending, the interaction is `superseded` and has no
107	input-to-flush latency. Timer and process roots similarly own redisplay done
108	before the event loop waits again. A redisplay invoked synchronously inside
109	any root already belongs to that root.
110
111	Each redisplay attempt gets a `redisplay_id` and names its serving cause.
112	The first nonempty write to the TTY stdio stream starts an `output_epoch`; a
113	successful stdio flush completes the greatest written epoch. One flush may
114	therefore complete several coalesced attempts. This attributes emitted work,
115	not arbitrary buffer mutations: an interaction with no bytes written has
116	completion latency but no input-to-flush latency.
117
118	4. Numbers recorded for every interaction
119
120	The following are raw observations. Durations are derived on reload so a
121	future reader can correct clock mappings without rewriting the trace.
122
123	\| Field \| Meaning \|
124	\|-------\|---------\|
125	\| `cause_id` \| Stable identifier for this interaction. \|
126	\| `parent_cause_id`, `scheduled_by_cause_id` \| Optional synchronous and deferred causal links. \|
127	\| `root_kind` \| TTY command, synthetic command, timer, process, file notification, resize, thread wakeup, or maintenance. \|
128	\| `terminal_id`, `frame_id`, `thread_id` \| TTY execution context; opaque session-local identifiers. \|
129	\| `first_source`, `decisive_source` \| Batch and byte ranges for the first and decisive physical events. \|
130	\| `t_input_earliest`, `t_input_latest` \| Decisive-input arrival bounds; latest uses first-readable only when the decisive source is the first byte after empty. \|
131	\| `t_input_ready` \| First non-consuming readable observation for the batch, when available. \|
132	\| `t_read_begin`, `t_read_end` \| TTY read interval for the decisive batch. \|
133	\| `t_queue_insert`, `t_queue_remove` \| Queue interval for the decisive raw event. \|
134	\| `t_key_decode_done` \| Completion of key-sequence decoding and keymap lookup. \|
135	\| `t_dispatch_begin`, `t_root_work_end` \| Operational execution interval for the root. \|
136	\| `t_pre_command_begin`, `t_pre_command_end` \| Pre-command hook interval. \|
137	\| `t_command_begin`, `t_command_end` \| `command-execute` interval, including abnormal exit status. \|
138	\| `t_post_command_begin`, `t_post_command_end` \| Post-command hook interval. \|
139	\| `first_redisplay_id`, `last_redisplay_id`, `final_output_epoch` \| Associated presentation range, or zero. \|
140	\| `t_redisplay_begin`, `t_redisplay_end` \| C redisplay interval serving the cause. \|
141	\| `t_output_begin`, `t_output_end` \| Encoding and writes for the relevant output epoch. \|
142	\| `t_flush_begin`, `t_flush_end` \| Successful stdio `fflush` interval for the output epoch. \|
143	\| `thread_cpu_begin`, `thread_cpu_end` \| Main-thread CPU clock at interaction boundaries. \|
144	\| `status` \| Normal, quit, error, throw, superseded, dropped, or still open. \|
145	\| `validity_mask` \| Which optional timestamps and counters are present. \|
146	\| `input_uncertainty_ns` \| `t_input_latest - t_input_earliest`. \|
147
148	The reader derives at least these numbers:
149
150	\| Metric \| Definition \|
151	\|--------\|------------\|
152	\| `batch_ready_wait_ns` \| `t_read_begin - t_input_ready`; time the batch waited after its first readable observation. \|
153	\| `input_queue_ns` \| `t_queue_remove - t_queue_insert`. \|
154	\| `key_decode_ns` \| `t_key_decode_done - t_queue_remove`. \|
155	\| `pre_command_ns` \| `t_pre_command_end - t_pre_command_begin`. \|
156	\| `command_ns` \| `t_command_end - t_command_begin`. \|
157	\| `post_command_ns` \| `t_post_command_end - t_post_command_begin`. \|
158	\| `redisplay_wait_ns` \| `t_redisplay_begin - t_root_work_end`. \|
159	\| `redisplay_ns` \| `t_redisplay_end - t_redisplay_begin`. \|
160	\| `tty_emit_ns` \| `t_flush_end - t_output_begin`; encoding through stdio flush, not terminal rendering. \|
161	\| `input_to_command_min_ns` \| `t_command_end - t_input_latest`. \|
162	\| `input_to_command_max_ns` \| `t_command_end - t_input_earliest`. \|
163	\| `input_to_flush_min_ns` \| `t_flush_end - t_input_latest`. \|
164	\| `input_to_flush_max_ns` \| `t_flush_end - t_input_earliest`. \|
165	\| `main_cpu_ns` \| `thread_cpu_end - thread_cpu_begin`. \|
166	\| `main_off_cpu_ns` \| Interaction wall time minus main-thread CPU time; inclusive of nested roots. \|
167	\| `prefix_dwell_min_ns` \| `max(0, decisive_earliest - first_latest)`; excluded from command latency. \|
168	\| `prefix_dwell_max_ns` \| `decisive_latest - first_earliest`; excluded from command latency. \|
169
170	Reports show count, minimum, median, p90, p95, p99, maximum, arithmetic mean,
171	total time, and input-arrival uncertainty. They add p99.9 only when the sample
172	count supports it. Means are never the only displayed latency statistic.
173	Interactions without terminal output and superseded interactions are separate
174	populations. Session comparison adds confidence intervals only after a stable
175	replay workload exists.
176
177	5. Work and state recorded per interaction
178
179	The first implementation records only counters already available at the hook
180	or cheaply accumulated inside the measured loop:
181
182	* input queue depth at insertion and removal, read-batch byte count, decoded
183	event count, key-sequence length, prefix depth, and extra reads while
184	decoding;
185	* selected-buffer size and modification-tick delta, selected-window rows and
186	columns, and window count;
187	* GC count and GC wall time during the interaction;
188	* redisplay attempts, preemptions and retry reasons, frames and windows
189	considered and updated, rows generated and reused, glyphs emitted, terminal
190	bytes requested, stdio calls, and flush failures; and
191	* profiler records emitted and dropped, ring high-water mark, writer CPU time,
192	clock-read time, and measured observer cost.
193
194	Partial writes, `EAGAIN`, and kernel output-queue depth are not claimed while
195	TTY output uses stdio, which hides those details. Syscall interposition,
196	per-object allocation accounting, hardware counters, every wait kind, and
197	every redisplay subphase are added only in a targeted diagnostic build after a
198	coarser span identifies a need. Existing OS tools are preferred for hardware
199	and scheduler counters.
200
201	`SESSION` records the Emacs revision, executable and dump build IDs, configure
202	and optimization settings, GC settings, OS and CPU identity, the profiler
203	clock, TTY type/capabilities/size/coding/baud, local-device versus PTY, and
204	profiler settings. It records whether an SSH-related variable is present, not
205	its value, and never records the terminal device path.
206
207	Symbol and feature names can reveal installed packages. The initial file
208	contains no Lisp symbol or feature names. A later command-grouping or sampling
209	phase may add them behind an explicit option. Startup profiling,
210	loaded-feature inventories, and file-name hashes are outside the initial
211	subsystem.
212
213	6. Spans
214
215	Manual spans cover the semantic boundaries needed for the first report:
216
217	* TTY readiness observation, read, queue insert/remove, key decoding, and
218	command loop;
219	* pre-command, command, and post-command;
220	* GC as one span;
221	* the central event poll; and
222	* redisplay decision, desired-matrix construction, matrix comparison, terminal
223	encoding/write, and flush.
224
225	The profiler does not claim to observe blocking hidden in arbitrary Lisp,
226	modules, libraries, or syscalls. Add a wait span at an Emacs wrapper only when
227	the initial waterfall leaves material unknown off-CPU time. Timer, process,
228	file-notification, and resize spans arrive with their root kinds in the second
229	phase.
230
231	A completed span stores `span_id`, `parent_span_id`, `cause_id`, kind,
232	start time, wall duration, thread CPU duration, thread ID, status, detail ID,
233	and the counter deltas relevant to that kind. Spans open during a crash are
234	reconstructed from begin records and marked incomplete.
235
236	7. Function attribution
237
238	Function attribution is not part of the first implementation. Phase timing
239	comes first; controlled reruns use the existing Lisp profiler and an external
240	OS profiler such as `perf` for function-level evidence. This reuses working
241	machinery and avoids adding a second signal source before semantic boundaries
242	have proved useful.
243
244	If those reruns cannot explain a repeatable slow phase, a later phase may add
245	one randomized on-CPU sampler. It may reuse the timer setup in
246	`src/profiler.c`, but not its Lisp-object hash-table signal path. Samples go
247	to a fixed numeric ring and carry the current `cause_id` and `span_id`. A
248	mixed C/Lisp stack requires a separately validated shadow stack across
249	interpreted, byte-code, native-compiled, primitive, module, and nonlocal-exit
250	gateways. The signal path performs no allocation, hashing, Lisp call, symbol
251	lookup, locking, or file I/O.
252
253	Off-CPU time initially comes from wall time minus thread CPU time and known
254	wait spans. A second wall-clock sampling signal is justified only if material
255	unknown off-CPU time remains. Compiler-wide C/Lisp function tracing is out of
256	scope: it changes the workload too much and duplicates targeted spans and
257	existing external tools.
258
259	8. C and Lisp symbols
260
261	The initial file contains no function identities. If command grouping or
262	mixed-stack sampling is added later, signal records contain only
263	module-relative PCs and fixed Lisp frame IDs. Symbolization happens off the
264	signal path using executable/shared-object build IDs and opt-in Lisp names.
265	Unknown frames remain explicit module-relative addresses. A prefix-tree stack
266	dictionary is useful only after sample volume demonstrates that it saves more
267	space than it costs in writer complexity.
268
269	9. Collection and overhead control
270
271	The main thread and the optional input observer each write to a preallocated
272	single-producer ring. Producers never allocate, lock, wait, symbolize, or do
273	file I/O. A native writer thread drains the rings without taking the Lisp
274	lock, forms chunks, and writes with `writev`. Its CPU time is reported.
275
276	Because the first phase stores aggregate interactions and spans rather than
277	function samples or per-glyph events, it retains every record and needs no
278	flight recorder. Ring, chunk, and rotation sizes are configuration values
279	chosen by a burst test, not architectural constants. The default file has no
280	compression and no forced `fsync`; a process crash can lose a partial current
281	chunk, while a machine or filesystem failure can lose more.
282
283	Any loss emits a `LOSS` record with stream, first/last time, records, bytes,
284	and reason. A producer increments an out-of-ring drop counter before it
285	continues, so a full ring cannot hide the loss record it prevents. Silent
286	loss invalidates a profile.
287
288	The acceptance budget for continuous mode is less than 1% additional CPU,
289	less than 1% median command latency, and less than 250 microseconds added to
290	p99 command latency on the reference workload. These are gates, not presumed
291	facts. The input observer is measured separately and disabled automatically
292	when its platform implementation cannot meet the gate.
293
294	10. File format (`.emlat`)
295
296	Integers are little-endian in fixed headers. Record integers use unsigned or
297	signed LEB128. Timestamps are deltas from the first timestamp in their block.
298	IDs start at one; zero means absent. Strings are UTF-8 byte strings. No Lisp
299	reader syntax or dumped Lisp object appears in the file. IDs are stable only
300	within one rotated session. Cross-session comparison uses explicit metadata,
301	never coincidentally equal numeric IDs.
302
303	10.1 File header
304
305	The fixed 64-byte header is:
306
307	\| Offset \| Size \| Field \|
308	\|--------\|------\|-------\|
309	\| `0` \| `8` \| Magic bytes `EMLAT\0\r\n`. \|
310	\| `8` \| `2` \| Format major version. \|
311	\| `10` \| `2` \| Format minor version. \|
312	\| `12` \| `4` \| Header flags. \|
313	\| `16` \| `16` \| Random session UUID. \|
314	\| `32` \| `8` \| Rotated-file sequence number. \|
315	\| `40` \| `8` \| Realtime start in Unix nanoseconds. \|
316	\| `48` \| `8` \| Monotonic start tick in nanoseconds. \|
317	\| `56` \| `4` \| CRC32C of bytes `0..55`. \|
318	\| `60` \| `4` \| Reserved zero. \|
319
320	10.2 Chunks
321
322	The remainder is a sequence of independently valid chunks, padded with zeros
323	to an eight-byte boundary. A 40-byte chunk header contains:
324
325	\| Offset \| Size \| Field \|
326	\|--------\|------\|-------\|
327	\| `0` \| `4` \| Chunk magic `CHNK`. \|
328	\| `4` \| `2` \| Chunk type. \|
329	\| `6` \| `2` \| Type-specific version. \|
330	\| `8` \| `4` \| Flags, including compression. \|
331	\| `12` \| `4` \| Header size, initially `40`. \|
332	\| `16` \| `4` \| Stored payload bytes. \|
333	\| `20` \| `4` \| Uncompressed payload bytes. \|
334	\| `24` \| `8` \| Monotonic chunk sequence. \|
335	\| `32` \| `4` \| Payload CRC32C after decompression. \|
336	\| `36` \| `4` \| Header CRC32C with this field zeroed. \|
337
338	The core format requires no compression library. Delta encoding, varints, and
339	dictionaries provide the baseline compactness. If compression is added, a
340	numeric codec ID lives in the chunk flags so a reader can identify or skip the
341	chunk without first decoding `SESSION`. Unsupported compression is reported
342	as loss.
343
344	Chunk types are:
345
346	\| Type \| Payload \|
347	\|------\|---------\|
348	\| `SESSION` \| Build/configuration, clocks, OS, CPU, TTY capabilities, profiler settings, privacy policy, and feature flags. \|
349	\| `STRING` \| ID, byte length, bytes. \|
350	\| `INPUT` \| Read-batch bounds and byte-provenance ranges, without byte contents. \|
351	\| `INTERACTION` \| Cause links, output epochs, validity mask, raw timestamps, status, and fixed counters. \|
352	\| `SPAN` \| Completed or open semantic spans and their counter deltas. \|
353	\| `LOSS` \| Dropped/corrupt/unsupported data intervals. \|
354
355	Dictionary definitions precede their first use. Each record block begins with
356	a base timestamp, thread ID when homogeneous, record count, and schema
357	version. Records then use timestamp deltas and varint IDs. Unknown chunk
358	types and newer type-specific versions are skipped by chunk length and
359	reported, rather than guessed or misdecoded. Sampling can add `MODULE`,
360	`SYMBOL`, `STACK`, and `SAMPLE` chunk types in a later format-minor version.
361
362	The writer constructs a complete chunk in memory before appending its header
363	and payload. On reload, scanning stops at the first incomplete header,
364	impossible size, or failed CRC; every preceding chunk remains usable. An
365	index can be added later if sequential scans become measurably slow. Rotation
366	never reuses IDs within a session.
367
368	11. Embedded interface and reports
369
370	Profiling starts after a termcap terminal exists, through a Lisp primitive or
371	`--latency-profile[=FILE]`. Lisp primitives start, stop, rotate, and query
372	status. Dump loading and earlier startup are outside the initial clock.
373
374	The first loader scans chunks sequentially and materializes summaries, not one
375	Lisp object per record. Memory mapping and lazy iterators are added only if
376	profile size makes that necessary. The first reports are:
377
378	1. command-latency distributions, split into flushed, no-output, and
379	superseded interactions;
380	2. a waterfall for one selected tail interaction, including input uncertainty;
381	3. redisplay work and emitted TTY bytes for that interaction; and
382	4. profiler overhead, loss, and unavailable-boundary diagnostics.
383
384	The report distinguishes wall time, main-thread CPU time, measured wait time,
385	unknown off-CPU time, and TTY emission time. Function rankings and session
386	comparisons wait for the later attribution and replay phases.
387
388	12. Initial implementation map
389
390	The semantic recorder belongs in a small native `latency` core; it does not
391	modify `src/profiler.c` or install a sampling signal. The existing Lisp CPU
392	and memory profiler remains unchanged. Only a later mixed-stack sampler may
393	share timer setup with `src/profiler.c`.
394
395	The first semantic hooks belong at these existing funnels:
396
397	* `wait_reading_process_output` for poll start/end, ready descriptor classes,
398	fallback readiness, and main-thread idle time;
399	* `tty_read_avail_input`, `kbd_buffer_store_event`,
400	`kbd_buffer_get_event`, `read_decoded_event_from_main_queue`, and
401	`read_key_sequence` for TTY read batches, queue latency, provenance folding,
402	and key decoding;
403	* `command_loop_1` around pre-command hooks, `command-execute`, post-command
404	hooks, and root completion;
405	* the redisplay decision in `read_char`, which is where the command's following
406	redisplay is actually performed or skipped;
407	* `garbage_collect` for GC attribution;
408	* `redisplay_internal`, `redisplay_window`, `display_line`, `update_frame`,
409	and `update_window` for redisplay decisions and work volume;
410	* one small TTY-output wrapper used by the glyph `fwrite` sites and `cmputc`,
411	plus one flush wrapper used by `tty_update_end`, `flush_terminal`, and the
412	other normal redisplay flush paths, for byte counts and stdio-flush latency.
413
414	Timer callbacks, process filters/sentinels, file notifications, and resize
415	roots are the next phase after the command path meets its overhead budget.
416	The input observer follows only if main-thread input intervals are too wide to
417	answer the latency question. Mixed-stack sampling follows only if semantic
418	spans plus controlled `profiler.el`/OS-profiler reruns leave a repeatable tail
419	unexplained.
420
421	The native core exposes inline, allocation-free producer operations. Lisp
422	report code is loaded only when a profile is opened.
423
424	13. Validation
425
426	Before optimization work uses these files, the first phase needs:
427
428	* round-trip tests for each initial record and dictionary type;
429	* truncated, corrupted, unknown-version, and rotated-file tests;
430	* fuzzing of the loader with strict allocation and record-count limits;
431	* synthetic CPU, sleep, GC, redisplay, error, quit, and nonlocal-exit commands
432	with known durations and causal relationships;
433	* tests that force ring overflow and verify explicit `LOSS` records;
434	* PTY tests that control read batching, queueing, redisplay, terminal output,
435	backpressure, and stdio-flush timing;
436	* privacy tests that scan default traces for injected keys, buffer text, file
437	names, process output, terminal paths, and non-dumped symbol names; and
438	* A/B runs of an identical replay workload to enforce the continuous-mode
439	budget. The observer gets a separate A/B gate when added.
440
441	Only after these checks should profile results choose optimization targets.
442	The first target is the largest repeatable component of p95/p99
443	input-to-flush latency, not the hottest function in an average CPU profile.
444