EMACS HOLISTIC LATENCY PROFILER

Status: design

Initial scope: POSIX termcap (`-nw`) sessions whose terminal input and output
have file descriptors.  GUI event delivery, the Windows console, MS-DOS,
terminal-emulator rendering, compositors, and physical presentation timing are
deliberately excluded.

1. Objective

The profiler answers one question: after work becomes observable to Emacs,
where does time pass before Emacs has emitted the resulting terminal output?

The primary unit is an interaction, not a function sample.  A source becomes
observable when terminal input is reported readable or read, a timer becomes
due, process output is reported readable, or a file notification is queued.
An execution interaction begins when Emacs dispatches that source.  It ends
when synchronous dispatch finishes and the immediately associated redisplay is
completed, skipped, or superseded.  If that redisplay emits terminal bytes, the
interaction also records the final successful stdio flush.  Deferred work is a
new interaction linked to the interaction that scheduled it; it does not keep
the original interaction open indefinitely.

This operational boundary is intentionally narrower than "all causally
related work", which is not generally decidable in Emacs Lisp.  The primary
output is a latency distribution and a latency budget for each interaction
class.  CPU throughput is supporting evidence.

The profiler must:

* run continuously and silently while explicitly enabled;
* cover TTY input, the command loop, timers and process events, GC, redisplay,
  and TTY output before adding function attribution;
* retain every interaction and its aggregate semantic spans;
* optionally write an append-only, compact, process-crash-recoverable file;
* state the uncertainty and availability of every clock boundary;
* account for its own overhead and all lost records; and
* avoid recording keys, buffer contents, file names, process output, or other
  user data by default.

This instrumentation can identify work whose removal would yield a large
latency reduction.  It cannot promise that such work exists.  It measures only
the portion controlled by Emacs: terminal-input readiness through a successful
`fflush`.  That means bytes have left the C library for the kernel; it does not
mean a terminal, PTY peer, emulator, SSH client, or display has consumed or
rendered them.  `tcdrain` may be used only by an explicit diagnostic run,
because waiting for it changes the latency being measured.

2. Terms and clocks

All internal timestamps are unsigned 64-bit nanoseconds from one
profiler-private monotonic clock selected when profiling starts.  Prefer
`CLOCK_MONOTONIC_RAW` where available, then `CLOCK_MONOTONIC`, with a
platform-specific monotonic fallback.  Profiling does not change the clocks
used by normal Emacs timekeeping.  The file records the clock identity,
advertised resolution, measured read cost, and whether the clock includes
suspend time.  Wall-clock time appears only in session metadata.  An
unavailable boundary is absent, never zero.

TTY input normally has no per-byte kernel timestamp.  An optional native
observer polls the input descriptor without reading it and records the first
readable observation while the main thread may be busy.  The main thread marks
the descriptor empty only after its existing nonblocking drain finds no input;
the observer then rearms.  Closing or replacing a terminal descriptor must
first detach it from the observer, and the observer never changes descriptor
flags or consumes input.

A read batch records the last known-empty time, first observed-readable time,
and read begin/end.  Input in that batch arrived no earlier than the known-empty
time and no later than read completion.  The first-readable observation is a
tighter upper bound only when the decisive source range ends with the first
byte after an empty state; it says nothing about later bytes received while the
descriptor remained readable.  If the observer is unavailable, the main
thread's poll/read observation is a lower-fidelity fallback.

Each queued TTY byte carries a sidecar provenance token: batch ID and byte
offset.  Decoding combines tokens into a source range; function-key
translation and replay propagate that range.  Fabricated, unread, macro, and
synthetic events have explicit synthetic or unknown provenance.  The trace
never stores the bytes themselves.

Each timestamp has one of these qualities: observer-ready, main-poll-ready,
read, internal, write, flush, or inferred.  An input arrival interval carries
its width as uncertainty.  A derived latency therefore has lower and upper
bounds when exact input arrival is unavailable.

3. Causal model

Every dispatched root of work receives a monotonically increasing 64-bit
`cause_id`.  Root kinds are TTY command, synthetic command, timer, process
data, process sentinel, file notification, terminal resize, thread wakeup, and
internal maintenance.  Nested synchronous work keeps the current `cause_id`
and a `parent_span_id`.  A nested root also records `parent_cause_id`.
Deferred work records `scheduled_by_cause_id` and receives a new `cause_id`
when it eventually runs.

A key sequence can contain user think time.  It therefore records the source
range of both the first physical event and the decisive final physical event.
Command latency begins at the decisive event's arrival interval; prefix dwell
time is reported separately.  A command without physical provenance has no
input-to-command latency.

For commands, `read_char` associates the next requested redisplay with the
command that just finished.  If Emacs deliberately skips that redisplay
because input was already pending, the interaction is `superseded` and has no
input-to-flush latency.  Timer and process roots similarly own redisplay done
before the event loop waits again.  A redisplay invoked synchronously inside
any root already belongs to that root.

Each redisplay attempt gets a `redisplay_id` and names its serving cause.
The first nonempty write to the TTY stdio stream starts an `output_epoch`; a
successful stdio flush completes the greatest written epoch.  One flush may
therefore complete several coalesced attempts.  This attributes emitted work,
not arbitrary buffer mutations: an interaction with no bytes written has
completion latency but no input-to-flush latency.

4. Numbers recorded for every interaction

The following are raw observations.  Durations are derived on reload so a
future reader can correct clock mappings without rewriting the trace.

| Field | Meaning |
|-------|---------|
| `cause_id` | Stable identifier for this interaction. |
| `parent_cause_id`, `scheduled_by_cause_id` | Optional synchronous and deferred causal links. |
| `root_kind` | TTY command, synthetic command, timer, process, file notification, resize, thread wakeup, or maintenance. |
| `terminal_id`, `frame_id`, `thread_id` | TTY execution context; opaque session-local identifiers. |
| `first_source`, `decisive_source` | Batch and byte ranges for the first and decisive physical events. |
| `t_input_earliest`, `t_input_latest` | Decisive-input arrival bounds; latest uses first-readable only when the decisive source is the first byte after empty. |
| `t_input_ready` | First non-consuming readable observation for the batch, when available. |
| `t_read_begin`, `t_read_end` | TTY read interval for the decisive batch. |
| `t_queue_insert`, `t_queue_remove` | Queue interval for the decisive raw event. |
| `t_key_decode_done` | Completion of key-sequence decoding and keymap lookup. |
| `t_dispatch_begin`, `t_root_work_end` | Operational execution interval for the root. |
| `t_pre_command_begin`, `t_pre_command_end` | Pre-command hook interval. |
| `t_command_begin`, `t_command_end` | `command-execute` interval, including abnormal exit status. |
| `t_post_command_begin`, `t_post_command_end` | Post-command hook interval. |
| `first_redisplay_id`, `last_redisplay_id`, `final_output_epoch` | Associated presentation range, or zero. |
| `t_redisplay_begin`, `t_redisplay_end` | C redisplay interval serving the cause. |
| `t_output_begin`, `t_output_end` | Encoding and writes for the relevant output epoch. |
| `t_flush_begin`, `t_flush_end` | Successful stdio `fflush` interval for the output epoch. |
| `thread_cpu_begin`, `thread_cpu_end` | Main-thread CPU clock at interaction boundaries. |
| `status` | Normal, quit, error, throw, superseded, dropped, or still open. |
| `validity_mask` | Which optional timestamps and counters are present. |
| `input_uncertainty_ns` | `t_input_latest - t_input_earliest`. |

The reader derives at least these numbers:

| Metric | Definition |
|--------|------------|
| `batch_ready_wait_ns` | `t_read_begin - t_input_ready`; time the batch waited after its first readable observation. |
| `input_queue_ns` | `t_queue_remove - t_queue_insert`. |
| `key_decode_ns` | `t_key_decode_done - t_queue_remove`. |
| `pre_command_ns` | `t_pre_command_end - t_pre_command_begin`. |
| `command_ns` | `t_command_end - t_command_begin`. |
| `post_command_ns` | `t_post_command_end - t_post_command_begin`. |
| `redisplay_wait_ns` | `t_redisplay_begin - t_root_work_end`. |
| `redisplay_ns` | `t_redisplay_end - t_redisplay_begin`. |
| `tty_emit_ns` | `t_flush_end - t_output_begin`; encoding through stdio flush, not terminal rendering. |
| `input_to_command_min_ns` | `t_command_end - t_input_latest`. |
| `input_to_command_max_ns` | `t_command_end - t_input_earliest`. |
| `input_to_flush_min_ns` | `t_flush_end - t_input_latest`. |
| `input_to_flush_max_ns` | `t_flush_end - t_input_earliest`. |
| `main_cpu_ns` | `thread_cpu_end - thread_cpu_begin`. |
| `main_off_cpu_ns` | Interaction wall time minus main-thread CPU time; inclusive of nested roots. |
| `prefix_dwell_min_ns` | `max(0, decisive_earliest - first_latest)`; excluded from command latency. |
| `prefix_dwell_max_ns` | `decisive_latest - first_earliest`; excluded from command latency. |

Reports show count, minimum, median, p90, p95, p99, maximum, arithmetic mean,
total time, and input-arrival uncertainty.  They add p99.9 only when the sample
count supports it.  Means are never the only displayed latency statistic.
Interactions without terminal output and superseded interactions are separate
populations.  Session comparison adds confidence intervals only after a stable
replay workload exists.

5. Work and state recorded per interaction

The first implementation records only counters already available at the hook
or cheaply accumulated inside the measured loop:

* input queue depth at insertion and removal, read-batch byte count, decoded
  event count, key-sequence length, prefix depth, and extra reads while
  decoding;
* selected-buffer size and modification-tick delta, selected-window rows and
  columns, and window count;
* GC count and GC wall time during the interaction;
* redisplay attempts, preemptions and retry reasons, frames and windows
  considered and updated, rows generated and reused, glyphs emitted, terminal
  bytes requested, stdio calls, and flush failures; and
* profiler records emitted and dropped, ring high-water mark, writer CPU time,
  clock-read time, and measured observer cost.

Partial writes, `EAGAIN`, and kernel output-queue depth are not claimed while
TTY output uses stdio, which hides those details.  Syscall interposition,
per-object allocation accounting, hardware counters, every wait kind, and
every redisplay subphase are added only in a targeted diagnostic build after a
coarser span identifies a need.  Existing OS tools are preferred for hardware
and scheduler counters.

`SESSION` records the Emacs revision, executable and dump build IDs, configure
and optimization settings, GC settings, OS and CPU identity, the profiler
clock, TTY type/capabilities/size/coding/baud, local-device versus PTY, and
profiler settings.  It records whether an SSH-related variable is present, not
its value, and never records the terminal device path.

Symbol and feature names can reveal installed packages.  The initial file
contains no Lisp symbol or feature names.  A later command-grouping or sampling
phase may add them behind an explicit option.  Startup profiling,
loaded-feature inventories, and file-name hashes are outside the initial
subsystem.

6. Spans

Manual spans cover the semantic boundaries needed for the first report:

* TTY readiness observation, read, queue insert/remove, key decoding, and
  command loop;
* pre-command, command, and post-command;
* GC as one span;
* the central event poll; and
* redisplay decision, desired-matrix construction, matrix comparison, terminal
  encoding/write, and flush.

The profiler does not claim to observe blocking hidden in arbitrary Lisp,
modules, libraries, or syscalls.  Add a wait span at an Emacs wrapper only when
the initial waterfall leaves material unknown off-CPU time.  Timer, process,
file-notification, and resize spans arrive with their root kinds in the second
phase.

A completed span stores `span_id`, `parent_span_id`, `cause_id`, kind,
start time, wall duration, thread CPU duration, thread ID, status, detail ID,
and the counter deltas relevant to that kind.  Spans open during a crash are
reconstructed from begin records and marked incomplete.

7. Function attribution

Function attribution is not part of the first implementation.  Phase timing
comes first; controlled reruns use the existing Lisp profiler and an external
OS profiler such as `perf` for function-level evidence.  This reuses working
machinery and avoids adding a second signal source before semantic boundaries
have proved useful.

If those reruns cannot explain a repeatable slow phase, a later phase may add
one randomized on-CPU sampler.  It may reuse the timer setup in
`src/profiler.c`, but not its Lisp-object hash-table signal path.  Samples go
to a fixed numeric ring and carry the current `cause_id` and `span_id`.  A
mixed C/Lisp stack requires a separately validated shadow stack across
interpreted, byte-code, native-compiled, primitive, module, and nonlocal-exit
gateways.  The signal path performs no allocation, hashing, Lisp call, symbol
lookup, locking, or file I/O.

Off-CPU time initially comes from wall time minus thread CPU time and known
wait spans.  A second wall-clock sampling signal is justified only if material
unknown off-CPU time remains.  Compiler-wide C/Lisp function tracing is out of
scope: it changes the workload too much and duplicates targeted spans and
existing external tools.

8. C and Lisp symbols

The initial file contains no function identities.  If command grouping or
mixed-stack sampling is added later, signal records contain only
module-relative PCs and fixed Lisp frame IDs.  Symbolization happens off the
signal path using executable/shared-object build IDs and opt-in Lisp names.
Unknown frames remain explicit module-relative addresses.  A prefix-tree stack
dictionary is useful only after sample volume demonstrates that it saves more
space than it costs in writer complexity.

9. Collection and overhead control

The main thread and the optional input observer each write to a preallocated
single-producer ring.  Producers never allocate, lock, wait, symbolize, or do
file I/O.  A native writer thread drains the rings without taking the Lisp
lock, forms chunks, and writes with `writev`.  Its CPU time is reported.

Because the first phase stores aggregate interactions and spans rather than
function samples or per-glyph events, it retains every record and needs no
flight recorder.  Ring, chunk, and rotation sizes are configuration values
chosen by a burst test, not architectural constants.  The default file has no
compression and no forced `fsync`; a process crash can lose a partial current
chunk, while a machine or filesystem failure can lose more.

Any loss emits a `LOSS` record with stream, first/last time, records, bytes,
and reason.  A producer increments an out-of-ring drop counter before it
continues, so a full ring cannot hide the loss record it prevents.  Silent
loss invalidates a profile.

The acceptance budget for continuous mode is less than 1% additional CPU,
less than 1% median command latency, and less than 250 microseconds added to
p99 command latency on the reference workload.  These are gates, not presumed
facts.  The input observer is measured separately and disabled automatically
when its platform implementation cannot meet the gate.

10. File format (`.emlat`)

Integers are little-endian in fixed headers.  Record integers use unsigned or
signed LEB128.  Timestamps are deltas from the first timestamp in their block.
IDs start at one; zero means absent.  Strings are UTF-8 byte strings.  No Lisp
reader syntax or dumped Lisp object appears in the file.  IDs are stable only
within one rotated session.  Cross-session comparison uses explicit metadata,
never coincidentally equal numeric IDs.

10.1 File header

The fixed 64-byte header is:

| Offset | Size | Field |
|--------|------|-------|
| `0` | `8` | Magic bytes `EMLAT\0\r\n`. |
| `8` | `2` | Format major version. |
| `10` | `2` | Format minor version. |
| `12` | `4` | Header flags. |
| `16` | `16` | Random session UUID. |
| `32` | `8` | Rotated-file sequence number. |
| `40` | `8` | Realtime start in Unix nanoseconds. |
| `48` | `8` | Monotonic start tick in nanoseconds. |
| `56` | `4` | CRC32C of bytes `0..55`. |
| `60` | `4` | Reserved zero. |

10.2 Chunks

The remainder is a sequence of independently valid chunks, padded with zeros
to an eight-byte boundary.  A 40-byte chunk header contains:

| Offset | Size | Field |
|--------|------|-------|
| `0` | `4` | Chunk magic `CHNK`. |
| `4` | `2` | Chunk type. |
| `6` | `2` | Type-specific version. |
| `8` | `4` | Flags, including compression. |
| `12` | `4` | Header size, initially `40`. |
| `16` | `4` | Stored payload bytes. |
| `20` | `4` | Uncompressed payload bytes. |
| `24` | `8` | Monotonic chunk sequence. |
| `32` | `4` | Payload CRC32C after decompression. |
| `36` | `4` | Header CRC32C with this field zeroed. |

The core format requires no compression library.  Delta encoding, varints, and
dictionaries provide the baseline compactness.  If compression is added, a
numeric codec ID lives in the chunk flags so a reader can identify or skip the
chunk without first decoding `SESSION`.  Unsupported compression is reported
as loss.

Chunk types are:

| Type | Payload |
|------|---------|
| `SESSION` | Build/configuration, clocks, OS, CPU, TTY capabilities, profiler settings, privacy policy, and feature flags. |
| `STRING` | ID, byte length, bytes. |
| `INPUT` | Read-batch bounds and byte-provenance ranges, without byte contents. |
| `INTERACTION` | Cause links, output epochs, validity mask, raw timestamps, status, and fixed counters. |
| `SPAN` | Completed or open semantic spans and their counter deltas. |
| `LOSS` | Dropped/corrupt/unsupported data intervals. |

Dictionary definitions precede their first use.  Each record block begins with
a base timestamp, thread ID when homogeneous, record count, and schema
version.  Records then use timestamp deltas and varint IDs.  Unknown chunk
types and newer type-specific versions are skipped by chunk length and
reported, rather than guessed or misdecoded.  Sampling can add `MODULE`,
`SYMBOL`, `STACK`, and `SAMPLE` chunk types in a later format-minor version.

The writer constructs a complete chunk in memory before appending its header
and payload.  On reload, scanning stops at the first incomplete header,
impossible size, or failed CRC; every preceding chunk remains usable.  An
index can be added later if sequential scans become measurably slow.  Rotation
never reuses IDs within a session.

11. Embedded interface and reports

Profiling starts after a termcap terminal exists, through a Lisp primitive or
`--latency-profile[=FILE]`.  Lisp primitives start, stop, rotate, and query
status.  Dump loading and earlier startup are outside the initial clock.

The first loader scans chunks sequentially and materializes summaries, not one
Lisp object per record.  Memory mapping and lazy iterators are added only if
profile size makes that necessary.  The first reports are:

1. command-latency distributions, split into flushed, no-output, and
   superseded interactions;
2. a waterfall for one selected tail interaction, including input uncertainty;
3. redisplay work and emitted TTY bytes for that interaction; and
4. profiler overhead, loss, and unavailable-boundary diagnostics.

The report distinguishes wall time, main-thread CPU time, measured wait time,
unknown off-CPU time, and TTY emission time.  Function rankings and session
comparisons wait for the later attribution and replay phases.

12. Initial implementation map

The semantic recorder belongs in a small native `latency` core; it does not
modify `src/profiler.c` or install a sampling signal.  The existing Lisp CPU
and memory profiler remains unchanged.  Only a later mixed-stack sampler may
share timer setup with `src/profiler.c`.

The first semantic hooks belong at these existing funnels:

* `wait_reading_process_output` for poll start/end, ready descriptor classes,
  fallback readiness, and main-thread idle time;
* `tty_read_avail_input`, `kbd_buffer_store_event`,
  `kbd_buffer_get_event`, `read_decoded_event_from_main_queue`, and
  `read_key_sequence` for TTY read batches, queue latency, provenance folding,
  and key decoding;
* `command_loop_1` around pre-command hooks, `command-execute`, post-command
  hooks, and root completion;
* the redisplay decision in `read_char`, which is where the command's following
  redisplay is actually performed or skipped;
* `garbage_collect` for GC attribution;
* `redisplay_internal`, `redisplay_window`, `display_line`, `update_frame`,
  and `update_window` for redisplay decisions and work volume;
* one small TTY-output wrapper used by the glyph `fwrite` sites and `cmputc`,
  plus one flush wrapper used by `tty_update_end`, `flush_terminal`, and the
  other normal redisplay flush paths, for byte counts and stdio-flush latency.

Timer callbacks, process filters/sentinels, file notifications, and resize
roots are the next phase after the command path meets its overhead budget.
The input observer follows only if main-thread input intervals are too wide to
answer the latency question.  Mixed-stack sampling follows only if semantic
spans plus controlled `profiler.el`/OS-profiler reruns leave a repeatable tail
unexplained.

The native core exposes inline, allocation-free producer operations.  Lisp
report code is loaded only when a profile is opened.

13. Validation

Before optimization work uses these files, the first phase needs:

* round-trip tests for each initial record and dictionary type;
* truncated, corrupted, unknown-version, and rotated-file tests;
* fuzzing of the loader with strict allocation and record-count limits;
* synthetic CPU, sleep, GC, redisplay, error, quit, and nonlocal-exit commands
  with known durations and causal relationships;
* tests that force ring overflow and verify explicit `LOSS` records;
* PTY tests that control read batching, queueing, redisplay, terminal output,
  backpressure, and stdio-flush timing;
* privacy tests that scan default traces for injected keys, buffer text, file
  names, process output, terminal paths, and non-dumped symbol names; and
* A/B runs of an identical replay workload to enforce the continuous-mode
  budget.  The observer gets a separate A/B gate when added.

Only after these checks should profile results choose optimization targets.
The first target is the largest repeatable component of p95/p99
input-to-flush latency, not the hottest function in an average CPU profile.