IMI Library — Reference · 2026-05-31 · Brian Borg · v1.0
What this document is. A recording becomes a list of utterance rows in two layers. First, the pipeline runs a fixed 7-rule turn-based consolidation (Step 4.5) that turns raw transcript turns into clean rows. Second, the review page lets a reviewer re-derive those rows live with four tuning dials, without reprocessing the audio.
This doc explains the rules, the order each is applied during processing, and exactly what each dial does to the displayed utterances. A short Known issues section at the end records what is currently broken (2026-05-31) so the descriptions aren't mistaken for current behavior.
Everything below lives in one engine, written twice and kept in lockstep:
pipeline/enrich/consolidate/turn_based.py runs once,
during enrichment, using fixed constants (20-minute Swami split, 10-second tiny-row absorb, 15-second
interjection cap). Its output is the neutral baseline.review-ui/js/tuning.js is a faithful browser port of the
same engine. It re-runs the rules live with the four dials substituted for those fixed constants, so a
reviewer can tighten or loosen the result without reprocessing.Three columns on the media row carry the data between the two layers:
| Column | What it holds | Mutable? |
|---|---|---|
consolidation_inputs | The immutable raw inputs the rules run on (raw transcript utterances, speaker map, music candidates, topic shifts) | No — fixed at processing time |
pipeline_original | The neutral consolidated rows — the pipeline's own output at dial position zero | No — the baseline snapshot |
utterances | What the Review UI displays. Equals pipeline_original until a dial is moved, then holds the re-derived rows | Yes — re-derived by the dials |
The rules all execute at one point (Step 4.5, consolidation), but each depends on a signal produced by an earlier step. If an earlier step is missing or fails, the rule that depends on it has nothing to act on.
utterances, pipeline_original, consolidation_inputsmedia rowconsolidation_inputsThe order of the steps is fixed, but the rules themselves are applied in a single pass at Step 4.5 in the order below: regions are carved first (Rules 1–2), then the speech timeline is walked turn by turn (Rules 3–7), then a cleanup pass absorbs tiny mis-diarized fragments.
Applied in this order, in consolidate_turn_based(). "Acts on" is the signal the rule reads;
"Needs" is the upstream step that must have produced it.
Each music region becomes a single row labeled "Song." Any transcribed speech that falls inside a music region is removed — lyrics don't become visitor turns.
A stretch where multiple people talk over each other collapses to a single row labeled "Multiple speakers" rather than a tangle of half-caught turns.
When someone who isn't Swami speaks, their first stretch of speech becomes one row — the only row they get for that turn.
Swami's reply is a single row. It starts when he begins and ends when a different speaker takes the seat, a song begins, or the recording ends. Silence inside the answer stays part of his row.
When the same visitor makes a short interjection during Swami's answer (under 15 seconds), it's folded into Swami's row in time only — the interjection's text is dropped, and no extra row appears. A longer re-engagement is treated as a new turn instead (Rule 7).
If Swami speaks for 20 minutes or more and Claude detected topic shifts inside that stretch, the single row is split into one row per topic segment so a long talk is navigable.
When a different non-Swami speaker takes over, the cycle restarts — Rules 3 through 6 repeat for the new exchange.
After the seven rules, a final pass removes visitor rows shorter than 10 seconds whose speaker the voiceprint never identified. These are usually a 1–2 second sliver of Swami's own voice that AssemblyAI mis-labeled as a new speaker. The sliver is absorbed back into the surrounding Swami rows so one continuous answer doesn't fracture into three. Identified visitors — even short ones — are always kept.
Four dials, each from −5 to +5, with 0 = the pipeline's own output, verbatim.
Moving a dial re-runs the engine on the stored consolidation_inputs with one constant replaced by
the dial value. The dials only change what is displayed; they never touch the immutable inputs, so any
setting is reversible by returning to neutral. Source: dialsToParams() in
review-ui/js/tuning.js.
| Dial (screen label) | Acts on | Toward + | Toward − |
|---|---|---|---|
| Music more speech ↔ more song |
Rule 1 — which audio is carved out as song | Carves more audio as song | Carves less; shaky "music" reverts to speech |
| Swami vocal match looser ↔ stricter |
Voiceprint match threshold (0.80 at neutral) | Stricter, up to 0.92 — fewer turns accepted as Swami | Looser, down to 0.65 — more turns accepted as Swami |
| Swami speeches | Rule 6 — splitting long Swami stretches | Splits less → fewer Swami rows (never splits at +5) | Splits readily → more Swami rows |
| Other speeches | Rule 5 + tiny-absorb thresholds | Absorbs more interjections → fewer visitor rows | Absorbs nothing → every interjection survives |
Decides which YAMNet candidates are carved out as song. Neutral passes every detected music candidate and leaves the speech-reject density at its baseline.
Re-thresholds the stored per-utterance voiceprint similarity. An utterance is treated as Swami only when its match score clears the threshold.
Controls how readily a long Swami stretch splits into multiple rows at topic boundaries — the split gate and the topic-shift confidence required to act.
Controls how aggressively short visitor interjections and unidentified slivers are absorbed versus kept as their own rows.
The on-screen captions for Swami speeches and Other speeches currently read "fewer entries" on the left (−) and "more entries" on the right (+) — the opposite of what the engine does (+ yields fewer rows, − yields more; the module's own header says "strict (+5) = fewer rows"). The Music and Swami-vocal-match captions match the engine. The two inverted captions are on the fix list.
When a dial moves, the browser calls
deriveRows(consolidation_inputs, pipeline_original, dials):
pipeline_original
verbatim — the port is never exercised at neutral, so it can't drift from the Python output.consolidation_inputs with
the dial-substituted thresholds, then writes the result to the displayed utterances.Because the inputs are immutable, returning every dial to 0 always restores the pipeline's original rows.
The behavior above is the intended design. As of this date the following are broken and on the fix list —
noted here so the descriptions aren't read as current reality. Full analysis:
imi-library-wiki/wiki/lessons/2026-05-31-tuning-view-missing-columns-zeroes-utterances.md.
event_media_view, which doesn't expose pipeline_original or
consolidation_inputs (added later in migration 038). The dial engine gets empty inputs, returns
an empty list at every setting, and autosaves it over utterances. Highresemblyzer is not installed on the bundle), so
0 speakers are identified. The three speaker-based dials have nothing to act on until this is fixed.audio_segments empty), so Rule 1 emits nothing — a song should still show one "Song" row.