| 1 | # Devanagari terminal-shaping spike (#4790, v0.9.2) |
| 2 | |
| 3 | Status: **spike complete — code-level guarantees only. No native-speaker or |
| 4 | per-terminal visual signoff has been performed or is claimed here.** |
| 5 | |
| 6 | ## Question |
| 7 | |
| 8 | Can the TUI render Hindi (Devanagari) UI copy without corrupting conjunct |
| 9 | clusters when strings are clipped or wrapped at narrow terminal widths |
| 10 | (40/60/80 columns)? |
| 11 | |
| 12 | ## What Devanagari needs |
| 13 | |
| 14 | Devanagari renders through complex text shaping: a consonant + virama |
| 15 | (U+094D) + consonant forms a conjunct glyph (क + ् + ष → क्ष), vowel signs |
| 16 | reorder around their base (क + ि → कि, the ि renders *before* the consonant), |
| 17 | and nukta forms (क़) combine a base with U+093C. Cutting a string between a |
| 18 | base and its virama, or between a base and a combining sign, leaves a |
| 19 | dangling halant or an orphaned mark — visibly broken copy. |
| 20 | |
| 21 | ## Findings |
| 22 | |
| 23 | 1. **Truncation was char-based and could split clusters.** The old |
| 24 | `truncate_to_width` iterated `chars()`; a width budget landing between |
| 25 | क and ् emitted a trailing virama. It now iterates extended grapheme |
| 26 | clusters (`unicode-segmentation`, already a workspace dependency) and |
| 27 | measures each cluster with `unicode-width`, so a cluster is kept whole |
| 28 | or dropped whole. Covered by |
| 29 | `truncate_to_width_never_splits_devanagari_clusters` (budgets |
| 30 | 1/2/3/5/7/40/60/80: no U+FFFD, no trailing virama/ZWJ/combining mark) |
| 31 | and `cyrillic_latin_extended_and_devanagari_rows_wrap_within_terminal_columns` |
| 32 | in `crates/tui/src/localization.rs`. |
| 33 | 2. **Wrapping is word-based and safe.** ratatui's `Paragraph::wrap` breaks |
| 34 | on whitespace/punctuation, not inside words, so conjuncts inside a word |
| 35 | are never split by wrapping. Verified at 40/60/80 columns with a Hindi |
| 36 | fixture row (same test module). |
| 37 | 3. **Width is the honest weak point.** `unicode-width` reports Devanagari |
| 38 | base letters as width 1 and combining marks as width 0, which matches |
| 39 | what a correctly shaping terminal displays *most of the time*. Some |
| 40 | conjuncts render narrower than their cluster sum on shaping terminals |
| 41 | and wider on non-shaping ones; the budget logic errs on the side of |
| 42 | clipping early, never overdrawing the row. |
| 43 | 4. **The test suite cannot see pixels.** These tests assert codepoint- and |
| 44 | cell-level invariants in ratatui's buffer model. They do not prove any |
| 45 | real terminal shaped the text correctly. |
| 46 | |
| 47 | ## Terminal support matrix (informed assessment, not tested on hardware) |
| 48 | |
| 49 | | Terminal | Devanagari shaping expectation | |
| 50 | |----------|-------------------------------| |
| 51 | | WezTerm, Kitty (recent), foot | HarfBuzz/pango-class shaping; conjuncts render correctly | |
| 52 | | Windows Terminal (recent) | Shaping via DirectWrite; generally correct | |
| 53 | | GNOME Terminal / VTE, Konsole | Correct via Pango/Qt | |
| 54 | | macOS Terminal.app, iTerm2 | CoreText shaping; generally correct | |
| 55 | | Alacritty | **No complex text shaping** — conjuncts render as base+visible halant; readable but wrong | |
| 56 | | tmux/screen | Pass-through cells; inherits the outer terminal's behavior, but cluster-aware cursor math is limited | |
| 57 | | Linux VT (fbcon), older conhost | No shaping; expect broken conjuncts | |
| 58 | | SSH into any of the above | Inherits the *local* terminal's shaping | |
| 59 | |
| 60 | **Recommendation:** the Hindi pack ships with this caveat documented; |
| 61 | users on Alacritty or the Linux console will see un-shaped conjuncts. |
| 62 | This is a terminal capability limit, not something the TUI can fix from |
| 63 | the cell grid. Native-speaker review of the pack copy and visual QA on at |
| 64 | least one shaping terminal (VTE-class or WezTerm) remain open follow-ups |
| 65 | before the pack should be called fully signed off. |
| 66 | |
| 67 | ## What was NOT verified |
| 68 | |
| 69 | - No native Hindi speaker has reviewed the pack. |
| 70 | - No physical terminal rendering was inspected (screenshot QA). |
| 71 | - `unicode-width` conjunct widths vs. real glyph advances on shaping |
| 72 | terminals (known approximation, see finding 3). |
| 73 |