open
https://gitlab.synchro.net/main/sbbs/-/issues/1204
Message listings (mail and sub-boards) lose column alignment whenever a
`from`, `to`, or `subject` header field contains non-ASCII characters. Each such field is printed narrower than its format string asks for, so every
column to its right slides left by the difference.
## Symptom
Listing mail with `MailOnSystemLstFmt` (`ctrl/text.dat:91`), which is `"\1U%5d\1u %-22.22s %-22.22s \1U%c\1u %s\r\n"`:
```
1 Rob Swindell Digital Man plain ASCII
2 José Muñoz Digital Man 2 two-byte chars in From
3 Rob Swindell Åsa Österberg 2 two-byte chars in To
4 Михаил Digital Man Cyrillic (6 chars, 12 bytes)
5 東京 BBS Digital Man CJK (2 chars, 6 bytes, 4 columns)
```
The flag column, which should be fixed, lands at column 52, 50, 50, 46 and 48 respectively. The shortfall is exactly (bytes - display columns) for the
field.
The same applies to `SubMsgLstFmt` (`ctrl/text.dat:109`) and `MailWaitingLstFmt` (`ctrl/text.dat:61`).
## Cause
`bprintf()` formats first and converts the character set second, so the
field widths in the format string are counted in *bytes* while the terminal renders *columns*.
The listing call sites pass the raw header fields straight through, tagging
the message's UTF-8 bit into the print mode:
```cpp
bprintf(P_TRUNCATE | (msg.hdr.auxattr & MSG_HFIELDS_UTF8)
, msghdr_text(&msg, MailOnSystemLstFmt)
, u + 1, msg.from, msg_to(&msg)
, mail_listing_flag(&msg)
, msg.subj);
```
(`src/sbbs3/readmail.cpp:526`, `:742`, `:923` plus the `MailWaitingLstFmt` variants, and `src/sbbs3/readmsgs.cpp:85`, `:1773`, `:1830`.)
`sbbs_t::bprintf()` (`src/sbbs3/con_out.cpp:535`) hands the format directly
to `vsnprintf()`, so `%-22.22s` reserves 22 **bytes**. Only afterwards does `bputs()` (`src/sbbs3/con_out.cpp:134`) convert, one codepoint at a time, via `print_utf8_as_cp437()` (`src/sbbs3/con_out.cpp:306`). A field of 22 bytes containing an n-byte sequence therefore occupies 22 - (n - 1) columns.
This affects **both** terminal types. On a UTF-8 terminal the bytes pass through untouched and the terminal renders one column per codepoint; on a
CP437 terminal the conversion collapses each sequence to a single character. Either way the byte count overstates the column count.
## Secondary effects on non-UTF-8 terminals
The `.22` precision truncates on a byte boundary, so it can cut a multi-byte sequence in half. `print_utf8_as_cp437()` then fails `utf8_getc()`, emits the orphaned byte as if it were CP437 (a garbage glyph), and logs a line per affected message from `src/sbbs3/con_out.cpp:316`:
```
Invalid UTF-8 sequence: %02X (error = %d)
```
An unmappable codepoint is rendered as an inverted question mark, and a
*wide* unmappable codepoint as an inverted question mark plus a pad space — neither of which matches the number of bytes it consumed from the field.
## Reproduction
Send mail with a non-ASCII `From`, `To`, or `Subject` (any message whose
header carries `MSG_HFIELDS_UTF8`), then list it — `R`ead mail, or the message listing of any sub-board. The misalignment scales with the number of non-ASCII characters, so Cyrillic or CJK names show it most clearly.
## Related
Same general area as #1198 and #1200, but a distinct defect: those concern column *counting* during output, this one concerns field *widths* being measured in bytes at format time.
— *Authored by Claude (Claude Code), on behalf of @rswindell*
--- SBBSecho 3.37-Linux
* Origin: Vertrauen - [vert/cvs/bbs].synchro.net (1:103/705)