busybox-w32/libbb/unicode.c, branch cron_changes

win32: fix printable-chars with no-op utf8 manifest

2025-12-26T12:29:54+00:00

When unicode is enabled at build time, but disabled at runtime (commit 208649d7), we already do some runtime ACP tests and act accordingly (we know unicode is not in effect) in mingw non-upstream code (UTF8_OUTPUT, etc - see previous commit). However, there's at least one upstream unicode code path which was not touched until now - printable chars. This code is used by applets like "ls" to display "?" instead of non-printable chars, where upstream considers byte values above 127 as non-printable, but the busybox-w32 code should consider them printable when unicode is disabled (ANSI codepage). We already have code for that when unicode is disabled at build time (at printable_string2), but when unicode is enabled at build time then it enters a different code path (which still tests dynamically whether unicode is enabled - and correctly deduces that it isn't when unicode is disabled despite the utf8 manifest). Modify unicode_conv_to_printable2 similar to how printable_string2 is modified - but only when unicode is disabled due to the manifest not being in effect (unicode build on win XP/7/8). A unicode build on XP/7/8 _should_ now behave reasonably close to a non-unicode build on such systems. Known issues: There's one known issue, and that's tab completion doesn't case-fix the completion. E.g. ~/desk completes to ~/desktop/ instead of to ~/Desktop/ (capital D). This issue exists in unicode builds also on win10+, and is caused by the fact that when unicode is enabled at build-time, line-editing uses (32bit) wchar_t, and of the mingw changes in line-edit, only the backslash-to-slash was ported also to the wchar_t editing, while case-fix was not (see FIXME at libbb/lineedit.c in commit 0efc7474). status: This combination of unicode-enabled build running on XP/7/8 was NOT tested extensively, so some issues might remain, but other than the case-fix with tab-completion, there are no known issues. Main area where more testing is needed is editing (probably mainly interactive shell command prompt and vi). Testing: To test how a unicode-enabled build behaves when unicode is disabled in runtime, other than actually running it on win XP/7/8, one way would be to find a combination of build-config values to enable all the unicode features but not the manifest itself, but that's tricky. Instead, one could simply delete the manifest from a pre-compiled binary (mingw{64u,32w}_defconfig), so that it would run in ANSI mode even on win10+, with the build-time unicode code paths still enabled at the binary. One tool to edit windows binaries is https://github.com/avih/perc . To delete the manifest, use "perc -D -t MANIFEST busybox.exe" .

win32: unicode: new wcwidth: allow enabling bidi

2024-04-02T18:02:26+00:00

interval, in_interval_table, and in_uint16_table were previously not compiled when using the new wcwidth (commit c188a345a) because they're used by the old wcwidth but not by the new one. But they're also used by the BIDI routines. mingw64u_defconfig doesn't enable bidi (rightly - it's not working well), but it'd still be nice to allow enabling bidi while the new wcwidth is in effect. Enable the tables lookup code if BIDI is enabled.

Revert "unicode: identify emoji width and modifiers"

2024-03-29T14:32:57+00:00

This reverts commit 878b3cd27fe83f2b0ff476b884c34d165be0072c. It's no longer required, since the last commit uses a new wcwidth implementation which covers the cases added by commit 878b3cd2 .

win32: unicode: use newer wcwidth by default

2024-03-29T14:26:13+00:00

This commit adds a new wcwidth implementation at libbb/wcwidth_alt.c, and uses it instead of the existing implementation when compiling for windows and CONFIG_LAST_SUPPORTED_WCHAR >= 0x30000 - which is the case with the unicode configs/mingw64u_defconfig. The windows-target condition keeps non-windows build unmodified, and the last supported wchar threshold is a semi-hack to allow switching between implementations without adding a new config option (the old code supports codepoints up to 0x2ffff). The new file wcwidth_alt.c was generated by a new scripts/mkwcwidth, which prints a wcwidth implementation using latest unicode data from a local clone of https://github.com/jquast/wcwidth . This repo is the main python wcwidth implementation, and is maintained and up to date. Functional differences from the existing implementation: - Unicode 15.1.0 (latest) with the new version (about 450 ranges of wide and zero-width codepoints), compared to roughly Unicode 5.0 of the existing code (nearly 20 years old spec, about 150 ranges). The new spec includes, among others, various wide icons and emojis, which can now be edited correctly at the shell prompt, have correct alignment in 'ls', etc. - The old implementation returns -1 (non-printable) for surrogates, while the new code returns 1, though this is inconsequential, and POSIX doesn't care. Also libc implementations vary in this regard. Technical differences: - The old version compiles less code/data when the last supported wchar is smaller, while the new version doesn't. This doesn't matter because the new version is enabled only for the full range. - The new version is smaller and relatively straight forward, and fully automated (generated), so updates to newer spec is trivial. The old version mixes data, ad-hoc code (tailored to the data), and preprocessor checks, and is hard to automate updates. The old version has various forms of 32 and 16 bit data ranges, in several arrays, while the new version uses single data array with unified form of 32 bits per range, with two rules: - A data range can't span Unicode planes (enforced, but unlikely required, and if yes, code to split ranges would be simple). - A range can't hold more than 32768 codepoints, so bigger ranges are split automatically (currently there are 2 such ranges). Performance wise, the new version should be faster, even with three times the data ranges. Both versions do effectively at most one binary search in one Unicode plane data, but the new version finds both zero-width and wide-width results in this one search, while the old version only finds zero-width, and to detect wide-width it does an additional linear series of manual range tests, but since most results are width 1, this sequence is performed in most (non-ASCII) calls. In a cursory comparison of the new wcwidth with glibc and musl-libc (both use O(1) lookup tables), with few bodies of text, we're in the same ballpark, with typical speed of 60% or better. Bloat-wise, the new version is about 180 bytes code and 1800 bytes data. If it had similar number of data ranges as the old code (150), the new version would be about 200 bytes smaller, but because the new version has 450 data ranges, it's about 1K bigger.

unicode: identify emoji width and modifiers

2023-07-23T11:42:10+00:00

This adds the Emoticons block U+1F600..U+1F64F as double-width codepoints, and the skin tone modifiers range U+1F3FB..U+1F3FF as combining codepoints. The Emoticons variant modifiers U+FE0E and U+FE0F were already in. It's unclear how to test UNICODE_COMBINING_WCHARS and UNICODE_WIDE_WCHARS in general and also here specifically, but at least the data on Emojis width and combinings now exits.

win32: support build with FEATURE_UNICODE_SUPPORT

2023-07-22T06:40:16+00:00

FEATURE_UTF8_MANIFEST enables Unicode args and filenames on Win 10+. FEATURE_UTF8_INPUT allows the shell prompt to digest correctly Unicode strings (as UTF8) which are typed or pasted. This commit adds support for building with FEATURE_UNICODE_SUPPORT (mostly by supporting 32 bit wchar_t which busybox expects): - Unicode-aware line-edit - for the most part cursor movement/del being (UTF8) codepoint-aware rather than assuming that one-byte equals one-char-on-screen. - Codepoint-aware operations in some other utils, like rev or wc -c. - When UNICODE_COMBINING_WCHARS and UNICODE_WIDE_WCHARS are enabled, some screen-width-aware operations, like with fold, ls, expand, etc. The busybox Unicode support is incomplete, and even less so with the builtin libc replacement functions, like wcwidth, which are active when UNICODE_USING_LOCALE is unset (mingw lacks those functions). FEATURE_CHECK_UNICODE_IN_ENV should be set so that Unicode is not hardcoded but rather depends on the ANSI codepage and some env vars: LC_ALL=C disables Unicode support, else it's enabled if ACP is UTF8. There's at least one known issue where the tab-completion-prefix-case is not updated correctly, e.g. ~/desk completes to ~/desktop/ instead of ~/Desktop/, because the code which handles it exists only at the non-unicode code paths, but that's not very critical. That seems to be the only case where mingw-specific code is disabled when Unicode is enabled, but there could be other unknown issues. None of the Unicode options is enabled by default, and the next commit will make it easier to create a build which supports Unicode.

unicode: relax array alignment for tables

2020-11-30T10:30:17+00:00

text data bss dec hex filename 1022075 559 5052 1027686 fae66 busybox_old 1021988 559 5052 1027599 fae0f busybox_unstripped Signed-off-by: Denys Vlasenko

unicode: fix handling of short 1-4 char tables

2020-11-30T10:18:00+00:00

function old new delta in_uint16_table 92 107 +15 Signed-off-by: Denys Vlasenko

unicode: code shrink in character width determination

2019-07-23T09:26:54+00:00

function old new delta bb_wcwidth 267 238 -29 Signed-off-by: Denys Vlasenko

unzip: use printable_string() for printing filenames

2018-09-30T14:56:56+00:00

function old new delta unzip_main 2726 2792 +66 printable_string2 - 57 +57 identify 4329 4336 +7 expmeta 659 663 +4 add_interface 99 103 +4 beep_main 286 289 +3 changepath 192 194 +2 builtin_type 115 117 +2 devmem_main 469 470 +1 input_tab 1076 1074 -2 create_J 1821 1819 -2 poplocalvars 314 311 -3 doCommands 2222 2214 -8 do_load 918 902 -16 printable_string 57 9 -48 ------------------------------------------------------------------------------ (add/remove: 1/0 grow/shrink: 8/6 up/down: 146/-79) Total: 67 bytes Signed-off-by: Denys Vlasenko