cleanup and documentation

5 files changed, 161 insertions, 30 deletions

diff --git a/doc_topics/03-terminal.md b/doc_topics/03-terminal.md
new file mode 100644
index 0000000..06a6b96
--- /dev/null
+++ b/doc_topics/03-terminal.md
@@ -0,0 +1,124 @@
+# 3. Terminal functionality
+
+Terminals are fundamentally different on Windows and Posix. So even though
+`luasystem` provides primitives to manipulate both the Windows and Posix terminals,
+the user will still have to write platform specific code.
+
+To mitigate this a little, all functions are available on all platforms. They just
+will be a no-op if invoked on another platform. This means that no platform specific
+branching is required (but still possible) in user code. The user must simply set
+up both platforms to make it work.
+
+## 3.1 Backup and Restore terminal settings
+
+Since there are a myriad of settings available;
+
+- `system.setconsoleflags` (Windows)
+- `system.setconsolecp` (Windows)
+- `system.setconsoleoutputcp` (Windows)
+- `system.setnonblock` (Posix)
+- `system.tcsetattr` (Posix)
+
+Some helper functions are available to backup and restore them all at once.
+See `termbackup`, `termrestore`, `autotermrestore` and `termwrap`.
+
+
+## 3.1 Terminal ANSI sequences
+
+Windows is catching up with this. In Windows 10 (since 2019), the Windows Terminal application (not to be
+mistaken for the `cmd` console application) supports ANSI sequences. However this
+might not be enabled by default.
+
+ANSI processing can be set up both on the input (key sequences, reading cursor position)
+as well as on the output (setting colors and cursor shapes).
+
+To enable it use `system.setconsoleflags` like this:
+
+    -- setup Windows console to handle ANSI processing on output
+    sys.setconsoleflags(io.stdout, sys.getconsoleflags(io.stdout) + sys.COF_VIRTUAL_TERMINAL_PROCESSING)
+    sys.setconsoleflags(io.stderr, sys.getconsoleflags(io.stderr) + sys.COF_VIRTUAL_TERMINAL_PROCESSING)
+
+    -- setup Windows console to handle ANSI processing on input
+    sys.setconsoleflags(io.stdin, sys.getconsoleflags(io.stdin) + sys.CIF_VIRTUAL_TERMINAL_INPUT)
+
+
+## 3.2 UTF-8 in/output and display width
+
+### 3.2.1 UTF-8 in/output
+
+Where (most) Posix systems use UTF-8 by default, Windows internally uses UTF-16. More
+recent versions of Lua also have UTF-8 support. So `luasystem` also focusses on UTF-8.
+
+On Windows UTF-8 output can be enabled by setting the output codepage like this:
+
+    -- setup Windows output codepage to UTF-8; 65001
+    sys.setconsoleoutputcp(65001)
+
+Terminal input is handled by the [`_getwchar()`](https://learn.microsoft.com/en-us/cpp/c-runtime-library/reference/getchar-getwchar) function on Windows which returns
+UTF-16 surrogate pairs. `luasystem` will automatically convert those to UTF-8.
+So when using `readkey` or `readansi` to read keyboard input no additional changes
+are required.
+
+### 3.2.2 UTF-8 display width
+
+Typical western characters and symbols are single width characters and will use only
+a single column when displayed on a terminal. However many characters from other
+languages/cultures or emojis require 2 columns for display.
+
+Typically the `wcwidth` function is used on Posix to check the number of columns
+required for display. However since Windows doesn't provide this functionality a
+custom implementation is included based on [the work by Markus Kuhn](http://www.cl.cam.ac.uk/~mgk25/ucs/wcwidth.c).
+
+2 functions are provided, `system.utf8cwidth` for a single character, and `system.utf8swidth` for
+a string. When writing terminal applications the display width is relevant to
+positioning the cursor properly. For an example see the [`examples/readline.lua`](../examples/readline.lua.html) file.
+
+
+## 3.3 reading keyboard input
+
+### 3.3.1 Non-blocking
+
+There are 2 functions for keyboard input (actually 3, if taking `system._readkey` into
+account): `readkey` and `readansi`.
+
+`readkey` is a low level function and should preferably not be used, it returns
+a byte at a time, and hence can leave stray/invalid byte sequences in the buffer if
+only the start of a UTF-8 or ANSI sequence is consumed.
+
+The preferred way is to use `readansi` which will parse and return entire characters in
+single or multiple bytes, or a full ANSI sequence.
+
+On Windows the input is read using [`_getwchar()`](https://learn.microsoft.com/en-us/cpp/c-runtime-library/reference/getchar-getwchar) which bypasses the terminal and reads
+the input directly from the keyboard buffer. This means however that the character is
+also not being echoed to the terminal (independent of the echo settings used with
+`system.setconsoleflags`).
+
+On Posix the traditional file approach is used, which:
+
+- is blocking by default
+- echoes input to the terminal
+- requires enter to be pressed to pass the input (canonical mode)
+
+To use non-blocking input here's how to set it up:
+
+    -- setup Windows console to disable echo and line input (not required since _getwchar is used, just for consistency)
+    sys.setconsoleflags(io.stdin, sys.getconsoleflags(io.stdin) - sys.CIF_ECHO_INPUT - sys.CIF_LINE_INPUT)
+
+    -- setup Posix by disabling echo, canonical mode, and making non-blocking
+    local of_attr = sys.tcgetattr(io.stdin)
+    sys.tcsetattr(io.stdin, sys.TCSANOW, {
+      lflag = of_attr.lflag - sys.L_ICANON - sys.L_ECHO,
+    })
+    sys.setnonblock(io.stdin, true)
+
+
+Both functions require a timeout to be provided which allows for proper asynchronous
+code to be written. Since the underlying sleep method used is `system.sleep`, just patching
+that function with a coroutine based yielding one should be all that is needed to make
+the result work with asynchroneous coroutine schedulers.
+
+### 3.3.2 Blocking input
+
+When using traditional input method like `io.stdin:read()` (which is blocking) the echo
+and newline properties should be set on Windows similar to Posix.
+For an example see [`examples/password_input.lua`](../examples/password_input.lua.html).
diff --git a/examples/readline.lua b/examples/readline.lua
index f1e6258..286522c 100644
--- a/examples/readline.lua
+++ b/examples/readline.lua
@@ -1,3 +1,9 @@
+--- An example class for reading a line of input from the user in a non-blocking way.
+-- It uses ANSI escape sequences to move the cursor and handle input.
+-- It can be used to read a line of input from the user, with a prompt.
+-- It can handle double-width UTF-8 characters.
+-- It can be used asynchroneously if `system.sleep` is patched to yield to a coroutine scheduler.
+
 local sys = require("system")
 
 
@@ -134,7 +140,7 @@ readline.__index = readline
 --- Create a new readline object.
 -- @tparam table opts the options for the readline object
 -- @tparam[opt=""] string opts.prompt the prompt to display
--- @tparam[opt=80] number opts.max_length the maximum length of the input
+-- @tparam[opt=80] number opts.max_length the maximum length of the input (in characters, not bytes)
 -- @tparam[opt=""] string opts.value the default value
 -- @tparam[opt=`#value`] number opts.position of the cursor in the input
 -- @tparam[opt={"\10"/"\13"}] table opts.exit_keys an array of keys that will cause the readline to exit
@@ -425,29 +431,25 @@ end
 
 
 
--- return readline
+-- return readline  -- normally we'd return here, but for the example we continue
+
 
 
 
+local backup = sys.termbackup()
 
 -- setup Windows console to handle ANSI processing
-local of_in = sys.getconsoleflags(io.stdin)
-local cp_in = sys.getconsolecp()
--- sys.setconsolecp(65001)
-sys.setconsolecp(850)
-local of_out = sys.getconsoleflags(io.stdout)
-local cp_out = sys.getconsoleoutputcp()
-sys.setconsoleoutputcp(65001)
 sys.setconsoleflags(io.stdout, sys.getconsoleflags(io.stdout) + sys.COF_VIRTUAL_TERMINAL_PROCESSING)
 sys.setconsoleflags(io.stdin, sys.getconsoleflags(io.stdin) + sys.CIF_VIRTUAL_TERMINAL_INPUT)
+-- set output to UTF-8
+sys.setconsoleoutputcp(65001)
 
--- setup Posix terminal to use non-blocking mode, and disable line-mode
-local of_attr = sys.tcgetattr(io.stdin)
-local of_block = sys.getnonblock(io.stdin)
-sys.setnonblock(io.stdin, true)
+-- setup Posix terminal to disable canonical mode and echo
 sys.tcsetattr(io.stdin, sys.TCSANOW, {
-  lflag = of_attr.lflag - sys.L_ICANON - sys.L_ECHO, -- disable canonical mode and echo
+  lflag = sys.tcgetattr(io.stdin).lflag - sys.L_ICANON - sys.L_ECHO,
 })
+-- setup stdin to non-blocking mode
+sys.setnonblock(io.stdin, true)
 
 
 local rl = readline.new{
@@ -467,10 +469,4 @@ print("Exit-Key (bytes):", key:byte(1,-1))
 
 
 -- Clean up afterwards
-sys.setnonblock(io.stdin, false)
-sys.setconsoleflags(io.stdout, of_out)
-sys.setconsoleflags(io.stdin, of_in)
-sys.tcsetattr(io.stdin, sys.TCSANOW, of_attr)
-sys.setnonblock(io.stdin, of_block)
-sys.setconsolecp(cp_in)
-sys.setconsoleoutputcp(cp_out)
+sys.termrestore(backup)
diff --git a/examples/terminalsize.lua b/examples/terminalsize.lua
index 78d1910..ed66792 100644
--- a/examples/terminalsize.lua
+++ b/examples/terminalsize.lua
@@ -26,11 +26,12 @@ end
 
 local w, h
 print("Change the terminal window size, press any key to exit")
-while not sys.readkey(0.2) do
+while not sys.readansi(0.2) do  -- use readansi to not leave stray bytes in the input buffer
   local nw, nh = sys.termsize()
   if w ~= nw or h ~= nh then
     w, h = nw, nh
     local text = "Terminal size: " .. w .. "x" .. h .. "     "
     io.write(text .. cursor_move_horiz(-#text))
+    io.flush()
   end
 end
diff --git a/src/term.c b/src/term.c
index 99d3b2b..7020f09 100644
--- a/src/term.c
+++ b/src/term.c
@@ -337,7 +337,7 @@ To see flag status and constant names check `listconsoleflags`.
 Note: not all combinations of flags are allowed, as some are mutually exclusive or mutually required.
 See [setconsolemode documentation](https://learn.microsoft.com/en-us/windows/console/setconsolemode)
 @function setconsoleflags
-@tparam file file the file-handle to set the flags on
+@tparam file file file handle to operate on, one of `io.stdin`, `io.stdout`, `io.stderr`
 @tparam bitflags bitflags the flags to set/unset
 @treturn[1] boolean `true` on success
 @treturn[2] nil
@@ -378,8 +378,17 @@ static int lst_setconsoleflags(lua_State *L)
 
 /***
 Gets console flags (Windows).
+The `CIF_` and `COF_` constants are available on the module table. Where `CIF` are the
+input flags (for use with `io.stdin`) and `COF` are the output flags (for use with
+`io.stdout`/`io.stderr`).
+
+_Note_: See [setconsolemode documentation](https://learn.microsoft.com/en-us/windows/console/setconsolemode)
+for more information on the flags.
+
+
+
 @function getconsoleflags
-@tparam file file the file-handle to get the flags from.
+@tparam file file file handle to operate on, one of `io.stdin`, `io.stdout`, `io.stderr`
 @treturn[1] bitflags the current console flags.
 @treturn[2] nil
 @treturn[2] string error message
@@ -433,8 +442,8 @@ The terminal attributes is a table with the following fields:
 
 - `iflag` input flags
 - `oflag` output flags
-- `cflag` control flags
 - `lflag` local flags
+- `cflag` control flags
 - `ispeed` input speed
 - `ospeed` output speed
 - `cc` control characters
@@ -528,9 +537,6 @@ flags for the `iflags`, `oflags`, and `lflags` bitmasks.
 To see flag status and constant names check `listtermflags`. For their meaning check
 [the manpage](https://www.man7.org/linux/man-pages/man3/termios.3.html).
 
-_Note_: not all combinations of flags are allowed, as some are mutually exclusive or mutually required.
-See [setconsolemode documentation](https://learn.microsoft.com/en-us/windows/console/setconsolemode)
-
 _Note_: only `iflag`, `oflag`, and `lflag` are supported at the moment. The other fields are ignored.
 @function tcsetattr
 @tparam file fd file handle to operate on, one of `io.stdin`, `io.stdout`, `io.stderr`
@@ -722,6 +728,7 @@ directly, but through the `system.readkey` or `system.readansi` functions. It
 will return the next byte from the input stream, or `nil` if no key was pressed.
 
 On Posix, `io.stdin` must be set to non-blocking mode using `setnonblock`
+and canonical mode must be turned off using `tcsetattr`,
 before calling this function. Otherwise it will block. No conversions are
 done on Posix, so the byte read is returned as-is.
 
diff --git a/system/init.lua b/system/init.lua
index b9a4f6f..8049167 100644
--- a/system/init.lua
+++ b/system/init.lua
@@ -7,7 +7,7 @@ local sys = require 'system.core'
 do
   local backup_mt = {}
 
-  --- Returns a backup of terminal setting for stdin/out/err.
+  --- Returns a backup of terminal settings for stdin/out/err.
   -- Handles terminal/console flags, Windows codepage, and non-block flags on the streams.
   -- Backs up terminal/console flags only if a stream is a tty.
   -- @return table with backup of terminal settings
@@ -227,8 +227,11 @@ do
   -- This function uses `system.sleep` to wait until either a byte is available or the timeout is reached.
   -- The sleep period is exponentially backing off, starting at 0.0125 seconds, with a maximum of 0.2 seconds.
   -- It returns immediately if a byte is available or if `timeout` is less than or equal to `0`.
+  --
+  -- Using `system.readansi` is preferred over this function. Since this function can leave stray/invalid
+  -- byte-sequences in the input buffer, while `system.readansi` reads full ANSI and UTF8 sequences.
   -- @tparam number timeout the timeout in seconds.
-  -- @treturn[1] integer the key code of the key that was received
+  -- @treturn[1] byte the byte value that was read.
   -- @treturn[2] nil if no key was read
   -- @treturn[2] string error message; `"timeout"` if the timeout was reached.
   function sys.readkey(timeout)