New centralised version of local line editing.

This takes over from both the implementation in ldisc.c and the one in
term_get_userpass_input, which were imperfectly duplicating each
other's functionality. The new version should be more consistent
between the two already, and also, it means further improvements can
now be made in just one place.

In the course of this, I've restructured the inside of ldisc.c by
moving the input_queue bufchain to the other side of the translation
code in ldisc_send. Previously, ldisc_send received a string, an
optional 'dedicated key' indication (bodgily signalled by a negative
length) and an 'interactive' flag, translated that somehow into a
combination of raw backend output and specials, and saved the latter
in input_queue. Now it saves the original (string, dedicated flag,
interactive flag) data in input_queue, and doesn't do the translation
until the data is pulled back _out_ of the queue. That's because the
new line editing system expects to receive something much closer to
the original data format.

The term_get_userpass_input system is also substantially restructured.
Instead of ldisc.c handing each individual keystroke to terminal.c so
that it can do line editing on it, terminal.c now just gives the Ldisc
a pointer to its instance of the new TermLineEditor object - and then
ldisc.c can put keystrokes straight into that, in the same way it
would put them into its own TermLineEditor, without having to go via
terminal.c at all. So the term_get_userpass_input edifice is only
called back when the line editor actually delivers the answer to a
username or password prompt.

(I considered not _even_ having a separate TermLineEditor for password
prompts, and just letting ldisc.c use its own. But the problem is that
some of the behaviour differences between the two line editors are
deliberate, for example the use of ^D to signal 'abort this prompt',
and the use of Escape as an alternative line-clearing command. So
TermLineEditor has a flags word that allows ldisc and terminal to set
it up differently. Also this lets me give the two TermLineEditors a
different vtable of callback functions, which is a convenient way for
terminal.c to get notified when a prompt has been answered.)

The new line editor still passes all the tests I wrote for the old
one. But it already has a couple of important improvements, both in
the area of UTF-8 handling:

Firstly, when we display a UTF-8 character on the terminal, we check
with the terminal how many character cells it occupied, and then if
the user deletes it again from the editing buffer, we can emit the
right number of backspace-space-backspace sequences. (The old ldisc
line editor incorrectly assumed all Unicode characters had terminal
with 1, partly because its buffer was byte- rather than character-
oriented and so it was more than enough work just finding where the
character _start_ was.)

Secondly, terminal.c's userpass line editor would never emit a byte in
the 80-BF range to the terminal at all, which meant that nontrivial
UTF-8 characters always came out as U+FFFD blobs!

test/fuzzterm.c

@@ -119,10 +119,7 @@ static const TermWinVtable fuzz_termwin_vt = {
 
 void ldisc_send(Ldisc *ldisc, const void *buf, int len, bool interactive) {}
 void ldisc_echoedit_update(Ldisc *ldisc) {}
-bool ldisc_has_input_buffered(Ldisc *ldisc) { return false; }
-LdiscInputToken ldisc_get_input_token(Ldisc *ldisc)
-{ unreachable("This fake ldisc never has any buffered input"); }
-void ldisc_enable_prompt_callback(Ldisc *ldisc, prompts_t *p)
+void ldisc_provide_userpass_le(Ldisc *ldisc, TermLineEditor *le)
 { unreachable("This fake ldisc should never be used for user/pass prompts"); }
 void modalfatalbox(const char *fmt, ...) { exit(0); }
 void nonfatal(const char *fmt, ...) { }

test/test_lineedit.c

@@ -656,9 +656,10 @@ static void test_edit(Mock *mk, bool echo)
     conf_set_bool(mk->conf, CONF_telnet_keyboard, false);
     ldisc_configure(mk->ldisc, mk->conf);
 
-    /* Test UTF-8 characters of various lengths and ensure deleting one
-     * deletes the whole character from the buffer (by pressing Return and
-     * seeing what gets sent) but only sends one BSB */
+    /* Test UTF-8 characters of various lengths and ensure deleting
+     * one deletes the whole character from the buffer (by pressing
+     * Return and seeing what gets sent) but sends a number of BSBs
+     * corresponding to the character's terminal width */
     mk->term->utf = true;
 
     ldisc_send(mk->ldisc, "\xC2\xA0\xC2\xA1", 4, false);
@@ -692,6 +693,29 @@ static void test_edit(Mock *mk, bool echo)
     EXPECT(mk, backend, PTRLEN_LITERAL("\xF0\x90\x80\x80\x0D\x0A"));
     reset(mk);
 
+    /* Double-width characters (Hangul, as it happens) */
+    ldisc_send(mk->ldisc, "\xEA\xB0\x80\xEA\xB0\x81", 6, false);
+    EXPECT(mk, backend, PTRLEN_LITERAL(""));
+    EXPECT_TERMINAL(mk, PTRLEN_LITERAL("\xEA\xB0\x80\xEA\xB0\x81"));
+    ldisc_send(mk->ldisc, "\x08", -1, false);
+    EXPECT(mk, backend, PTRLEN_LITERAL(""));
+    EXPECT_TERMINAL(mk, PTRLEN_LITERAL("\xEA\xB0\x80\xEA\xB0\x81"
+                                       "\x08 \x08\x08 \x08"));
+    ldisc_send(mk->ldisc, "\x0D", -1, false);
+    EXPECT(mk, backend, PTRLEN_LITERAL("\xEA\xB0\x80\x0D\x0A"));
+    reset(mk);
+
+    /* Zero-width characters */
+    ldisc_send(mk->ldisc, "\xE2\x80\x8B\xE2\x80\x8B", 6, false);
+    EXPECT(mk, backend, PTRLEN_LITERAL(""));
+    EXPECT_TERMINAL(mk, PTRLEN_LITERAL("\xE2\x80\x8B\xE2\x80\x8B"));
+    ldisc_send(mk->ldisc, "\x08", -1, false);
+    EXPECT(mk, backend, PTRLEN_LITERAL(""));
+    EXPECT_TERMINAL(mk, PTRLEN_LITERAL("\xE2\x80\x8B\xE2\x80\x8B"));
+    ldisc_send(mk->ldisc, "\x0D", -1, false);
+    EXPECT(mk, backend, PTRLEN_LITERAL("\xE2\x80\x8B\x0D\x0A"));
+    reset(mk);
+
     /* And reset back to non-UTF-8 mode and expect high-bit-set bytes
      * to be treated individually, as characters in a single-byte
      * charset. (In our case, given the test config, that will be