In SSH-1 there's a function that takes a void * that it casts to the
state of the login layer. The corresponding function in SSH-2 casts it
to the state of a differently named layer, but I had still called the
parameter 'loginv'.
I happened to notice in passing that this function doesn't have any
tests (although it will have been at least somewhat tested by the
cmdgen interop test system).
This involved writing a wrapper that passes the passphrase and salt as
ptrlens, and I decided it made more sense to make the same change to
the original function too and adjust the call sites appropriately.
I derived a test case by getting OpenSSH itself to make an encrypted
key file, and then using the inputs and output from the password hash
operation that decrypted it again.
This fills in the remaining gap in the interactive prompt rework of
the proxy system in general. If you used the Telnet proxy with a
command containing %user or %pass, and hadn't filled in those
variables in the PuTTY config, then proxy/telnet.c would prompt you at
run time to enter the proxy auth details. But the local proxy command,
which uses the same format_telnet_command function, would not do that.
Now it does!
I've implemented this by moving the formatting of the proxy command
into a new module proxy/local.c, shared between both the Unix and
Windows local-proxy implementations. That module implements a
DeferredSocketOpener, which constructs the proxy command (prompting
first if necessary), and once it's constructed, hands it to a
per-platform function platform_setup_local_proxy().
So each platform-specific proxy function, instead of starting a
subprocess there and then and passing its details to make_fd_socket or
make_handle_socket, now returns a _deferred_ version of one of those
sockets, with the DeferredSocketOpener being the thing in
proxy/local.c. When that calls back to platform_setup_local_proxy(),
we actually start the subprocess and pass the resulting fds/handles to
the deferred socket to un-defer it.
A side effect of the rewrite is that when proxy commands are logged in
the Event Log, they now get the same amenities as in the Telnet proxy
type: the proxy password is sanitised out, and any difficult
characters are escaped.
Previously, a setup function returning one of these socket types (such
as platform_new_connection) had to do all its setup synchronously,
because if it was going to call make_fd_socket or make_handle_socket,
it had to have the actual fds or HANDLEs ready-made. If some kind of
asynchronous operation were needed before those fds become available,
there would be no way the function could achieve it, except by
becoming a whole extra permanent Socket wrapper layer.
Now there is, because you can make an FdSocket when you don't yet have
the fds, or a HandleSocket without the HANDLEs. Instead, you provide
an instance of the new trait 'DeferredSocketOpener', which is
responsible for setting in motion whatever asynchronous setup
procedure it needs, and when that finishes, calling back to
setup_fd_socket / setup_handle_socket to provide the missing pieces.
In the meantime, the FdSocket or HandleSocket will sit there inertly,
buffering any data the client might eagerly hand it via sk_write(),
and waiting for its setup to finish. When it does finish, buffered
data will be released.
In FdSocket, this is easy enough, because we were doing our own
buffering anyway - we called the uxsel system to find out when the fds
were readable/writable, and then wrote to them from our own bufchain.
So more or less all I had to do was make the try_send function do
nothing if the setup phase wasn't finished yet.
In HandleSocket, on the other hand, we're passing all our data to the
underlying handle-io.c system, and making _that_ deferrable in the
same way would be much more painful, because that's the place where
the scary threads live. So instead I've arranged it by replacing the
whole vtable, so that a deferred HandleSocket and a normal
HandleSocket are effectively separate trait implementations that can
share their state structure. And in fact that state struct itself now
contains a big anonymous union, containing one branch to go with each
vtable.
Nothing yet uses this system, but the next commit will do so.
This will mean that platform-specific proxy types will also be able to
set themselves up as child Interactors and prompt the user
interactively for passwords and the like.
NFC: nothing uses the new parameter yet.
Now it's done using the same code as in write_c_string_literal(), by
means of factoring the latter into a version that targets any old
BinarySink and a convenience wrapper taking a FILE *.
Now, when you resize the Event Log window, the list box grows in both
directions. Previously, as a side effect of the Columns-based layout,
it grew only horizontally.
I've arranged this by adding an extra wrinkle to the Columns layout
system, which allows you to tag _exactly one_ widget in the whole
container as the 'vexpand' widget. When the Columns is size-allocated
taller than its minimum height, the vexpand widget is given all the
extra space.
This technique ports naturally across all versions of GTK (since the
hard part is done in my own code). But it's limited: you can't set
more than one widget to be vexpand (which saves having to figure out
whether they're side by side and can expand in parallel, or vertically
separated and each have to get half the available extra space, etc).
And in complex layouts where the widget you really want to expand is
in a sub-Columns, there's no system for recursively searching down to
find it.
In other words, this is a one-shot bodge for the Event Log, and it
will want more work if we ever plan to extend it to list boxes in the
main config dialog.
Colin points out that in the migration to cmake, I accidentally
stopped putting some of the pre-built docs in the tarball - only the
man pages are still there.
This is a piece I forgot in the initial implementation of HTTP Digest:
an HTTP server can send _more than one_ authentication request header
(WWW-Authenticate for normal servers, Proxy-Authenticate for proxies),
and if it does, they're supposed to be treated as alternatives to each
other, so that the client chooses one to reply to.
I suppose that technically we were 'complying' with that spec already,
in that HttpProxyNegotiator would have read each new header and
overwritten all the fields set by the previous one, so we'd always
have gone with the last header presented by the server. But that seems
inelegant: better to choose the one we actually like best.
So now we do that. All the details of an auth header are moved out of
the main HttpProxyNegotiator struct into a sub-struct we can have
multiple copies of. Each new header is parsed into a fresh struct of
that kind, and then we can compare it with the previous one and decide
which we prefer.
The preference order, naturally, is 'more secure is better': Digest
beats Basic, and between two Digest headers, SHA-256 beats MD5. (And
anything beats a header we can't make sense of at all.)
Another side effect of this change is that a 407 response which
contains _no_ Proxy-Authenticate headers will trigger an error message
saying so, instead of just going with whatever happened to be left in
the relevant variables from the previous attempt.
While re-testing on Wayland after all this churn of the window
resizing code, I discovered that the window constantly came out a few
pixels too small, losing a character cell in width and height. This
stopped happening once I experimentally stopped setting geometry
hints.
Source-diving in GTK, it turns out that this is because the GDK
Wayland backend is applying the geometry hints to the size of the
window including 'margins', which are a very large extra space around
a window beyond even the visible 'non-client-area' furniture like the
title bar. And I have no idea how you find out the size of those
margins, so I can't allow for them in the geometry hints.
I also noticed that gtk_window_set_geometry_hints is removed in GTK 4,
which suggests that GTK upstream are probably not interested in
fiddling with them until they work more usefully (even if they would
agree with me that this is a bug in the first place, which I have no
idea). A simpler workaround is to avoid setting geometry hints at all
on any GDK backend other than X11.
So, that's what this commit does. On Wayland (or other backends), the
window can now be resized a pixel at a time, and if its size doesn't
work out to a whole number of character cells, then you just get some
dead space at the edges. Not especially nice, but better than the
alternatives I can see.
One other job the geometry hints were doing was to forbid resizing if
the backend sets the BACKEND_RESIZE_FORBIDDEN flag (which SUPDUP
does). That's now done at window creation time, via
gtk_window_set_resizable.
The window size set in the geometry hints when the backend has the
BACKEND_RESIZE_FORBIDDEN flag was computed in a simplistic way that
forgot to take account of window furniture like scrollbars and menu
bars. Now it's computed based on the rest of the geometry hints, which
are more accurate.
If the Seat that the pty backend is talking to starts to back up, then
we ought to temporarily stop reading from the pty device, to pass that
back-pressure on to whatever's running in the terminal.
Previously, this didn't matter because a Seat running a GUI terminal
never backed up anyway. But now it does, so we should support it all
the way through the system.
Normally, the GTK code runs toplevel callbacks from a GTK 'idle
function'. But those mean what they say: they are considered
low-priority, to be run _only_ when the system is idle - so they can
fail to run at all in conditions of a steady stream of higher-priority
things, e.g. something is throwing data at the application so fast
that every main-loop iteration finds a readable fd.
And that's not good, because _we_ don't think our callbacks are
low-priority: they do a lot of really important work like redrawing
the window. So if they never get round to happening, PuTTY or pterm
can appear to lock up.
Simple solution to that one: whenever we process a select notification
on any fd, we _also_ call run_toplevel_callbacks(). Then our callbacks
are bound to happen reasonably regularly.
The return value of term_data() is used as the return value from the
GUI-terminal versions of the Seat output method, which means backends
will take it to be the amount of standard-output data currently
buffered, and exert back-pressure on the remote peer if it gets too
big (e.g. by ceasing to extend the window in that particular SSH-2
channel).
Historically, as a comment in term_data() explained, we always just
returned 0 from that function, on the basis that we were processing
all the terminal data through our terminal emulation code immediately,
and never retained any of it in the buffer at all. If the terminal
emulation code were to start running slowly, then it would slow down
the _whole_ PuTTY system, due to single-threadedness, and
back-pressure of a sort would be exerted on the remote by it simply
failing to get round to reading from the network socket. But by the
time we got back to the top level of term_data(), we'd have finished
reading all the data we had, so it was still appropriate to return 0.
That comment is still correct if you're thinking about the limiting
factor on terminal data processing being the CPU usage in term_out().
But now that's no longer the whole story, because sometimes we leave
data in term->inbuf without having processed it: during drag-selects
in the terminal window, and (just introduced) while waiting for the
response to a pending window resize request. For both those reasons,
we _don't_ always have a buffer size of zero when we return from
term_data().
So now that hole in our buffer size management is filled in:
term_data() returns the true size of the remaining unprocessed
terminal output, so that back-pressure will be exerted if the terminal
is currently not consuming it. And when processing resumes and we
start to clear our backlog, we call backend_unthrottle to let the
backend know it can relax the back-pressure if necessary.
Previously, the Windows implementation of win_request_resize would
call term_size() to tell the terminal about its new size, _before_
calling SetWindowPos to actually change the window size.
If SetWindowPos ends up not getting the exact window size it asks for,
Windows notifies the application by sending back a WM_SIZE message -
synchronously, by calling back to the window procedure from within
SetWindowPos. So after the first over-optimistic term_size(), the
WM_SIZE handler would trigger a second one, resetting the terminal
again to the _actual_ size.
This was more or less harmless, since current handling of terminal
resizes is such that no terminal data gets too confused: any lines
pushed into the scrollback by the first term_size will be pulled back
out by the second one if needed (or vice versa), and since commit
271de3e4ec, individual termlines are not eagerly truncated by
resizing them twice.
However, it's more work than we really wanted to be doing, and it
seems presumptuous to send term_size() before we know it's right! So
now we send term_size() after SetWindowPos returns, unless it already
got sent by a re-entrant call to the WM_SIZE handler _before_
SetWindowPos returned. That way, the terminal should get exactly one
term_size() call in response to win_request_resize(), whether it got
the size it was expecting or not.
This is the payoff from the last few commits of refactoring. It fixes
the following race-condition bug in terminal application redraw:
* server sends a window-resizing escape sequence
* terminal requests a window resize from the front end
* server sends further escape sequences to perform a redraw of some
full-screen application, which assume that the window resize has
occurred and the window is already its new size
* terminal processes all those sequences in the context of the old
window size, while the front end is still thinking
* window resize completes in the front end and term_size() tells the
terminal it now has its new size, but it's too late, the screen
redraw has made a total mess.
(Perhaps the server might even send its window resize + followup
redraw all in one SSH packet, so that it's all queued in term->inbuf
in one go.)
As far as I can see, handling of this case has been broken more or
less forever in the GTK frontend (where window resizes are inherently
asynchronous due to the way X11 works, and we've never done anything
to compensate for that). On Windows, where window size is changed via
SetWindowPos which is synchronous, it used to work, but broke in
commit d74308e90e (i.e. between 0.74 and 0.75), which made all
the ancillary window updates run on the same delayed-action timer as
ordinary text display.
So, it's time to fix it, and I think now I should be able to fix it in
GTK as well as on Windows.
Now, as soon as we've set the term->win_resize_pending flag (in
response to a resize escape sequence), the next return to the top of
the main loop in term_out will terminate output processing early,
leaving any further terminal data still in the term->inbuf bufchain.
Once we get a term_size() callback from the front end telling us our
new size, we reset term->win_resize_pending, which unblocks output
processing again, and we also queue a toplevel callback to have
another try at term_out() so that it will be unblocked promptly.
To implement this I've changed term->win_resize_pending from a bool
into a three-state enumeration, so that we can tell the difference
between 'pending' in the sense of not yet having sent our resize
request to the frontend, and in the sense of waiting for the frontend
to reply. That way, a window resize from the GUI user at least won't
be mistaken for the response to our resize request if it arrives in
the former state. (It can still be mistaken for one in the latter
case, but if the user is resizing the window at the same time as the
server-side application is doing critically size-dependent redrawing,
I don't think there can be any reasonable expectation of nothing going
wrong.)
As mentioned in the previous commit, some failure modes under X11 (in
particular the window manager process getting wedged in some way) can
result in no response being received to a ConfigureWindow request. In
that situation, it seems to me that we really _shouldn't_ sit there
waiting forever - perhaps it's technically the WM's fault and not
ours, but what kind of X window are you most likely to want to use to
do emergency WM repair? A terminal window, of course, so it would be
exceptionally unhelpful to make any terminal window stop working
completely in this situation! Hence, there's a fallback timeout in
terminal.c, so that if we don't receive a response in _too_ long,
we'll assume one is not forthcoming, and resume processing terminal
data at the old window size. The fallback timeout is set to 5 seconds,
following existing practice in libXt (DEFAULT_WM_TIMEOUT).
When the terminal asks its TermWin for a resize, the resize operation
happens asynchronously (or can do), and sooner or later, the terminal
will see a term_size() telling it the resize has actually taken
effect.
If the resize _doesn't_ take effect for any reason - e.g. because the
window is maximised, or because the X11 window manager is a tiling one
which will refuse requests to change the window size anyway - then the
terminal never got any explicit notification of refusal to resize. Now
it should, in as many cases as I can arrange.
One obvious case of this is the early exit I recently added to
gtkwin_request_resize() when the window is known to be in a maximised
or tiled state preventing a resize: in that situation, when our own
code knows we're not even attempting the resize, we also queue a
toplevel callback to tell the terminal so.
The more interesting case is when the request is refused for a reason
GTK _didn't_ know in advance, e.g. because the user is running an X11
tiling window manager such as i3, which generally refuses windows'
resize requests. In X11, if a window manager refuses an attempt to
change the window's size via ConfigureWindow, ICCCM says it should
respond by sending a synthetic ConfigureNotify event restating the
same size. Such no-op configure events do reach the "configure_event"
handler in a GTK program, but they weren't previously getting as far
as term_size(), because the call to term_size() was triggered from the
GTK "size_allocate" event on the GtkDrawingArea inside the window (via
drawing_area_setup()), so GTK would detect that nothing had changed.
Now we queue a last-ditch toplevel callback which ensures that if the
configure event doesn't also cause a size_allocate and a call to
drawing_area_setup(), then we cause one of our own once the dust has
settled. And drawing_area_setup(), in turn, now unconditionally calls
term_size() even if the size is the same as it was last time, instead
of taking an early exit. (It still does take an early exit to avoid
unnecessary faffing with Cairo surfaces etc, but _after_ term_size()).
This won't be 100% reliable, because it's the window manager's
responsibility to send those synthetic ConfigureNotify events, and a
window manager is a fallible process which could get into a stuck
state. But it covers all the cases I know of that _can_ be sensibly
covered - so now, when terminal.c asks the front end to resize the
window, it ought to find out in a timely manner whether or not that
has happened, in _almost_ all cases.
There's no actual need to copy the data from term->inbuf into a local
variable inside term_out(). We can simply store a pointer and length,
and use the data _in situ_ - as long as we remember how much of it
we've used, and bufchain_consume() it when the routine exits.
Getting rid of that awkward and size-limited local array should
marginally improve performance. But also, it opens up the possibility
to suddenly suspend handling of terminal data and leave everything not
yet processed in the bufchain, because now we never remove anything
from the bufchain until _after_ it's been processed, so there's no
need to awkwardly push the unused segment of localbuf[] back on to the
front of the bufchain if we need to do that.
NFC, but as usual, I have a plan to use the new capability in a
followup commit.
This tiny refactoring makes a convenient function for setting all the
'pending' flags and triggering a callback for the next window update.
This saves a bit of code, but that's not really the main point (or
else I'd have done the same to all the other similar things like
window moves). The point is that in a future commit I'm going to want
to do an extra thing on every server-controlled window resize, and
this refactoring gives me a single place to put that extra action.
This tiny refactoring replaces three identical checks at call sites,
not all as well commented as each other, with a check in just one
place with the best of the three comments.
This is another thing that seems harmless on X11 but causes window
redraws to semipermanently stop happening on Wayland: if we try to
gtk_window_resize() a window that is maximised at the time, then
something mysterious goes wrong and we stop ever getting "draw" events.
The event should return a 'gboolean', indicating whether the event
needs propagating any further. We were returning void, which meant
that the default handling might be accidentally suppressed.
On Wayland, this had the particularly nasty effect that window redraws
would stop completely if you maximised the terminal window.
(By trial and error I found that this stopped happening if I removed
GDK_HINT_RESIZE_INC from the geometry hints, from which I guess that
the default window-state-event handler is doing something important
relating to that hint, or would have been if we hadn't accidentally
suppressed it. But the bug is clearly here.)
These alternate frontends using the GtkApplication class don't work
before GTK3, because the GtkApplication class didn't exist. In the old
mkfiles.pl system, the simplest way to prevent a build failure was to
just compile them anyway but make them reduce to a stub main(). But
now, with the new library-based code organisation, library search
order issues mean that these applications won't build at all.
Happily, with cmake, it's also easy to simply omit these binaries from
the build completely depending on our GTK version.
In commit 2675f9578d, when I added 'stripctrl' sanitisation to
many uses of filenames throughout the file transfer tools, I made a
copy-and-paste error in the message 'remote host sent a compound
pathname X / renaming local file to Y' in which both X and Y were the
same pathname. Oops. Fixed the other one.
At some point while setting up the utils subdirectory, I apparently
only got half way through renaming miscucs.c to dup_mb_to_wc.c: I
created the new copy of the file, but I didn't delete the old one, I
didn't mention it in utils/CMakeLists.txt, and I didn't change the
comment at the top.
Now done all three, so we now have just one copy of this utility
module.
I recently encountered a paper [1] which catalogues all kinds of
things that can go wrong when one party in a discrete-log system
invents a prime and the other party chooses an exponent. In
particular, some choices of prime make it reasonable to use a short
exponent to save time, but others make that strategy very bad.
That paper is about the ElGamal encryption scheme used in OpenPGP,
which is basically integer Diffie-Hellman with one side's key being
persistent: a shared-secret integer is derived exactly as in DH, and
then it's used to communicate a message integer by simply multiplying
the shared secret by the message, mod p.
I don't _know_ that any problem of this kind arises in the SSH usage
of Diffie-Hellman: the standard integer DH groups in SSH are safe
primes, and as far as I know, the usual generation of prime moduli for
DH group exchange also picks safe primes. So the short exponents PuTTY
has been using _should_ be OK.
However, the range of imaginative other possibilities shown in that
paper make me nervous, even so! So I think I'm going to retire the
short exponent strategy, on general principles of overcaution.
This slows down 4096-bit integer DH by about a factor of 3-4 (which
would be worse if it weren't for the modpow speedup in the previous
commit). I think that's OK, because, firstly, computers are a lot
faster these days than when I originally chose to use short exponents,
and secondly, more and more implementations are now switching to
elliptic-curve DH, which is unaffected by this change (and with which
we've always been using maximum-length exponents).
[1] On the (in)security of ElGamal in OpenPGP. Luca De Feo, Bertram
Poettering, Alessandro Sorniotti. https://eprint.iacr.org/2021/923
Instead of the basic square-and-multiply strategy which requires a
square and a multiply per exponent bit (i.e. two modular
multiplications per bit in total), we instead reduce to a square per
exponent bit and an extra multiply only every 5 bits, because the
value we're multiplying in is derived from 5 of the exponent bits at
once via a table lookup.
To avoid the obvious side-channel leakage of a literal table lookup,
we read the whole table every time, mp_selecting the right value into
the multiplication input. This isn't as slow as it sounds when the
alternative is four entire modular multiplications! In my testing,
this commit speeds up large modpows by a factor of just over 1.5, and
it still gets a clean pass from 'testsc'.
It's unquestionably a test program, and I'm generally clearing those
out of the top level. I only missed it in the last clearout because I
was looking for things with 'test' in the name.
They were there to work around that annoying feature of VS's
preprocessor when it expands __VA_ARGS__ into the argument list of
another macro. But I've just thought of a workaround that I can apply
in testcrypt.c itself, so that those parens don't have to appear in
every function definition in the header file.
The trick is, instead of writing
destination_macro(__VA_ARGS__)
you instead write
JUXTAPOSE(destination_macro, (__VA_ARGS__))
where JUXTAPOSE is defined to be a macro that simply expands its two
arguments next to each other:
#define JUXTAPOSE(first, second) first second
This works because the arguments to JUXTAPOSE get macro-expanded
_before_ passing them to JUXTAPOSE itself - the same reason that the
standard tricks with STR_INNER and CAT_INNER work (as seen in defs.h
here). So this defuses the magic behaviour of commas expanded from
__VA_ARGS__, and causes the destination macro to get all its arguments
in the expected places again.
I was dubious about it to begin with, when I found that RFC 7616's
example seemed to be treating it as a 256-bit truncation of SHA-512,
and not the thing FIPS 180-4 section 6.7 specifies as "SHA-512/256"
(which also changes the initial hash state). Having failed to get a
clarifying response from the RFC authors, I had the idea this morning
of testing other HTTP clients to see what _they_ thought that hash
function meant, and then at least I could go with an existing
in-practice consensus.
There is no in-practice consensus. Firefox doesn't support that
algorithm at all (but they do support SHA-256); wget doesn't support
anything that RFC 7616 added to the original RFC 2617. But the prize
for weirdness goes to curl, which does accept the name "SHA-512-256"
and ... treats it as an alias for SHA-256!
So I think the situation among real clients is too confusing to even
try to work with, and I'm going to stop adding to it. PuTTY will
follow Firefox's policy: if a proxy server asks for SHA-256 digests
we'll happily provide them, but if they ask for SHA-512-256 we'll
refuse on the grounds that it's not clear enough what it means.
This commit replaces all those fiddly little linking modules
(be_all.c, be_none.c, be_ssh.c etc) with a single source file
controlled by ifdefs, and introduces a function be_list() in
setup.cmake that makes it easy to compile a version of it appropriate
to each application.
This is a net reduction in code according to 'git diff --stat', even
though I've introduced more comments. It also gets rid of another pile
of annoying little source files in the top-level directory that didn't
deserve to take up so much room in 'ls'.
More concretely, doing this has some maintenance advantages.
Centralisation means less to maintain (e.g. n_ui_backends is worked
out once in a way that makes sense everywhere), and also, 'appname'
can now be reliably set per program. Previously, some programs got the
wrong appname due to sharing the same linking module (e.g. Plink had
appname="PuTTY"), which was a latent bug that would have manifested if
I'd wanted to reuse the same string in another context.
One thing I've changed in this rework is that Windows pterm no longer
has the ConPTY backend in its backends[]: it now has an empty one. The
special be_conpty.c module shouldn't really have been there in the
first place: it was used in the very earliest uncommitted drafts of
the ConPTY work, where I was using another method of selecting that
backend, but now that Windows pterm has a dedicated
backend_vt_from_conf() that refers to conpty_backend by name, it has
no need to live in backends[] at all, just as it doesn't have to in
Unix pterm.
I'm actually quite surprised there was only _one_ copy of each of
these standard macros in the code base, given my general habit of
casually redefining them anywhere I need them! But each one was in a
silly place. Moved them up to the top level where they're available
globally.
While I'm in the mood for cleaning up the top-level directory here:
all the 'nostuff.c' files have moved into a new 'stubs' directory, and
I broke up be_misc.c into smaller modules that can live in 'utils'.
Now testcrypt has _two_ header files, that's more files than I want at
the top level, so I decided to move it.
It has a good claim to live in either 'test' or 'crypto', but in the
end I decided it wasn't quite specific enough to crypto (it already
also tests things in keygen and proxy), and also, the Python half of
the mechanism already lives in 'test', so it can live alongside that.
Having done that, it seemed silly to leave testsc and testzlib at the
top level: those have 'test' in the names as well, so they can go in
the test subdir as well.
While I'm renaming, also renamed testcrypt.h to testcrypt-func.h to
distinguish it from the new testcrypt-enum.h.
This allows the Python side of testcrypt to check in advance if a
given string is a valid element of an enumeration, and if not, cleanly
throw a Python-level exception without terminating the testcrypt
subprocess.
Should be useful in both manual use (when I'm trying something out by
hand and make a typo or misremember a spelling), and automated use (if
I make the same kind of error in cryptsuite.py then the exception dump
will make more sense).
In order to do this, the new handle_checkenum() function has to
recognise all the enumerated types by name and match them up to their
lookup functions - which is just the kind of thing that can now be
done easily be reincluding testcrypt-enum.h with different #defines.
Now all the lookup functions like get_hashalg() and
get_primegenpolicy() are autogenerated by macro expansion from a
header a little like the existing testcrypt.h. But the macros are much
simpler this time.
This doesn't really change anything, _except_ that now I'll be able to
reinclude the same header to do other things with the list of
enumerated types.
This slightly simplifies the lookup function get_dh_group(), but
mostly, the point is to make it more similar to the other lookup
functions, because I'm planning to have those autogenerated.
Now those names appear in help files, I thought it was worth giving
them a read-through and spotting any really obviously confusing or
wrong ones. Quite a few make more sense in the original context of C
than in the derived Python (e.g. 'BinarySink *bs' as a place to write
output to makes sense, but the output 'val_string bs' is less
helpful).
A couple were so confusing that I also corrected them in the original
C, notably the misuse of 'wc' for the elliptic curve point input to
ecc_weierstrass_point_copy. ('wc' in that section of the code is
normally a parameter describing a whole curve.)
Now that testcrypt.py parses it using a C-like lexer, we don't have
the constraint any more that each function definition has to live on
exactly one source line for the sake of the previous line-by-line
parser. So we can put whitespace wherever we like, and reformat for
legibility!
I've done the initial pass of this using clang-format, which will
optimise for (a) everything fitting in 80 columns, and (b) individual
ARG(...) specifications generally not being broken across lines. Those
seem like a good combination to me.
Making a virtue of the necessity of adding parameter names to
testcrypt.h a couple of commits ago, we can now use those names to
improve diagnostics, so that if you use the wrong type in a Python
function call the error message will tell you the name as well as the
index of the offending argument.
Also, the repr() text for the function itself will now print a full
prototype (albeit in a nasty hybrid of C, Python and testcrypt.h
syntax) which shows all the parameter names. That should be handy when
trying to remember the order of arguments at the REPL prompt.
FUNC_WRAPPED is an alternative keyword to FUNC which you can use to
introduce a function specification in testcrypt.h, indicating that the
function is _not_ the one of the same name used in the main PuTTY
code, but instead a wrapper on it in testcrypt.c whose API was
reworked to be more friendly to translation into Python.
There are a lot of those wrappers already, and previously they passed
without comment in testcrypt.h, and were put into service by #defining
over the top of each name before expanding the marshalling functions.
Now, all those #defines are gone, because the use of FUNC_WRAPPED in
testcrypt.h is enough to clue in the marshalling wrapper to be
generated with a call to foo_wrapper() instead of foo().
Mostly the purpose of this is to make testcrypt.h a bit more
self-documenting: if you see FUNC_WRAPPED, you know not to be confused
by the Python and C function definitions totally failing to match.
Yesterday's commit 52ee636b09 which further extended the huge
pile of arity-specific annoying wrapper macros pushed me over the edge
and inspired me to give some harder thought to finding a way to handle
all arities at once. And this time I found one!
The new technique changes the syntax of the function specifications in
testcrypt.h. In particular, they now have to specify a _name_ for each
parameter as well as a type, because the macros generating the C
marshalling wrappers will need a structure field for each parameter
and cpp isn't flexible enough to generate names for those fields
automatically. Rather than tediously name them arg1, arg2 etc, I've
reused the names of the parameters from the prototypes or definitions
of the underlying real functions (via a one-off auto-extraction
process starting from the output of 'clang -Xclang -dump-ast' plus
some manual polishing), which means testcrypt.h is now a bit more
self-documenting.
The testcrypt.py end of the mechanism is rewritten to eat the new
format. Since it's got more complicated syntax and nested parens and
things, I've written something a bit like a separated lexer/parser
system in place of the previous crude regex matcher, which should
enforce that the whole header file really does conform to the
restricted syntax it has to fit into.
The new system uses a lot less code in testcrypt.c, but I've made up
for that by also writing a long comment explaining how it works, which
was another thing the previous system lacked! Similarly, the new
testcrypt.h has some long-overdue instructions at the top.
Spotted in passing: the cryptsuite test functions iterate 'hashname'
through all the available implementations of SHA-512 (or SHA-384), but
then, in each iteration, ignore that loop variable completely and
always test the default algorithm. So on a platform where more than
one implementation is available, we were only actually testing one of
them. Oops!
In http.c, this drops in reasonably neatly alongside the existing
support for Basic, now that we're waiting for an initial 407 response
from the proxy to tell us which auth mechanism it would prefer to use.
The rest of this patch is mostly contriving to add testcrypt support
for the function in cproxy.c that generates the complicated output
header to go in the HTTP request: you need about a dozen assorted
parameters, the actual response hash has two more hashes in its
preimage, and there's even an option to hash the username as well if
necessary. Much more complicated than CHAP (which is just plain
HMAC-MD5), so it needs testing!
Happily, RFC 7616 comes with some reasonably useful test cases, and
I've managed to transcribe them directly into cryptsuite.py and
demonstrate that my response-generator agrees with them.
End-to-end testing of the whole system was done against Squid 4.13
(specifically, the squid package in Debian bullseye, version 4.13-10).
Now, we always try an initial CONNECT request with no auth at all, and
wait for the proxy to reject it before sending a second try with
auth.
That way, we can wait to see what _kind_ of authentication the proxy
requests, which will enable us to support something more secure than
Basic, such as HTTP Digest.
(I mean, it would _work_ to try Basic in request #1 and then retrying
with Digest in #2 when the proxy asks for it. But if the aim of using
Digest is to avoid sending the password in cleartext, it defeats the
entire purpose to have sent it in cleartext anyway by the time you
realise the server is prepared to do something better!)
