This brings various concrete advantages over the previous system:
- consistent support for out-of-tree builds on all platforms
- more thorough support for Visual Studio IDE project files
- support for Ninja-based builds, which is particularly useful on
Windows where the alternative nmake has no parallel option
- a really simple set of build instructions that work the same way on
all the major platforms (look how much shorter README is!)
- better decoupling of the project configuration from the toolchain
configuration, so that my Windows cross-building doesn't need
(much) special treatment in CMakeLists.txt
- configure-time tests on Windows as well as Linux, so that a lot of
ad-hoc #ifdefs second-guessing a particular feature's presence from
the compiler version can now be replaced by tests of the feature
itself
Also some longer-term software-engineering advantages:
- other people have actually heard of CMake, so they'll be able to
produce patches to the new build setup more easily
- unlike the old mkfiles.pl, CMake is not my personal problem to
maintain
- most importantly, mkfiles.pl was just a horrible pile of
unmaintainable cruft, which even I found it painful to make changes
to or to use, and desperately needed throwing in the bin. I've
already thrown away all the variants of it I had in other projects
of mine, and was only delaying this one so we could make the 0.75
release branch first.
This change comes with a noticeable build-level restructuring. The
previous Recipe worked by compiling every object file exactly once,
and then making each executable by linking a precisely specified
subset of the same object files. But in CMake, that's not the natural
way to work - if you write the obvious command that puts the same
source file into two executable targets, CMake generates a makefile
that compiles it once per target. That can be an advantage, because it
gives you the freedom to compile it differently in each case (e.g.
with a #define telling it which program it's part of). But in a
project that has many executable targets and had carefully contrived
to _never_ need to build any module more than once, all it does is
bloat the build time pointlessly!
To avoid slowing down the build by a large factor, I've put most of
the modules of the code base into a collection of static libraries
organised vaguely thematically (SSH, other backends, crypto, network,
...). That means all those modules can still be compiled just once
each, because once each library is built it's reused unchanged for all
the executable targets.
One upside of this library-based structure is that now I don't have to
manually specify exactly which objects go into which programs any more
- it's enough to specify which libraries are needed, and the linker
will figure out the fine detail automatically. So there's less
maintenance to do in CMakeLists.txt when the source code changes.
But that reorganisation also adds fragility, because of the trad Unix
linker semantics of walking along the library list once each, so that
cyclic references between your libraries will provoke link errors. The
current setup builds successfully, but I suspect it only just manages
it.
(In particular, I've found that MinGW is the most finicky on this
score of the Windows compilers I've tried building with. So I've
included a MinGW test build in the new-look Buildscr, because
otherwise I think there'd be a significant risk of introducing
MinGW-only build failures due to library search order, which wasn't a
risk in the previous library-free build organisation.)
In the longer term I hope to be able to reduce the risk of that, via
gradual reorganisation (in particular, breaking up too-monolithic
modules, to reduce the risk of knock-on references when you included a
module for function A and it also contains function B with an
unsatisfied dependency you didn't really need). Ideally I want to
reach a state in which the libraries all have sensibly described
purposes, a clearly documented (partial) order in which they're
permitted to depend on each other, and a specification of what stubs
you have to put where if you're leaving one of them out (e.g.
nocrypto) and what callbacks you have to define in your non-library
objects to satisfy dependencies from things low in the stack (e.g.
out_of_memory()).
One thing that's gone completely missing in this migration,
unfortunately, is the unfinished MacOS port linked against Quartz GTK.
That's because it turned out that I can't currently build it myself,
on my own Mac: my previous installation of GTK had bit-rotted as a
side effect of an Xcode upgrade, and I haven't yet been able to
persuade jhbuild to make me a new one. So I can't even build the MacOS
port with the _old_ makefiles, and hence, I have no way of checking
that the new ones also work. I hope to bring that port back to life at
some point, but I don't want it to block the rest of this change.
The number of people has been steadily increasing who read our source
code with an editor that thinks tab stops are 4 spaces apart, as
opposed to the traditional tty-derived 8 that the PuTTY code expects.
So I've been wondering for ages about just fixing it, and switching to
a spaces-only policy throughout the code. And I recently found out
about 'git blame -w', which should make this change not too disruptive
for the purposes of source-control archaeology; so perhaps now is the
time.
While I'm at it, I've also taken the opportunity to remove all the
trailing spaces from source lines (on the basis that git dislikes
them, and is the only thing that seems to have a strong opinion one
way or the other).
Apologies to anyone downstream of this code who has complicated patch
sets to rebase past this change. I don't intend it to be needed again.
Now there are lines in the checksum lists that don't correspond
directly to files we uploaded as part of the main release process,
release.pl needs to ignore them to avoid embarrassing upload failures.
chiark's ftp server sometimes randomly refuses downloads. In the case
where this happens at postcheck time, this isn't really
release-blocking (all the files have been test-downloaded by precheck
already, so the main aim at this stage is to check that the 'latest'
symlink points to the right place, and even one or two successful
downloads are good enough to confirm that in practice). So now I can
add --no-ftp to the postcheck command line if that makes my life
easier.
This should have been part of commit ea0ab1c82; it's part of the
general revamp that we regenerate the autoconf files ourselves in a
clean directory - so we don't depend on them being present, but we
also don't depend on them being _absent_ either.
But when I made that commit my immediate priority was to get --setver
to work from a completely clean checkout, not from one already
littered with cruft, so I didn't check quite as carefully that my
changes fixed the problem in the latter case too :-)
Previously, it demanded that your checkout was in a state where you
had run autoconf but not configure; so if you previously did have a
Makefile then you had to 'make distclean' to remove it, whereas if you
previously had no generated files at all (e.g. starting from a
completely clean checkout) then you had to run mkfiles.pl and
mkauto.sh to generate 'configure'.
This is obviously confusing, and moreover, the dependence on prior
generated files is fragile and prone to them having been generated
wrong. Adjusted the script so that it uses 'git archive' to get a
clean directory containing only the version-controlled files, and then
runs scripts itself to get that directory into the state it wants.
The main tartarus.org host has changed since the last release, so we
now have to upload things to somewhere different. Updated the release
automation in release.pl, and all the mentions of atreus in the manual
checklist too.
The code that copies the link maps of the release Windows builds into
the place I store them for later debugging should now not
embarrassingly look in the wrong place when we make our first
post-VS2015 release.
I've added extra modes to release.pl which should automate the more
tedious parts of the deployment phase: uploading the release build to
all the places it needs to go, checking its integrity once it gets
there, verifying that everything can be downloaded again usefully,
checking content-types etc.
The new version should check more thoroughly (it checks the whole FTP
and HTTP download directories, so it will spot errors like failing to
update the FTP 'latest' symlink), and take fewer commands to run.
I've added a few sample shell commands in the upload procedure (mostly
so that I don't have to faff about remembering how rsync trailing
slashes work every time), and also written a script called
'release.pl', which automates the updating of the version number in
all the various places it needs to be done and also ensures the PSCP
and Plink transcripts in the docs will match the release itself.
