Unlike ssh-add, we can identify the key by its comment or by a prefix
of its fingerprint as well as using a public key file on disk. The
string given as an argument to -d is interpreted as whichever of those
things matches; disambiguating prefixes are available if needed.