nhyatt/putty-source
1
0
Fork 0
mirror of https://git.tartarus.org/simon/putty.git synced 2025-01-09 17:38:00 +00:00
Code Issues Projects Releases Wiki Activity
f58ddf26fe
putty-source/test/testcrypt.py

442 lines
17 KiB
Python
Raw Normal View History

New test system for mp_int and cryptography. I've written a new standalone test program which incorporates all of PuTTY's crypto code, including the mp_int and low-level elliptic curve layers but also going all the way up to the implementations of the MAC, hash, cipher, public key and kex abstractions. The test program itself, 'testcrypt', speaks a simple line-oriented protocol on standard I/O in which you write the name of a function call followed by some inputs, and it gives you back a list of outputs preceded by a line telling you how many there are. Dynamically allocated objects are assigned string ids in the protocol, and there's a 'free' function that tells testcrypt when it can dispose of one. It's possible to speak that protocol by hand, but cumbersome. I've also provided a Python module that wraps it, by running testcrypt as a persistent subprocess and gatewaying all the function calls into things that look reasonably natural to call from Python. The Python module and testcrypt.c both read a carefully formatted header file testcrypt.h which contains the name and signature of every exported function, so it costs minimal effort to expose a given function through this test API. In a few cases it's necessary to write a wrapper in testcrypt.c that makes the function look more friendly, but mostly you don't even need that. (Though that is one of the motivations between a lot of API cleanups I've done recently!) I considered doing Python integration in the more obvious way, by linking parts of the PuTTY code directly into a native-code .so Python module. I decided against it because this way is more flexible: I can run the testcrypt program on its own, or compile it in a way that Python wouldn't play nicely with (I bet compiling just that .so with Leak Sanitiser wouldn't do what you wanted when Python loaded it!), or attach a debugger to it. I can even recompile testcrypt for a different CPU architecture (32- vs 64-bit, or even running it on a different machine over ssh or under emulation) and still layer the nice API on top of that via the local Python interpreter. All I need is a bidirectional data channel.
2019-01-01 19:08:37 +00:00
import sys
import os
import numbers
import subprocess
import re
Rewrite the testcrypt.c macro system. Yesterday's commit 52ee636b092c199 which further extended the huge pile of arity-specific annoying wrapper macros pushed me over the edge and inspired me to give some harder thought to finding a way to handle all arities at once. And this time I found one! The new technique changes the syntax of the function specifications in testcrypt.h. In particular, they now have to specify a _name_ for each parameter as well as a type, because the macros generating the C marshalling wrappers will need a structure field for each parameter and cpp isn't flexible enough to generate names for those fields automatically. Rather than tediously name them arg1, arg2 etc, I've reused the names of the parameters from the prototypes or definitions of the underlying real functions (via a one-off auto-extraction process starting from the output of 'clang -Xclang -dump-ast' plus some manual polishing), which means testcrypt.h is now a bit more self-documenting. The testcrypt.py end of the mechanism is rewritten to eat the new format. Since it's got more complicated syntax and nested parens and things, I've written something a bit like a separated lexer/parser system in place of the previous crude regex matcher, which should enforce that the whole header file really does conform to the restricted syntax it has to fit into. The new system uses a lot less code in testcrypt.c, but I've made up for that by also writing a long comment explaining how it works, which was another thing the previous system lacked! Similarly, the new testcrypt.h has some long-overdue instructions at the top.
2021-11-21 10:27:30 +00:00
import string
testcrypt / cryptsuite: another set of Python 3 fixes. One of these days I'll manage not to mess this up in every new test I add ... perhaps.
2019-01-23 23:40:32 +00:00
import struct
New test system for mp_int and cryptography. I've written a new standalone test program which incorporates all of PuTTY's crypto code, including the mp_int and low-level elliptic curve layers but also going all the way up to the implementations of the MAC, hash, cipher, public key and kex abstractions. The test program itself, 'testcrypt', speaks a simple line-oriented protocol on standard I/O in which you write the name of a function call followed by some inputs, and it gives you back a list of outputs preceded by a line telling you how many there are. Dynamically allocated objects are assigned string ids in the protocol, and there's a 'free' function that tells testcrypt when it can dispose of one. It's possible to speak that protocol by hand, but cumbersome. I've also provided a Python module that wraps it, by running testcrypt as a persistent subprocess and gatewaying all the function calls into things that look reasonably natural to call from Python. The Python module and testcrypt.c both read a carefully formatted header file testcrypt.h which contains the name and signature of every exported function, so it costs minimal effort to expose a given function through this test API. In a few cases it's necessary to write a wrapper in testcrypt.c that makes the function look more friendly, but mostly you don't even need that. (Though that is one of the motivations between a lot of API cleanups I've done recently!) I considered doing Python integration in the more obvious way, by linking parts of the PuTTY code directly into a native-code .so Python module. I decided against it because this way is more flexible: I can run the testcrypt program on its own, or compile it in a way that Python wouldn't play nicely with (I bet compiling just that .so with Leak Sanitiser wouldn't do what you wanted when Python loaded it!), or attach a debugger to it. I can even recompile testcrypt for a different CPU architecture (32- vs 64-bit, or even running it on a different machine over ssh or under emulation) and still layer the nice API on top of that via the local Python interpreter. All I need is a bidirectional data channel.
2019-01-01 19:08:37 +00:00
from binascii import hexlify
Migrate all Python scripts to Python 3. Most of them are now _mandatory_ P3 scripts, because I'm tired of maintaining everything to be compatible with both versions. The current exceptions are gdb.py (which has to live with whatever gdb gives it), and kh2reg.py (which is actually designed for other people to use, and some of them might still be stuck on P2 for the moment).
2020-03-04 21:23:49 +00:00
assert sys.version_info[:2] >= (3,0), "This is Python 3 code"
New test system for mp_int and cryptography. I've written a new standalone test program which incorporates all of PuTTY's crypto code, including the mp_int and low-level elliptic curve layers but also going all the way up to the implementations of the MAC, hash, cipher, public key and kex abstractions. The test program itself, 'testcrypt', speaks a simple line-oriented protocol on standard I/O in which you write the name of a function call followed by some inputs, and it gives you back a list of outputs preceded by a line telling you how many there are. Dynamically allocated objects are assigned string ids in the protocol, and there's a 'free' function that tells testcrypt when it can dispose of one. It's possible to speak that protocol by hand, but cumbersome. I've also provided a Python module that wraps it, by running testcrypt as a persistent subprocess and gatewaying all the function calls into things that look reasonably natural to call from Python. The Python module and testcrypt.c both read a carefully formatted header file testcrypt.h which contains the name and signature of every exported function, so it costs minimal effort to expose a given function through this test API. In a few cases it's necessary to write a wrapper in testcrypt.c that makes the function look more friendly, but mostly you don't even need that. (Though that is one of the motivations between a lot of API cleanups I've done recently!) I considered doing Python integration in the more obvious way, by linking parts of the PuTTY code directly into a native-code .so Python module. I decided against it because this way is more flexible: I can run the testcrypt program on its own, or compile it in a way that Python wouldn't play nicely with (I bet compiling just that .so with Leak Sanitiser wouldn't do what you wanted when Python loaded it!), or attach a debugger to it. I can even recompile testcrypt for a different CPU architecture (32- vs 64-bit, or even running it on a different machine over ssh or under emulation) and still layer the nice API on top of that via the local Python interpreter. All I need is a bidirectional data channel.
2019-01-01 19:08:37 +00:00
# Expect to be run from the 'test' subdirectory, one level down from
# the main source
putty_srcdir = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
Migrate all Python scripts to Python 3. Most of them are now _mandatory_ P3 scripts, because I'm tired of maintaining everything to be compatible with both versions. The current exceptions are gdb.py (which has to live with whatever gdb gives it), and kh2reg.py (which is actually designed for other people to use, and some of them might still be stuck on P2 for the moment).
2020-03-04 21:23:49 +00:00
def coerce_to_bytes(arg):
return arg.encode("UTF-8") if isinstance(arg, str) else arg
New test system for mp_int and cryptography. I've written a new standalone test program which incorporates all of PuTTY's crypto code, including the mp_int and low-level elliptic curve layers but also going all the way up to the implementations of the MAC, hash, cipher, public key and kex abstractions. The test program itself, 'testcrypt', speaks a simple line-oriented protocol on standard I/O in which you write the name of a function call followed by some inputs, and it gives you back a list of outputs preceded by a line telling you how many there are. Dynamically allocated objects are assigned string ids in the protocol, and there's a 'free' function that tells testcrypt when it can dispose of one. It's possible to speak that protocol by hand, but cumbersome. I've also provided a Python module that wraps it, by running testcrypt as a persistent subprocess and gatewaying all the function calls into things that look reasonably natural to call from Python. The Python module and testcrypt.c both read a carefully formatted header file testcrypt.h which contains the name and signature of every exported function, so it costs minimal effort to expose a given function through this test API. In a few cases it's necessary to write a wrapper in testcrypt.c that makes the function look more friendly, but mostly you don't even need that. (Though that is one of the motivations between a lot of API cleanups I've done recently!) I considered doing Python integration in the more obvious way, by linking parts of the PuTTY code directly into a native-code .so Python module. I decided against it because this way is more flexible: I can run the testcrypt program on its own, or compile it in a way that Python wouldn't play nicely with (I bet compiling just that .so with Leak Sanitiser wouldn't do what you wanted when Python loaded it!), or attach a debugger to it. I can even recompile testcrypt for a different CPU architecture (32- vs 64-bit, or even running it on a different machine over ssh or under emulation) and still layer the nice API on top of that via the local Python interpreter. All I need is a bidirectional data channel.
2019-01-01 19:08:37 +00:00
Handle crashes in the testcrypt binary more cleanly. Previously, if the testcrypt subprocess suffered any kind of crash or assertion failure during a run of the Python-based test system, the effect would be that ChildProcess.read_line() would get EOF, ignore it, and silently return the empty string. Then it would carry on doing that for the rest of the program, leading to a long string of error reports in tests that were nowhere near the code that actually caused the crash. Now ChildProcess.read_line() detects EOF and raises an exception, so that the test suite won't heedlessly carry on trying to do things once it's noticed that its subprocess has gone away. This is more fiddly than it sounds, however, because of the wrinkle that sometimes that function can be called while a Python __del__ method is asking testcrypt to free something. If that happens, the exception can't be propagated out of the __del__ (analogously to the rule that it's a really terrible idea for C++ destructors to throw). So you get an annoying warning message on standard error, and then the next command sent to testcrypt will be back in the same position. Worse still, this can also happen if testcrypt has _already_ crashed, because the __del__ methods will still run. To protect against _that_, ChildProcess caches the exception after throwing it, and then each subsequent write_line() will rethrow it. And __del__ catches and explicitly ignores the exception (to avoid the annoying warning if Python has to do the same). The combined result should be that if testcrypt crashes in normal (non-__del__) context, we should get a single exception that terminates the run cleanly without cascade failures, and whose backtrace localises the problem to the actual operation that caused the crash. If testcrypt crashes in __del__, we can't quite do that well, but we can still terminate with an exception at the next opportunity, avoiding multiple cascade failures. Also in this commit, I've got rid of the try-finally in cryptsuite.py's (trivial) main program.
2019-03-24 09:33:58 +00:00
class ChildProcessFailure(Exception):
pass
New test system for mp_int and cryptography. I've written a new standalone test program which incorporates all of PuTTY's crypto code, including the mp_int and low-level elliptic curve layers but also going all the way up to the implementations of the MAC, hash, cipher, public key and kex abstractions. The test program itself, 'testcrypt', speaks a simple line-oriented protocol on standard I/O in which you write the name of a function call followed by some inputs, and it gives you back a list of outputs preceded by a line telling you how many there are. Dynamically allocated objects are assigned string ids in the protocol, and there's a 'free' function that tells testcrypt when it can dispose of one. It's possible to speak that protocol by hand, but cumbersome. I've also provided a Python module that wraps it, by running testcrypt as a persistent subprocess and gatewaying all the function calls into things that look reasonably natural to call from Python. The Python module and testcrypt.c both read a carefully formatted header file testcrypt.h which contains the name and signature of every exported function, so it costs minimal effort to expose a given function through this test API. In a few cases it's necessary to write a wrapper in testcrypt.c that makes the function look more friendly, but mostly you don't even need that. (Though that is one of the motivations between a lot of API cleanups I've done recently!) I considered doing Python integration in the more obvious way, by linking parts of the PuTTY code directly into a native-code .so Python module. I decided against it because this way is more flexible: I can run the testcrypt program on its own, or compile it in a way that Python wouldn't play nicely with (I bet compiling just that .so with Leak Sanitiser wouldn't do what you wanted when Python loaded it!), or attach a debugger to it. I can even recompile testcrypt for a different CPU architecture (32- vs 64-bit, or even running it on a different machine over ssh or under emulation) and still layer the nice API on top of that via the local Python interpreter. All I need is a bidirectional data channel.
2019-01-01 19:08:37 +00:00
class ChildProcess(object):
def __init__(self):
self.sp = None
self.debug = None
cryptsuite: clean up exit handling. Now we only run the final memory-leak check if we didn't already have some other error to report, or some other exception that terminated the process. Also, we wait for the subprocess to terminate before returning control to the shell, so that any last-minute complaints from Leak Sanitiser appear before rather than after the shell prompt comes back. While I'm here, I've also made check_return_status tolerate the case in which the child process never got started at all. That way, if a failure manages to occur before even getting _that_ far, there won't be a cascade failure from check_return_status getting confused afterwards.
2019-03-24 09:56:57 +00:00
self.exitstatus = None
Handle crashes in the testcrypt binary more cleanly. Previously, if the testcrypt subprocess suffered any kind of crash or assertion failure during a run of the Python-based test system, the effect would be that ChildProcess.read_line() would get EOF, ignore it, and silently return the empty string. Then it would carry on doing that for the rest of the program, leading to a long string of error reports in tests that were nowhere near the code that actually caused the crash. Now ChildProcess.read_line() detects EOF and raises an exception, so that the test suite won't heedlessly carry on trying to do things once it's noticed that its subprocess has gone away. This is more fiddly than it sounds, however, because of the wrinkle that sometimes that function can be called while a Python __del__ method is asking testcrypt to free something. If that happens, the exception can't be propagated out of the __del__ (analogously to the rule that it's a really terrible idea for C++ destructors to throw). So you get an annoying warning message on standard error, and then the next command sent to testcrypt will be back in the same position. Worse still, this can also happen if testcrypt has _already_ crashed, because the __del__ methods will still run. To protect against _that_, ChildProcess caches the exception after throwing it, and then each subsequent write_line() will rethrow it. And __del__ catches and explicitly ignores the exception (to avoid the annoying warning if Python has to do the same). The combined result should be that if testcrypt crashes in normal (non-__del__) context, we should get a single exception that terminates the run cleanly without cascade failures, and whose backtrace localises the problem to the actual operation that caused the crash. If testcrypt crashes in __del__, we can't quite do that well, but we can still terminate with an exception at the next opportunity, avoiding multiple cascade failures. Also in this commit, I've got rid of the try-finally in cryptsuite.py's (trivial) main program.
2019-03-24 09:33:58 +00:00
self.exception = None
New test system for mp_int and cryptography. I've written a new standalone test program which incorporates all of PuTTY's crypto code, including the mp_int and low-level elliptic curve layers but also going all the way up to the implementations of the MAC, hash, cipher, public key and kex abstractions. The test program itself, 'testcrypt', speaks a simple line-oriented protocol on standard I/O in which you write the name of a function call followed by some inputs, and it gives you back a list of outputs preceded by a line telling you how many there are. Dynamically allocated objects are assigned string ids in the protocol, and there's a 'free' function that tells testcrypt when it can dispose of one. It's possible to speak that protocol by hand, but cumbersome. I've also provided a Python module that wraps it, by running testcrypt as a persistent subprocess and gatewaying all the function calls into things that look reasonably natural to call from Python. The Python module and testcrypt.c both read a carefully formatted header file testcrypt.h which contains the name and signature of every exported function, so it costs minimal effort to expose a given function through this test API. In a few cases it's necessary to write a wrapper in testcrypt.c that makes the function look more friendly, but mostly you don't even need that. (Though that is one of the motivations between a lot of API cleanups I've done recently!) I considered doing Python integration in the more obvious way, by linking parts of the PuTTY code directly into a native-code .so Python module. I decided against it because this way is more flexible: I can run the testcrypt program on its own, or compile it in a way that Python wouldn't play nicely with (I bet compiling just that .so with Leak Sanitiser wouldn't do what you wanted when Python loaded it!), or attach a debugger to it. I can even recompile testcrypt for a different CPU architecture (32- vs 64-bit, or even running it on a different machine over ssh or under emulation) and still layer the nice API on top of that via the local Python interpreter. All I need is a bidirectional data channel.
2019-01-01 19:08:37 +00:00
dbg = os.environ.get("PUTTY_TESTCRYPT_DEBUG")
if dbg is not None:
if dbg == "stderr":
self.debug = sys.stderr
else:
sys.stderr.write("Unknown value '{}' for PUTTY_TESTCRYPT_DEBUG"
" (try 'stderr'\n")
def start(self):
assert self.sp is None
override_command = os.environ.get("PUTTY_TESTCRYPT")
if override_command is None:
cmd = [os.path.join(putty_srcdir, "testcrypt")]
shell = False
else:
cmd = override_command
shell = True
self.sp = subprocess.Popen(
cmd, shell=shell, stdin=subprocess.PIPE, stdout=subprocess.PIPE)
def write_line(self, line):
Handle crashes in the testcrypt binary more cleanly. Previously, if the testcrypt subprocess suffered any kind of crash or assertion failure during a run of the Python-based test system, the effect would be that ChildProcess.read_line() would get EOF, ignore it, and silently return the empty string. Then it would carry on doing that for the rest of the program, leading to a long string of error reports in tests that were nowhere near the code that actually caused the crash. Now ChildProcess.read_line() detects EOF and raises an exception, so that the test suite won't heedlessly carry on trying to do things once it's noticed that its subprocess has gone away. This is more fiddly than it sounds, however, because of the wrinkle that sometimes that function can be called while a Python __del__ method is asking testcrypt to free something. If that happens, the exception can't be propagated out of the __del__ (analogously to the rule that it's a really terrible idea for C++ destructors to throw). So you get an annoying warning message on standard error, and then the next command sent to testcrypt will be back in the same position. Worse still, this can also happen if testcrypt has _already_ crashed, because the __del__ methods will still run. To protect against _that_, ChildProcess caches the exception after throwing it, and then each subsequent write_line() will rethrow it. And __del__ catches and explicitly ignores the exception (to avoid the annoying warning if Python has to do the same). The combined result should be that if testcrypt crashes in normal (non-__del__) context, we should get a single exception that terminates the run cleanly without cascade failures, and whose backtrace localises the problem to the actual operation that caused the crash. If testcrypt crashes in __del__, we can't quite do that well, but we can still terminate with an exception at the next opportunity, avoiding multiple cascade failures. Also in this commit, I've got rid of the try-finally in cryptsuite.py's (trivial) main program.
2019-03-24 09:33:58 +00:00
if self.exception is not None:
# Re-raise our fatal-error exception, if it previously
# occurred in a context where it couldn't be propagated (a
# __del__ method).
raise self.exception
New test system for mp_int and cryptography. I've written a new standalone test program which incorporates all of PuTTY's crypto code, including the mp_int and low-level elliptic curve layers but also going all the way up to the implementations of the MAC, hash, cipher, public key and kex abstractions. The test program itself, 'testcrypt', speaks a simple line-oriented protocol on standard I/O in which you write the name of a function call followed by some inputs, and it gives you back a list of outputs preceded by a line telling you how many there are. Dynamically allocated objects are assigned string ids in the protocol, and there's a 'free' function that tells testcrypt when it can dispose of one. It's possible to speak that protocol by hand, but cumbersome. I've also provided a Python module that wraps it, by running testcrypt as a persistent subprocess and gatewaying all the function calls into things that look reasonably natural to call from Python. The Python module and testcrypt.c both read a carefully formatted header file testcrypt.h which contains the name and signature of every exported function, so it costs minimal effort to expose a given function through this test API. In a few cases it's necessary to write a wrapper in testcrypt.c that makes the function look more friendly, but mostly you don't even need that. (Though that is one of the motivations between a lot of API cleanups I've done recently!) I considered doing Python integration in the more obvious way, by linking parts of the PuTTY code directly into a native-code .so Python module. I decided against it because this way is more flexible: I can run the testcrypt program on its own, or compile it in a way that Python wouldn't play nicely with (I bet compiling just that .so with Leak Sanitiser wouldn't do what you wanted when Python loaded it!), or attach a debugger to it. I can even recompile testcrypt for a different CPU architecture (32- vs 64-bit, or even running it on a different machine over ssh or under emulation) and still layer the nice API on top of that via the local Python interpreter. All I need is a bidirectional data channel.
2019-01-01 19:08:37 +00:00
if self.debug is not None:
self.debug.write("send: {}\n".format(line))
self.sp.stdin.write(line + b"\n")
self.sp.stdin.flush()
def read_line(self):
Handle crashes in the testcrypt binary more cleanly. Previously, if the testcrypt subprocess suffered any kind of crash or assertion failure during a run of the Python-based test system, the effect would be that ChildProcess.read_line() would get EOF, ignore it, and silently return the empty string. Then it would carry on doing that for the rest of the program, leading to a long string of error reports in tests that were nowhere near the code that actually caused the crash. Now ChildProcess.read_line() detects EOF and raises an exception, so that the test suite won't heedlessly carry on trying to do things once it's noticed that its subprocess has gone away. This is more fiddly than it sounds, however, because of the wrinkle that sometimes that function can be called while a Python __del__ method is asking testcrypt to free something. If that happens, the exception can't be propagated out of the __del__ (analogously to the rule that it's a really terrible idea for C++ destructors to throw). So you get an annoying warning message on standard error, and then the next command sent to testcrypt will be back in the same position. Worse still, this can also happen if testcrypt has _already_ crashed, because the __del__ methods will still run. To protect against _that_, ChildProcess caches the exception after throwing it, and then each subsequent write_line() will rethrow it. And __del__ catches and explicitly ignores the exception (to avoid the annoying warning if Python has to do the same). The combined result should be that if testcrypt crashes in normal (non-__del__) context, we should get a single exception that terminates the run cleanly without cascade failures, and whose backtrace localises the problem to the actual operation that caused the crash. If testcrypt crashes in __del__, we can't quite do that well, but we can still terminate with an exception at the next opportunity, avoiding multiple cascade failures. Also in this commit, I've got rid of the try-finally in cryptsuite.py's (trivial) main program.
2019-03-24 09:33:58 +00:00
line = self.sp.stdout.readline()
if len(line) == 0:
self.exception = ChildProcessFailure("received EOF from testcrypt")
raise self.exception
line = line.rstrip(b"\r\n")
New test system for mp_int and cryptography. I've written a new standalone test program which incorporates all of PuTTY's crypto code, including the mp_int and low-level elliptic curve layers but also going all the way up to the implementations of the MAC, hash, cipher, public key and kex abstractions. The test program itself, 'testcrypt', speaks a simple line-oriented protocol on standard I/O in which you write the name of a function call followed by some inputs, and it gives you back a list of outputs preceded by a line telling you how many there are. Dynamically allocated objects are assigned string ids in the protocol, and there's a 'free' function that tells testcrypt when it can dispose of one. It's possible to speak that protocol by hand, but cumbersome. I've also provided a Python module that wraps it, by running testcrypt as a persistent subprocess and gatewaying all the function calls into things that look reasonably natural to call from Python. The Python module and testcrypt.c both read a carefully formatted header file testcrypt.h which contains the name and signature of every exported function, so it costs minimal effort to expose a given function through this test API. In a few cases it's necessary to write a wrapper in testcrypt.c that makes the function look more friendly, but mostly you don't even need that. (Though that is one of the motivations between a lot of API cleanups I've done recently!) I considered doing Python integration in the more obvious way, by linking parts of the PuTTY code directly into a native-code .so Python module. I decided against it because this way is more flexible: I can run the testcrypt program on its own, or compile it in a way that Python wouldn't play nicely with (I bet compiling just that .so with Leak Sanitiser wouldn't do what you wanted when Python loaded it!), or attach a debugger to it. I can even recompile testcrypt for a different CPU architecture (32- vs 64-bit, or even running it on a different machine over ssh or under emulation) and still layer the nice API on top of that via the local Python interpreter. All I need is a bidirectional data channel.
2019-01-01 19:08:37 +00:00
if self.debug is not None:
self.debug.write("recv: {}\n".format(line))
return line
testcrypt.py: avoid restarting subprocess for frees. I just ran into a bug in which the testcrypt child process was cleanly terminated, but at least one Python object was left lying around containing the identifier of a testcrypt object that had never been freed. On program exit, the Python reference count on that object went to zero, the __del__ method was invoked, and childprocess.funcall started a _new_ instance of testcrypt just so it could tell it to free the object identifier - which, of course, the new testcrypt had never heard of! We can already tell the difference between a ChildProcess object which has no subprocess because it hasn't yet been started, and one which has no subprocess because it's terminated: the latter has exitstatus set to something other than None. So now we enforce by assertion that we don't ever restart the child process, and the __del__ method avoids doing anything if the child has already finished.
2020-02-29 10:54:16 +00:00
def already_terminated(self):
return self.sp is None and self.exitstatus is not None
New test system for mp_int and cryptography. I've written a new standalone test program which incorporates all of PuTTY's crypto code, including the mp_int and low-level elliptic curve layers but also going all the way up to the implementations of the MAC, hash, cipher, public key and kex abstractions. The test program itself, 'testcrypt', speaks a simple line-oriented protocol on standard I/O in which you write the name of a function call followed by some inputs, and it gives you back a list of outputs preceded by a line telling you how many there are. Dynamically allocated objects are assigned string ids in the protocol, and there's a 'free' function that tells testcrypt when it can dispose of one. It's possible to speak that protocol by hand, but cumbersome. I've also provided a Python module that wraps it, by running testcrypt as a persistent subprocess and gatewaying all the function calls into things that look reasonably natural to call from Python. The Python module and testcrypt.c both read a carefully formatted header file testcrypt.h which contains the name and signature of every exported function, so it costs minimal effort to expose a given function through this test API. In a few cases it's necessary to write a wrapper in testcrypt.c that makes the function look more friendly, but mostly you don't even need that. (Though that is one of the motivations between a lot of API cleanups I've done recently!) I considered doing Python integration in the more obvious way, by linking parts of the PuTTY code directly into a native-code .so Python module. I decided against it because this way is more flexible: I can run the testcrypt program on its own, or compile it in a way that Python wouldn't play nicely with (I bet compiling just that .so with Leak Sanitiser wouldn't do what you wanted when Python loaded it!), or attach a debugger to it. I can even recompile testcrypt for a different CPU architecture (32- vs 64-bit, or even running it on a different machine over ssh or under emulation) and still layer the nice API on top of that via the local Python interpreter. All I need is a bidirectional data channel.
2019-01-01 19:08:37 +00:00
def funcall(self, cmd, args):
if self.sp is None:
testcrypt.py: avoid restarting subprocess for frees. I just ran into a bug in which the testcrypt child process was cleanly terminated, but at least one Python object was left lying around containing the identifier of a testcrypt object that had never been freed. On program exit, the Python reference count on that object went to zero, the __del__ method was invoked, and childprocess.funcall started a _new_ instance of testcrypt just so it could tell it to free the object identifier - which, of course, the new testcrypt had never heard of! We can already tell the difference between a ChildProcess object which has no subprocess because it hasn't yet been started, and one which has no subprocess because it's terminated: the latter has exitstatus set to something other than None. So now we enforce by assertion that we don't ever restart the child process, and the __del__ method avoids doing anything if the child has already finished.
2020-02-29 10:54:16 +00:00
assert self.exitstatus is None
New test system for mp_int and cryptography. I've written a new standalone test program which incorporates all of PuTTY's crypto code, including the mp_int and low-level elliptic curve layers but also going all the way up to the implementations of the MAC, hash, cipher, public key and kex abstractions. The test program itself, 'testcrypt', speaks a simple line-oriented protocol on standard I/O in which you write the name of a function call followed by some inputs, and it gives you back a list of outputs preceded by a line telling you how many there are. Dynamically allocated objects are assigned string ids in the protocol, and there's a 'free' function that tells testcrypt when it can dispose of one. It's possible to speak that protocol by hand, but cumbersome. I've also provided a Python module that wraps it, by running testcrypt as a persistent subprocess and gatewaying all the function calls into things that look reasonably natural to call from Python. The Python module and testcrypt.c both read a carefully formatted header file testcrypt.h which contains the name and signature of every exported function, so it costs minimal effort to expose a given function through this test API. In a few cases it's necessary to write a wrapper in testcrypt.c that makes the function look more friendly, but mostly you don't even need that. (Though that is one of the motivations between a lot of API cleanups I've done recently!) I considered doing Python integration in the more obvious way, by linking parts of the PuTTY code directly into a native-code .so Python module. I decided against it because this way is more flexible: I can run the testcrypt program on its own, or compile it in a way that Python wouldn't play nicely with (I bet compiling just that .so with Leak Sanitiser wouldn't do what you wanted when Python loaded it!), or attach a debugger to it. I can even recompile testcrypt for a different CPU architecture (32- vs 64-bit, or even running it on a different machine over ssh or under emulation) and still layer the nice API on top of that via the local Python interpreter. All I need is a bidirectional data channel.
2019-01-01 19:08:37 +00:00
self.start()
Migrate all Python scripts to Python 3. Most of them are now _mandatory_ P3 scripts, because I'm tired of maintaining everything to be compatible with both versions. The current exceptions are gdb.py (which has to live with whatever gdb gives it), and kh2reg.py (which is actually designed for other people to use, and some of them might still be stuck on P2 for the moment).
2020-03-04 21:23:49 +00:00
self.write_line(coerce_to_bytes(cmd) + b" " + b" ".join(
coerce_to_bytes(arg) for arg in args))
New test system for mp_int and cryptography. I've written a new standalone test program which incorporates all of PuTTY's crypto code, including the mp_int and low-level elliptic curve layers but also going all the way up to the implementations of the MAC, hash, cipher, public key and kex abstractions. The test program itself, 'testcrypt', speaks a simple line-oriented protocol on standard I/O in which you write the name of a function call followed by some inputs, and it gives you back a list of outputs preceded by a line telling you how many there are. Dynamically allocated objects are assigned string ids in the protocol, and there's a 'free' function that tells testcrypt when it can dispose of one. It's possible to speak that protocol by hand, but cumbersome. I've also provided a Python module that wraps it, by running testcrypt as a persistent subprocess and gatewaying all the function calls into things that look reasonably natural to call from Python. The Python module and testcrypt.c both read a carefully formatted header file testcrypt.h which contains the name and signature of every exported function, so it costs minimal effort to expose a given function through this test API. In a few cases it's necessary to write a wrapper in testcrypt.c that makes the function look more friendly, but mostly you don't even need that. (Though that is one of the motivations between a lot of API cleanups I've done recently!) I considered doing Python integration in the more obvious way, by linking parts of the PuTTY code directly into a native-code .so Python module. I decided against it because this way is more flexible: I can run the testcrypt program on its own, or compile it in a way that Python wouldn't play nicely with (I bet compiling just that .so with Leak Sanitiser wouldn't do what you wanted when Python loaded it!), or attach a debugger to it. I can even recompile testcrypt for a different CPU architecture (32- vs 64-bit, or even running it on a different machine over ssh or under emulation) and still layer the nice API on top of that via the local Python interpreter. All I need is a bidirectional data channel.
2019-01-01 19:08:37 +00:00
argcount = int(self.read_line())
return [self.read_line() for arg in range(argcount)]
cryptsuite: clean up exit handling. Now we only run the final memory-leak check if we didn't already have some other error to report, or some other exception that terminated the process. Also, we wait for the subprocess to terminate before returning control to the shell, so that any last-minute complaints from Leak Sanitiser appear before rather than after the shell prompt comes back. While I'm here, I've also made check_return_status tolerate the case in which the child process never got started at all. That way, if a failure manages to occur before even getting _that_ far, there won't be a cascade failure from check_return_status getting confused afterwards.
2019-03-24 09:56:57 +00:00
def wait_for_exit(self):
if self.sp is not None:
self.sp.stdin.close()
self.exitstatus = self.sp.wait()
self.sp = None
New test system for mp_int and cryptography. I've written a new standalone test program which incorporates all of PuTTY's crypto code, including the mp_int and low-level elliptic curve layers but also going all the way up to the implementations of the MAC, hash, cipher, public key and kex abstractions. The test program itself, 'testcrypt', speaks a simple line-oriented protocol on standard I/O in which you write the name of a function call followed by some inputs, and it gives you back a list of outputs preceded by a line telling you how many there are. Dynamically allocated objects are assigned string ids in the protocol, and there's a 'free' function that tells testcrypt when it can dispose of one. It's possible to speak that protocol by hand, but cumbersome. I've also provided a Python module that wraps it, by running testcrypt as a persistent subprocess and gatewaying all the function calls into things that look reasonably natural to call from Python. The Python module and testcrypt.c both read a carefully formatted header file testcrypt.h which contains the name and signature of every exported function, so it costs minimal effort to expose a given function through this test API. In a few cases it's necessary to write a wrapper in testcrypt.c that makes the function look more friendly, but mostly you don't even need that. (Though that is one of the motivations between a lot of API cleanups I've done recently!) I considered doing Python integration in the more obvious way, by linking parts of the PuTTY code directly into a native-code .so Python module. I decided against it because this way is more flexible: I can run the testcrypt program on its own, or compile it in a way that Python wouldn't play nicely with (I bet compiling just that .so with Leak Sanitiser wouldn't do what you wanted when Python loaded it!), or attach a debugger to it. I can even recompile testcrypt for a different CPU architecture (32- vs 64-bit, or even running it on a different machine over ssh or under emulation) and still layer the nice API on top of that via the local Python interpreter. All I need is a bidirectional data channel.
2019-01-01 19:08:37 +00:00
def check_return_status(self):
cryptsuite: clean up exit handling. Now we only run the final memory-leak check if we didn't already have some other error to report, or some other exception that terminated the process. Also, we wait for the subprocess to terminate before returning control to the shell, so that any last-minute complaints from Leak Sanitiser appear before rather than after the shell prompt comes back. While I'm here, I've also made check_return_status tolerate the case in which the child process never got started at all. That way, if a failure manages to occur before even getting _that_ far, there won't be a cascade failure from check_return_status getting confused afterwards.
2019-03-24 09:56:57 +00:00
self.wait_for_exit()
if self.exitstatus is not None and self.exitstatus != 0:
Handle crashes in the testcrypt binary more cleanly. Previously, if the testcrypt subprocess suffered any kind of crash or assertion failure during a run of the Python-based test system, the effect would be that ChildProcess.read_line() would get EOF, ignore it, and silently return the empty string. Then it would carry on doing that for the rest of the program, leading to a long string of error reports in tests that were nowhere near the code that actually caused the crash. Now ChildProcess.read_line() detects EOF and raises an exception, so that the test suite won't heedlessly carry on trying to do things once it's noticed that its subprocess has gone away. This is more fiddly than it sounds, however, because of the wrinkle that sometimes that function can be called while a Python __del__ method is asking testcrypt to free something. If that happens, the exception can't be propagated out of the __del__ (analogously to the rule that it's a really terrible idea for C++ destructors to throw). So you get an annoying warning message on standard error, and then the next command sent to testcrypt will be back in the same position. Worse still, this can also happen if testcrypt has _already_ crashed, because the __del__ methods will still run. To protect against _that_, ChildProcess caches the exception after throwing it, and then each subsequent write_line() will rethrow it. And __del__ catches and explicitly ignores the exception (to avoid the annoying warning if Python has to do the same). The combined result should be that if testcrypt crashes in normal (non-__del__) context, we should get a single exception that terminates the run cleanly without cascade failures, and whose backtrace localises the problem to the actual operation that caused the crash. If testcrypt crashes in __del__, we can't quite do that well, but we can still terminate with an exception at the next opportunity, avoiding multiple cascade failures. Also in this commit, I've got rid of the try-finally in cryptsuite.py's (trivial) main program.
2019-03-24 09:33:58 +00:00
raise ChildProcessFailure("testcrypt returned exit status {}"
.format(self.exitstatus))
New test system for mp_int and cryptography. I've written a new standalone test program which incorporates all of PuTTY's crypto code, including the mp_int and low-level elliptic curve layers but also going all the way up to the implementations of the MAC, hash, cipher, public key and kex abstractions. The test program itself, 'testcrypt', speaks a simple line-oriented protocol on standard I/O in which you write the name of a function call followed by some inputs, and it gives you back a list of outputs preceded by a line telling you how many there are. Dynamically allocated objects are assigned string ids in the protocol, and there's a 'free' function that tells testcrypt when it can dispose of one. It's possible to speak that protocol by hand, but cumbersome. I've also provided a Python module that wraps it, by running testcrypt as a persistent subprocess and gatewaying all the function calls into things that look reasonably natural to call from Python. The Python module and testcrypt.c both read a carefully formatted header file testcrypt.h which contains the name and signature of every exported function, so it costs minimal effort to expose a given function through this test API. In a few cases it's necessary to write a wrapper in testcrypt.c that makes the function look more friendly, but mostly you don't even need that. (Though that is one of the motivations between a lot of API cleanups I've done recently!) I considered doing Python integration in the more obvious way, by linking parts of the PuTTY code directly into a native-code .so Python module. I decided against it because this way is more flexible: I can run the testcrypt program on its own, or compile it in a way that Python wouldn't play nicely with (I bet compiling just that .so with Leak Sanitiser wouldn't do what you wanted when Python loaded it!), or attach a debugger to it. I can even recompile testcrypt for a different CPU architecture (32- vs 64-bit, or even running it on a different machine over ssh or under emulation) and still layer the nice API on top of that via the local Python interpreter. All I need is a bidirectional data channel.
2019-01-01 19:08:37 +00:00
childprocess = ChildProcess()
testcrypt.py: fake some OO syntax. When I'm writing Python using the testcrypt API, I keep finding that I instinctively try to call vtable methods as if they were actual methods of the object. For example, calling key.sign(msg, 0) instead of ssh_key_sign(key, msg, 0). So this change to the Python side of the testcrypt mechanism panders to my inappropriate finger-macros by making them work! The idea is that I define a set of pairs (type, prefix), such that any function whose name begins with the prefix and whose first argument is of that type will be automatically translated into a method on the Python object wrapping a testcrypt value of that type. For example, any function of the form ssh_key_foo(val_ssh_key, other args) will automatically be exposed as a method key.foo(other args), simply because (val_ssh_key, "ssh_key_") appears in the translation table. This is particularly nice for the Python 3 REPL, which will let me tab-complete the right set of method names by knowing the type I'm trying to invoke one on. I haven't decided yet whether I want to switch to using it throughout cryptsuite.py. For namespace-cleanness, I've also renamed all the existing attributes of the Python Value class wrapper so that they start with '_', to leave the space of sensible names clear for the new OOish methods.
2020-02-29 09:48:00 +00:00
method_prefixes = {
testcrypt: permit multiple OO function prefixes for a type. This means if I have functions like foo_subfoo_bar and foo_baz that both operate on a foo, the Python testcrypt system can translate both into .bar() and .baz() methods on the object, even though they don't start with the same prefix.
2022-04-15 16:18:58 +00:00
'val_wpoint': ['ecc_weierstrass_'],
'val_mpoint': ['ecc_montgomery_'],
'val_epoint': ['ecc_edwards_'],
'val_hash': ['ssh_hash_'],
'val_mac': ['ssh2_mac_'],
'val_key': ['ssh_key_'],
'val_cipher': ['ssh_cipher_'],
'val_dh': ['dh_'],
'val_ecdh': ['ssh_ecdhkex_'],
'val_rsakex': ['ssh_rsakex_'],
'val_prng': ['prng_'],
'val_pcs': ['pcs_'],
'val_pockle': ['pockle_'],
'val_ntruencodeschedule': ['ntru_encode_schedule_', 'ntru_'],
testcrypt.py: fake some OO syntax. When I'm writing Python using the testcrypt API, I keep finding that I instinctively try to call vtable methods as if they were actual methods of the object. For example, calling key.sign(msg, 0) instead of ssh_key_sign(key, msg, 0). So this change to the Python side of the testcrypt mechanism panders to my inappropriate finger-macros by making them work! The idea is that I define a set of pairs (type, prefix), such that any function whose name begins with the prefix and whose first argument is of that type will be automatically translated into a method on the Python object wrapping a testcrypt value of that type. For example, any function of the form ssh_key_foo(val_ssh_key, other args) will automatically be exposed as a method key.foo(other args), simply because (val_ssh_key, "ssh_key_") appears in the translation table. This is particularly nice for the Python 3 REPL, which will let me tab-complete the right set of method names by knowing the type I'm trying to invoke one on. I haven't decided yet whether I want to switch to using it throughout cryptsuite.py. For namespace-cleanness, I've also renamed all the existing attributes of the Python Value class wrapper so that they start with '_', to leave the space of sensible names clear for the new OOish methods.
2020-02-29 09:48:00 +00:00
}
method_lists = {t: [] for t in method_prefixes}
testcrypt: introduce and use 'checkenum' protocol query. This allows the Python side of testcrypt to check in advance if a given string is a valid element of an enumeration, and if not, cleanly throw a Python-level exception without terminating the testcrypt subprocess. Should be useful in both manual use (when I'm trying something out by hand and make a typo or misremember a spelling), and automated use (if I make the same kind of error in cryptsuite.py then the exception dump will make more sense). In order to do this, the new handle_checkenum() function has to recognise all the enumerated types by name and match them up to their lookup functions - which is just the kind of thing that can now be done easily be reincluding testcrypt-enum.h with different #defines.
2021-11-22 18:39:45 +00:00
checked_enum_values = {}
New test system for mp_int and cryptography. I've written a new standalone test program which incorporates all of PuTTY's crypto code, including the mp_int and low-level elliptic curve layers but also going all the way up to the implementations of the MAC, hash, cipher, public key and kex abstractions. The test program itself, 'testcrypt', speaks a simple line-oriented protocol on standard I/O in which you write the name of a function call followed by some inputs, and it gives you back a list of outputs preceded by a line telling you how many there are. Dynamically allocated objects are assigned string ids in the protocol, and there's a 'free' function that tells testcrypt when it can dispose of one. It's possible to speak that protocol by hand, but cumbersome. I've also provided a Python module that wraps it, by running testcrypt as a persistent subprocess and gatewaying all the function calls into things that look reasonably natural to call from Python. The Python module and testcrypt.c both read a carefully formatted header file testcrypt.h which contains the name and signature of every exported function, so it costs minimal effort to expose a given function through this test API. In a few cases it's necessary to write a wrapper in testcrypt.c that makes the function look more friendly, but mostly you don't even need that. (Though that is one of the motivations between a lot of API cleanups I've done recently!) I considered doing Python integration in the more obvious way, by linking parts of the PuTTY code directly into a native-code .so Python module. I decided against it because this way is more flexible: I can run the testcrypt program on its own, or compile it in a way that Python wouldn't play nicely with (I bet compiling just that .so with Leak Sanitiser wouldn't do what you wanted when Python loaded it!), or attach a debugger to it. I can even recompile testcrypt for a different CPU architecture (32- vs 64-bit, or even running it on a different machine over ssh or under emulation) and still layer the nice API on top of that via the local Python interpreter. All I need is a bidirectional data channel.
2019-01-01 19:08:37 +00:00
class Value(object):
def __init__(self, typename, ident):
testcrypt.py: fake some OO syntax. When I'm writing Python using the testcrypt API, I keep finding that I instinctively try to call vtable methods as if they were actual methods of the object. For example, calling key.sign(msg, 0) instead of ssh_key_sign(key, msg, 0). So this change to the Python side of the testcrypt mechanism panders to my inappropriate finger-macros by making them work! The idea is that I define a set of pairs (type, prefix), such that any function whose name begins with the prefix and whose first argument is of that type will be automatically translated into a method on the Python object wrapping a testcrypt value of that type. For example, any function of the form ssh_key_foo(val_ssh_key, other args) will automatically be exposed as a method key.foo(other args), simply because (val_ssh_key, "ssh_key_") appears in the translation table. This is particularly nice for the Python 3 REPL, which will let me tab-complete the right set of method names by knowing the type I'm trying to invoke one on. I haven't decided yet whether I want to switch to using it throughout cryptsuite.py. For namespace-cleanness, I've also renamed all the existing attributes of the Python Value class wrapper so that they start with '_', to leave the space of sensible names clear for the new OOish methods.
2020-02-29 09:48:00 +00:00
self._typename = typename
self._ident = ident
for methodname, function in method_lists.get(self._typename, []):
setattr(self, methodname,
(lambda f: lambda *args: f(self, *args))(function))
def _consumed(self):
self._ident = None
New test system for mp_int and cryptography. I've written a new standalone test program which incorporates all of PuTTY's crypto code, including the mp_int and low-level elliptic curve layers but also going all the way up to the implementations of the MAC, hash, cipher, public key and kex abstractions. The test program itself, 'testcrypt', speaks a simple line-oriented protocol on standard I/O in which you write the name of a function call followed by some inputs, and it gives you back a list of outputs preceded by a line telling you how many there are. Dynamically allocated objects are assigned string ids in the protocol, and there's a 'free' function that tells testcrypt when it can dispose of one. It's possible to speak that protocol by hand, but cumbersome. I've also provided a Python module that wraps it, by running testcrypt as a persistent subprocess and gatewaying all the function calls into things that look reasonably natural to call from Python. The Python module and testcrypt.c both read a carefully formatted header file testcrypt.h which contains the name and signature of every exported function, so it costs minimal effort to expose a given function through this test API. In a few cases it's necessary to write a wrapper in testcrypt.c that makes the function look more friendly, but mostly you don't even need that. (Though that is one of the motivations between a lot of API cleanups I've done recently!) I considered doing Python integration in the more obvious way, by linking parts of the PuTTY code directly into a native-code .so Python module. I decided against it because this way is more flexible: I can run the testcrypt program on its own, or compile it in a way that Python wouldn't play nicely with (I bet compiling just that .so with Leak Sanitiser wouldn't do what you wanted when Python loaded it!), or attach a debugger to it. I can even recompile testcrypt for a different CPU architecture (32- vs 64-bit, or even running it on a different machine over ssh or under emulation) and still layer the nice API on top of that via the local Python interpreter. All I need is a bidirectional data channel.
2019-01-01 19:08:37 +00:00
def __repr__(self):
testcrypt.py: fake some OO syntax. When I'm writing Python using the testcrypt API, I keep finding that I instinctively try to call vtable methods as if they were actual methods of the object. For example, calling key.sign(msg, 0) instead of ssh_key_sign(key, msg, 0). So this change to the Python side of the testcrypt mechanism panders to my inappropriate finger-macros by making them work! The idea is that I define a set of pairs (type, prefix), such that any function whose name begins with the prefix and whose first argument is of that type will be automatically translated into a method on the Python object wrapping a testcrypt value of that type. For example, any function of the form ssh_key_foo(val_ssh_key, other args) will automatically be exposed as a method key.foo(other args), simply because (val_ssh_key, "ssh_key_") appears in the translation table. This is particularly nice for the Python 3 REPL, which will let me tab-complete the right set of method names by knowing the type I'm trying to invoke one on. I haven't decided yet whether I want to switch to using it throughout cryptsuite.py. For namespace-cleanness, I've also renamed all the existing attributes of the Python Value class wrapper so that they start with '_', to leave the space of sensible names clear for the new OOish methods.
2020-02-29 09:48:00 +00:00
return "Value({!r}, {!r})".format(self._typename, self._ident)
New test system for mp_int and cryptography. I've written a new standalone test program which incorporates all of PuTTY's crypto code, including the mp_int and low-level elliptic curve layers but also going all the way up to the implementations of the MAC, hash, cipher, public key and kex abstractions. The test program itself, 'testcrypt', speaks a simple line-oriented protocol on standard I/O in which you write the name of a function call followed by some inputs, and it gives you back a list of outputs preceded by a line telling you how many there are. Dynamically allocated objects are assigned string ids in the protocol, and there's a 'free' function that tells testcrypt when it can dispose of one. It's possible to speak that protocol by hand, but cumbersome. I've also provided a Python module that wraps it, by running testcrypt as a persistent subprocess and gatewaying all the function calls into things that look reasonably natural to call from Python. The Python module and testcrypt.c both read a carefully formatted header file testcrypt.h which contains the name and signature of every exported function, so it costs minimal effort to expose a given function through this test API. In a few cases it's necessary to write a wrapper in testcrypt.c that makes the function look more friendly, but mostly you don't even need that. (Though that is one of the motivations between a lot of API cleanups I've done recently!) I considered doing Python integration in the more obvious way, by linking parts of the PuTTY code directly into a native-code .so Python module. I decided against it because this way is more flexible: I can run the testcrypt program on its own, or compile it in a way that Python wouldn't play nicely with (I bet compiling just that .so with Leak Sanitiser wouldn't do what you wanted when Python loaded it!), or attach a debugger to it. I can even recompile testcrypt for a different CPU architecture (32- vs 64-bit, or even running it on a different machine over ssh or under emulation) and still layer the nice API on top of that via the local Python interpreter. All I need is a bidirectional data channel.
2019-01-01 19:08:37 +00:00
def __del__(self):
testcrypt.py: avoid restarting subprocess for frees. I just ran into a bug in which the testcrypt child process was cleanly terminated, but at least one Python object was left lying around containing the identifier of a testcrypt object that had never been freed. On program exit, the Python reference count on that object went to zero, the __del__ method was invoked, and childprocess.funcall started a _new_ instance of testcrypt just so it could tell it to free the object identifier - which, of course, the new testcrypt had never heard of! We can already tell the difference between a ChildProcess object which has no subprocess because it hasn't yet been started, and one which has no subprocess because it's terminated: the latter has exitstatus set to something other than None. So now we enforce by assertion that we don't ever restart the child process, and the __del__ method avoids doing anything if the child has already finished.
2020-02-29 10:54:16 +00:00
if self._ident is not None and not childprocess.already_terminated():
Handle crashes in the testcrypt binary more cleanly. Previously, if the testcrypt subprocess suffered any kind of crash or assertion failure during a run of the Python-based test system, the effect would be that ChildProcess.read_line() would get EOF, ignore it, and silently return the empty string. Then it would carry on doing that for the rest of the program, leading to a long string of error reports in tests that were nowhere near the code that actually caused the crash. Now ChildProcess.read_line() detects EOF and raises an exception, so that the test suite won't heedlessly carry on trying to do things once it's noticed that its subprocess has gone away. This is more fiddly than it sounds, however, because of the wrinkle that sometimes that function can be called while a Python __del__ method is asking testcrypt to free something. If that happens, the exception can't be propagated out of the __del__ (analogously to the rule that it's a really terrible idea for C++ destructors to throw). So you get an annoying warning message on standard error, and then the next command sent to testcrypt will be back in the same position. Worse still, this can also happen if testcrypt has _already_ crashed, because the __del__ methods will still run. To protect against _that_, ChildProcess caches the exception after throwing it, and then each subsequent write_line() will rethrow it. And __del__ catches and explicitly ignores the exception (to avoid the annoying warning if Python has to do the same). The combined result should be that if testcrypt crashes in normal (non-__del__) context, we should get a single exception that terminates the run cleanly without cascade failures, and whose backtrace localises the problem to the actual operation that caused the crash. If testcrypt crashes in __del__, we can't quite do that well, but we can still terminate with an exception at the next opportunity, avoiding multiple cascade failures. Also in this commit, I've got rid of the try-finally in cryptsuite.py's (trivial) main program.
2019-03-24 09:33:58 +00:00
try:
testcrypt.py: fake some OO syntax. When I'm writing Python using the testcrypt API, I keep finding that I instinctively try to call vtable methods as if they were actual methods of the object. For example, calling key.sign(msg, 0) instead of ssh_key_sign(key, msg, 0). So this change to the Python side of the testcrypt mechanism panders to my inappropriate finger-macros by making them work! The idea is that I define a set of pairs (type, prefix), such that any function whose name begins with the prefix and whose first argument is of that type will be automatically translated into a method on the Python object wrapping a testcrypt value of that type. For example, any function of the form ssh_key_foo(val_ssh_key, other args) will automatically be exposed as a method key.foo(other args), simply because (val_ssh_key, "ssh_key_") appears in the translation table. This is particularly nice for the Python 3 REPL, which will let me tab-complete the right set of method names by knowing the type I'm trying to invoke one on. I haven't decided yet whether I want to switch to using it throughout cryptsuite.py. For namespace-cleanness, I've also renamed all the existing attributes of the Python Value class wrapper so that they start with '_', to leave the space of sensible names clear for the new OOish methods.
2020-02-29 09:48:00 +00:00
childprocess.funcall("free", [self._ident])
Handle crashes in the testcrypt binary more cleanly. Previously, if the testcrypt subprocess suffered any kind of crash or assertion failure during a run of the Python-based test system, the effect would be that ChildProcess.read_line() would get EOF, ignore it, and silently return the empty string. Then it would carry on doing that for the rest of the program, leading to a long string of error reports in tests that were nowhere near the code that actually caused the crash. Now ChildProcess.read_line() detects EOF and raises an exception, so that the test suite won't heedlessly carry on trying to do things once it's noticed that its subprocess has gone away. This is more fiddly than it sounds, however, because of the wrinkle that sometimes that function can be called while a Python __del__ method is asking testcrypt to free something. If that happens, the exception can't be propagated out of the __del__ (analogously to the rule that it's a really terrible idea for C++ destructors to throw). So you get an annoying warning message on standard error, and then the next command sent to testcrypt will be back in the same position. Worse still, this can also happen if testcrypt has _already_ crashed, because the __del__ methods will still run. To protect against _that_, ChildProcess caches the exception after throwing it, and then each subsequent write_line() will rethrow it. And __del__ catches and explicitly ignores the exception (to avoid the annoying warning if Python has to do the same). The combined result should be that if testcrypt crashes in normal (non-__del__) context, we should get a single exception that terminates the run cleanly without cascade failures, and whose backtrace localises the problem to the actual operation that caused the crash. If testcrypt crashes in __del__, we can't quite do that well, but we can still terminate with an exception at the next opportunity, avoiding multiple cascade failures. Also in this commit, I've got rid of the try-finally in cryptsuite.py's (trivial) main program.
2019-03-24 09:33:58 +00:00
except ChildProcessFailure:
# If we see this exception now, we can't do anything
# about it, because exceptions don't propagate out of
# __del__ methods. Squelch it to prevent the annoying
# runtime warning from Python, and the
# 'self.exception' mechanism in the ChildProcess class
# will raise it again at the next opportunity.
#
# (This covers both the case where testcrypt crashes
# _during_ one of these free operations, and the
# silencing of cascade failures when we try to send a
# "free" command to testcrypt after it had already
# crashed for some other reason.)
pass
New test system for mp_int and cryptography. I've written a new standalone test program which incorporates all of PuTTY's crypto code, including the mp_int and low-level elliptic curve layers but also going all the way up to the implementations of the MAC, hash, cipher, public key and kex abstractions. The test program itself, 'testcrypt', speaks a simple line-oriented protocol on standard I/O in which you write the name of a function call followed by some inputs, and it gives you back a list of outputs preceded by a line telling you how many there are. Dynamically allocated objects are assigned string ids in the protocol, and there's a 'free' function that tells testcrypt when it can dispose of one. It's possible to speak that protocol by hand, but cumbersome. I've also provided a Python module that wraps it, by running testcrypt as a persistent subprocess and gatewaying all the function calls into things that look reasonably natural to call from Python. The Python module and testcrypt.c both read a carefully formatted header file testcrypt.h which contains the name and signature of every exported function, so it costs minimal effort to expose a given function through this test API. In a few cases it's necessary to write a wrapper in testcrypt.c that makes the function look more friendly, but mostly you don't even need that. (Though that is one of the motivations between a lot of API cleanups I've done recently!) I considered doing Python integration in the more obvious way, by linking parts of the PuTTY code directly into a native-code .so Python module. I decided against it because this way is more flexible: I can run the testcrypt program on its own, or compile it in a way that Python wouldn't play nicely with (I bet compiling just that .so with Leak Sanitiser wouldn't do what you wanted when Python loaded it!), or attach a debugger to it. I can even recompile testcrypt for a different CPU architecture (32- vs 64-bit, or even running it on a different machine over ssh or under emulation) and still layer the nice API on top of that via the local Python interpreter. All I need is a bidirectional data channel.
2019-01-01 19:08:37 +00:00
def __long__(self):
testcrypt.py: fake some OO syntax. When I'm writing Python using the testcrypt API, I keep finding that I instinctively try to call vtable methods as if they were actual methods of the object. For example, calling key.sign(msg, 0) instead of ssh_key_sign(key, msg, 0). So this change to the Python side of the testcrypt mechanism panders to my inappropriate finger-macros by making them work! The idea is that I define a set of pairs (type, prefix), such that any function whose name begins with the prefix and whose first argument is of that type will be automatically translated into a method on the Python object wrapping a testcrypt value of that type. For example, any function of the form ssh_key_foo(val_ssh_key, other args) will automatically be exposed as a method key.foo(other args), simply because (val_ssh_key, "ssh_key_") appears in the translation table. This is particularly nice for the Python 3 REPL, which will let me tab-complete the right set of method names by knowing the type I'm trying to invoke one on. I haven't decided yet whether I want to switch to using it throughout cryptsuite.py. For namespace-cleanness, I've also renamed all the existing attributes of the Python Value class wrapper so that they start with '_', to leave the space of sensible names clear for the new OOish methods.
2020-02-29 09:48:00 +00:00
if self._typename != "val_mpint":
New test system for mp_int and cryptography. I've written a new standalone test program which incorporates all of PuTTY's crypto code, including the mp_int and low-level elliptic curve layers but also going all the way up to the implementations of the MAC, hash, cipher, public key and kex abstractions. The test program itself, 'testcrypt', speaks a simple line-oriented protocol on standard I/O in which you write the name of a function call followed by some inputs, and it gives you back a list of outputs preceded by a line telling you how many there are. Dynamically allocated objects are assigned string ids in the protocol, and there's a 'free' function that tells testcrypt when it can dispose of one. It's possible to speak that protocol by hand, but cumbersome. I've also provided a Python module that wraps it, by running testcrypt as a persistent subprocess and gatewaying all the function calls into things that look reasonably natural to call from Python. The Python module and testcrypt.c both read a carefully formatted header file testcrypt.h which contains the name and signature of every exported function, so it costs minimal effort to expose a given function through this test API. In a few cases it's necessary to write a wrapper in testcrypt.c that makes the function look more friendly, but mostly you don't even need that. (Though that is one of the motivations between a lot of API cleanups I've done recently!) I considered doing Python integration in the more obvious way, by linking parts of the PuTTY code directly into a native-code .so Python module. I decided against it because this way is more flexible: I can run the testcrypt program on its own, or compile it in a way that Python wouldn't play nicely with (I bet compiling just that .so with Leak Sanitiser wouldn't do what you wanted when Python loaded it!), or attach a debugger to it. I can even recompile testcrypt for a different CPU architecture (32- vs 64-bit, or even running it on a different machine over ssh or under emulation) and still layer the nice API on top of that via the local Python interpreter. All I need is a bidirectional data channel.
2019-01-01 19:08:37 +00:00
raise TypeError("testcrypt values of types other than mpint"
" cannot be converted to integer")
testcrypt.py: fake some OO syntax. When I'm writing Python using the testcrypt API, I keep finding that I instinctively try to call vtable methods as if they were actual methods of the object. For example, calling key.sign(msg, 0) instead of ssh_key_sign(key, msg, 0). So this change to the Python side of the testcrypt mechanism panders to my inappropriate finger-macros by making them work! The idea is that I define a set of pairs (type, prefix), such that any function whose name begins with the prefix and whose first argument is of that type will be automatically translated into a method on the Python object wrapping a testcrypt value of that type. For example, any function of the form ssh_key_foo(val_ssh_key, other args) will automatically be exposed as a method key.foo(other args), simply because (val_ssh_key, "ssh_key_") appears in the translation table. This is particularly nice for the Python 3 REPL, which will let me tab-complete the right set of method names by knowing the type I'm trying to invoke one on. I haven't decided yet whether I want to switch to using it throughout cryptsuite.py. For namespace-cleanness, I've also renamed all the existing attributes of the Python Value class wrapper so that they start with '_', to leave the space of sensible names clear for the new OOish methods.
2020-02-29 09:48:00 +00:00
hexval = childprocess.funcall("mp_dump", [self._ident])[0]
New test system for mp_int and cryptography. I've written a new standalone test program which incorporates all of PuTTY's crypto code, including the mp_int and low-level elliptic curve layers but also going all the way up to the implementations of the MAC, hash, cipher, public key and kex abstractions. The test program itself, 'testcrypt', speaks a simple line-oriented protocol on standard I/O in which you write the name of a function call followed by some inputs, and it gives you back a list of outputs preceded by a line telling you how many there are. Dynamically allocated objects are assigned string ids in the protocol, and there's a 'free' function that tells testcrypt when it can dispose of one. It's possible to speak that protocol by hand, but cumbersome. I've also provided a Python module that wraps it, by running testcrypt as a persistent subprocess and gatewaying all the function calls into things that look reasonably natural to call from Python. The Python module and testcrypt.c both read a carefully formatted header file testcrypt.h which contains the name and signature of every exported function, so it costs minimal effort to expose a given function through this test API. In a few cases it's necessary to write a wrapper in testcrypt.c that makes the function look more friendly, but mostly you don't even need that. (Though that is one of the motivations between a lot of API cleanups I've done recently!) I considered doing Python integration in the more obvious way, by linking parts of the PuTTY code directly into a native-code .so Python module. I decided against it because this way is more flexible: I can run the testcrypt program on its own, or compile it in a way that Python wouldn't play nicely with (I bet compiling just that .so with Leak Sanitiser wouldn't do what you wanted when Python loaded it!), or attach a debugger to it. I can even recompile testcrypt for a different CPU architecture (32- vs 64-bit, or even running it on a different machine over ssh or under emulation) and still layer the nice API on top of that via the local Python interpreter. All I need is a bidirectional data channel.
2019-01-01 19:08:37 +00:00
return 0 if len(hexval) == 0 else int(hexval, 16)
def __int__(self):
return int(self.__long__())
testcrypt.py: marshal string literals more efficiently. The testcrypt protocol expects a string literal to be a concatenation of literal bytes other than '%' and '\n', and %-escaped hex digit pairs. But testcrypt.py was only ever using the latter format, so even a legible ASCII string like "123" was being sent to testcrypt as the unreadable and needlessly long "%31%32%33". When debugging, I often arrange to save the testcrypt input stream to a file, and sometimes I use that file as the starting point for editing. So it is actually useful to have the protocol exchange be legible to humans. Hence, here's a change to testcrypt.py which makes it only use the %-escape encoding for byte values that aren't printable ASCII.
2020-03-08 11:09:06 +00:00
def marshal_string(val):
val = coerce_to_bytes(val)
assert isinstance(val, bytes), "Bad type for val_string input"
return "".join(
chr(b) if (0x20 <= b < 0x7F and b != 0x25)
else "%{:02x}".format(b)
for b in val)
testcrypt.py: use parameter names in diagnostics. Making a virtue of the necessity of adding parameter names to testcrypt.h a couple of commits ago, we can now use those names to improve diagnostics, so that if you use the wrong type in a Python function call the error message will tell you the name as well as the index of the offending argument. Also, the repr() text for the function itself will now print a full prototype (albeit in a nasty hybrid of C, Python and testcrypt.h syntax) which shows all the parameter names. That should be handy when trying to remember the order of arguments at the REPL prompt.
2021-11-21 13:21:01 +00:00
def make_argword(arg, argtype, fnname, argindex, argname, to_preserve):
New test system for mp_int and cryptography. I've written a new standalone test program which incorporates all of PuTTY's crypto code, including the mp_int and low-level elliptic curve layers but also going all the way up to the implementations of the MAC, hash, cipher, public key and kex abstractions. The test program itself, 'testcrypt', speaks a simple line-oriented protocol on standard I/O in which you write the name of a function call followed by some inputs, and it gives you back a list of outputs preceded by a line telling you how many there are. Dynamically allocated objects are assigned string ids in the protocol, and there's a 'free' function that tells testcrypt when it can dispose of one. It's possible to speak that protocol by hand, but cumbersome. I've also provided a Python module that wraps it, by running testcrypt as a persistent subprocess and gatewaying all the function calls into things that look reasonably natural to call from Python. The Python module and testcrypt.c both read a carefully formatted header file testcrypt.h which contains the name and signature of every exported function, so it costs minimal effort to expose a given function through this test API. In a few cases it's necessary to write a wrapper in testcrypt.c that makes the function look more friendly, but mostly you don't even need that. (Though that is one of the motivations between a lot of API cleanups I've done recently!) I considered doing Python integration in the more obvious way, by linking parts of the PuTTY code directly into a native-code .so Python module. I decided against it because this way is more flexible: I can run the testcrypt program on its own, or compile it in a way that Python wouldn't play nicely with (I bet compiling just that .so with Leak Sanitiser wouldn't do what you wanted when Python loaded it!), or attach a debugger to it. I can even recompile testcrypt for a different CPU architecture (32- vs 64-bit, or even running it on a different machine over ssh or under emulation) and still layer the nice API on top of that via the local Python interpreter. All I need is a bidirectional data channel.
2019-01-01 19:08:37 +00:00
typename, consumed = argtype
if typename.startswith("opt_"):
if arg is None:
return "NULL"
typename = typename[4:]
if typename == "val_string":
testcrypt.py: marshal string literals more efficiently. The testcrypt protocol expects a string literal to be a concatenation of literal bytes other than '%' and '\n', and %-escaped hex digit pairs. But testcrypt.py was only ever using the latter format, so even a legible ASCII string like "123" was being sent to testcrypt as the unreadable and needlessly long "%31%32%33". When debugging, I often arrange to save the testcrypt input stream to a file, and sometimes I use that file as the starting point for editing. So it is actually useful to have the protocol exchange be legible to humans. Hence, here's a change to testcrypt.py which makes it only use the %-escape encoding for byte values that aren't printable ASCII.
2020-03-08 11:09:06 +00:00
retwords = childprocess.funcall("newstring", [marshal_string(arg)])
arg = make_retvals([typename], retwords, unpack_strings=False)[0]
to_preserve.append(arg)
New test system for mp_int and cryptography. I've written a new standalone test program which incorporates all of PuTTY's crypto code, including the mp_int and low-level elliptic curve layers but also going all the way up to the implementations of the MAC, hash, cipher, public key and kex abstractions. The test program itself, 'testcrypt', speaks a simple line-oriented protocol on standard I/O in which you write the name of a function call followed by some inputs, and it gives you back a list of outputs preceded by a line telling you how many there are. Dynamically allocated objects are assigned string ids in the protocol, and there's a 'free' function that tells testcrypt when it can dispose of one. It's possible to speak that protocol by hand, but cumbersome. I've also provided a Python module that wraps it, by running testcrypt as a persistent subprocess and gatewaying all the function calls into things that look reasonably natural to call from Python. The Python module and testcrypt.c both read a carefully formatted header file testcrypt.h which contains the name and signature of every exported function, so it costs minimal effort to expose a given function through this test API. In a few cases it's necessary to write a wrapper in testcrypt.c that makes the function look more friendly, but mostly you don't even need that. (Though that is one of the motivations between a lot of API cleanups I've done recently!) I considered doing Python integration in the more obvious way, by linking parts of the PuTTY code directly into a native-code .so Python module. I decided against it because this way is more flexible: I can run the testcrypt program on its own, or compile it in a way that Python wouldn't play nicely with (I bet compiling just that .so with Leak Sanitiser wouldn't do what you wanted when Python loaded it!), or attach a debugger to it. I can even recompile testcrypt for a different CPU architecture (32- vs 64-bit, or even running it on a different machine over ssh or under emulation) and still layer the nice API on top of that via the local Python interpreter. All I need is a bidirectional data channel.
2019-01-01 19:08:37 +00:00
if typename == "val_mpint" and isinstance(arg, numbers.Integral):
retwords = childprocess.funcall("mp_literal", ["0x{:x}".format(arg)])
arg = make_retvals([typename], retwords)[0]
to_preserve.append(arg)
if isinstance(arg, Value):
testcrypt.py: fake some OO syntax. When I'm writing Python using the testcrypt API, I keep finding that I instinctively try to call vtable methods as if they were actual methods of the object. For example, calling key.sign(msg, 0) instead of ssh_key_sign(key, msg, 0). So this change to the Python side of the testcrypt mechanism panders to my inappropriate finger-macros by making them work! The idea is that I define a set of pairs (type, prefix), such that any function whose name begins with the prefix and whose first argument is of that type will be automatically translated into a method on the Python object wrapping a testcrypt value of that type. For example, any function of the form ssh_key_foo(val_ssh_key, other args) will automatically be exposed as a method key.foo(other args), simply because (val_ssh_key, "ssh_key_") appears in the translation table. This is particularly nice for the Python 3 REPL, which will let me tab-complete the right set of method names by knowing the type I'm trying to invoke one on. I haven't decided yet whether I want to switch to using it throughout cryptsuite.py. For namespace-cleanness, I've also renamed all the existing attributes of the Python Value class wrapper so that they start with '_', to leave the space of sensible names clear for the new OOish methods.
2020-02-29 09:48:00 +00:00
if arg._typename != typename:
New test system for mp_int and cryptography. I've written a new standalone test program which incorporates all of PuTTY's crypto code, including the mp_int and low-level elliptic curve layers but also going all the way up to the implementations of the MAC, hash, cipher, public key and kex abstractions. The test program itself, 'testcrypt', speaks a simple line-oriented protocol on standard I/O in which you write the name of a function call followed by some inputs, and it gives you back a list of outputs preceded by a line telling you how many there are. Dynamically allocated objects are assigned string ids in the protocol, and there's a 'free' function that tells testcrypt when it can dispose of one. It's possible to speak that protocol by hand, but cumbersome. I've also provided a Python module that wraps it, by running testcrypt as a persistent subprocess and gatewaying all the function calls into things that look reasonably natural to call from Python. The Python module and testcrypt.c both read a carefully formatted header file testcrypt.h which contains the name and signature of every exported function, so it costs minimal effort to expose a given function through this test API. In a few cases it's necessary to write a wrapper in testcrypt.c that makes the function look more friendly, but mostly you don't even need that. (Though that is one of the motivations between a lot of API cleanups I've done recently!) I considered doing Python integration in the more obvious way, by linking parts of the PuTTY code directly into a native-code .so Python module. I decided against it because this way is more flexible: I can run the testcrypt program on its own, or compile it in a way that Python wouldn't play nicely with (I bet compiling just that .so with Leak Sanitiser wouldn't do what you wanted when Python loaded it!), or attach a debugger to it. I can even recompile testcrypt for a different CPU architecture (32- vs 64-bit, or even running it on a different machine over ssh or under emulation) and still layer the nice API on top of that via the local Python interpreter. All I need is a bidirectional data channel.
2019-01-01 19:08:37 +00:00
raise TypeError(
testcrypt.py: use parameter names in diagnostics. Making a virtue of the necessity of adding parameter names to testcrypt.h a couple of commits ago, we can now use those names to improve diagnostics, so that if you use the wrong type in a Python function call the error message will tell you the name as well as the index of the offending argument. Also, the repr() text for the function itself will now print a full prototype (albeit in a nasty hybrid of C, Python and testcrypt.h syntax) which shows all the parameter names. That should be handy when trying to remember the order of arguments at the REPL prompt.
2021-11-21 13:21:01 +00:00
"{}() argument #{:d} ({}) should be {} ({} given)".format(
fnname, argindex, argname, typename, arg._typename))
testcrypt.py: fake some OO syntax. When I'm writing Python using the testcrypt API, I keep finding that I instinctively try to call vtable methods as if they were actual methods of the object. For example, calling key.sign(msg, 0) instead of ssh_key_sign(key, msg, 0). So this change to the Python side of the testcrypt mechanism panders to my inappropriate finger-macros by making them work! The idea is that I define a set of pairs (type, prefix), such that any function whose name begins with the prefix and whose first argument is of that type will be automatically translated into a method on the Python object wrapping a testcrypt value of that type. For example, any function of the form ssh_key_foo(val_ssh_key, other args) will automatically be exposed as a method key.foo(other args), simply because (val_ssh_key, "ssh_key_") appears in the translation table. This is particularly nice for the Python 3 REPL, which will let me tab-complete the right set of method names by knowing the type I'm trying to invoke one on. I haven't decided yet whether I want to switch to using it throughout cryptsuite.py. For namespace-cleanness, I've also renamed all the existing attributes of the Python Value class wrapper so that they start with '_', to leave the space of sensible names clear for the new OOish methods.
2020-02-29 09:48:00 +00:00
ident = arg._ident
New test system for mp_int and cryptography. I've written a new standalone test program which incorporates all of PuTTY's crypto code, including the mp_int and low-level elliptic curve layers but also going all the way up to the implementations of the MAC, hash, cipher, public key and kex abstractions. The test program itself, 'testcrypt', speaks a simple line-oriented protocol on standard I/O in which you write the name of a function call followed by some inputs, and it gives you back a list of outputs preceded by a line telling you how many there are. Dynamically allocated objects are assigned string ids in the protocol, and there's a 'free' function that tells testcrypt when it can dispose of one. It's possible to speak that protocol by hand, but cumbersome. I've also provided a Python module that wraps it, by running testcrypt as a persistent subprocess and gatewaying all the function calls into things that look reasonably natural to call from Python. The Python module and testcrypt.c both read a carefully formatted header file testcrypt.h which contains the name and signature of every exported function, so it costs minimal effort to expose a given function through this test API. In a few cases it's necessary to write a wrapper in testcrypt.c that makes the function look more friendly, but mostly you don't even need that. (Though that is one of the motivations between a lot of API cleanups I've done recently!) I considered doing Python integration in the more obvious way, by linking parts of the PuTTY code directly into a native-code .so Python module. I decided against it because this way is more flexible: I can run the testcrypt program on its own, or compile it in a way that Python wouldn't play nicely with (I bet compiling just that .so with Leak Sanitiser wouldn't do what you wanted when Python loaded it!), or attach a debugger to it. I can even recompile testcrypt for a different CPU architecture (32- vs 64-bit, or even running it on a different machine over ssh or under emulation) and still layer the nice API on top of that via the local Python interpreter. All I need is a bidirectional data channel.
2019-01-01 19:08:37 +00:00
if consumed:
testcrypt.py: fake some OO syntax. When I'm writing Python using the testcrypt API, I keep finding that I instinctively try to call vtable methods as if they were actual methods of the object. For example, calling key.sign(msg, 0) instead of ssh_key_sign(key, msg, 0). So this change to the Python side of the testcrypt mechanism panders to my inappropriate finger-macros by making them work! The idea is that I define a set of pairs (type, prefix), such that any function whose name begins with the prefix and whose first argument is of that type will be automatically translated into a method on the Python object wrapping a testcrypt value of that type. For example, any function of the form ssh_key_foo(val_ssh_key, other args) will automatically be exposed as a method key.foo(other args), simply because (val_ssh_key, "ssh_key_") appears in the translation table. This is particularly nice for the Python 3 REPL, which will let me tab-complete the right set of method names by knowing the type I'm trying to invoke one on. I haven't decided yet whether I want to switch to using it throughout cryptsuite.py. For namespace-cleanness, I've also renamed all the existing attributes of the Python Value class wrapper so that they start with '_', to leave the space of sensible names clear for the new OOish methods.
2020-02-29 09:48:00 +00:00
arg._consumed()
New test system for mp_int and cryptography. I've written a new standalone test program which incorporates all of PuTTY's crypto code, including the mp_int and low-level elliptic curve layers but also going all the way up to the implementations of the MAC, hash, cipher, public key and kex abstractions. The test program itself, 'testcrypt', speaks a simple line-oriented protocol on standard I/O in which you write the name of a function call followed by some inputs, and it gives you back a list of outputs preceded by a line telling you how many there are. Dynamically allocated objects are assigned string ids in the protocol, and there's a 'free' function that tells testcrypt when it can dispose of one. It's possible to speak that protocol by hand, but cumbersome. I've also provided a Python module that wraps it, by running testcrypt as a persistent subprocess and gatewaying all the function calls into things that look reasonably natural to call from Python. The Python module and testcrypt.c both read a carefully formatted header file testcrypt.h which contains the name and signature of every exported function, so it costs minimal effort to expose a given function through this test API. In a few cases it's necessary to write a wrapper in testcrypt.c that makes the function look more friendly, but mostly you don't even need that. (Though that is one of the motivations between a lot of API cleanups I've done recently!) I considered doing Python integration in the more obvious way, by linking parts of the PuTTY code directly into a native-code .so Python module. I decided against it because this way is more flexible: I can run the testcrypt program on its own, or compile it in a way that Python wouldn't play nicely with (I bet compiling just that .so with Leak Sanitiser wouldn't do what you wanted when Python loaded it!), or attach a debugger to it. I can even recompile testcrypt for a different CPU architecture (32- vs 64-bit, or even running it on a different machine over ssh or under emulation) and still layer the nice API on top of that via the local Python interpreter. All I need is a bidirectional data channel.
2019-01-01 19:08:37 +00:00
return ident
if typename == "uint" and isinstance(arg, numbers.Integral):
return "0x{:x}".format(arg)
RSA generation: option to generate strong primes. A 'strong' prime, as defined by the Handbook of Applied Cryptography, is a prime p such that each of p-1 and p+1 has a large prime factor, and that the large factor q of p-1 is such that q-1 in turn _also_ has a large prime factor. HoAC says that making your RSA key using primes of this form defeats some factoring algorithms - but there are other faster algorithms to which it makes no difference. So this is probably not a useful precaution in practice. However, it has been recommended in the past by some official standards, and it's easy to implement given the new general facility in PrimeCandidateSource that lets you ask for your prime to satisfy an arbitrary modular congruence. (And HoAC also says there's no particular reason _not_ to use strong primes.) So I provide it as an option, just in case anyone wants to select it. The change to the key generation algorithm is entirely in sshrsag.c, and is neatly independent of the prime-generation system in use. If you're using Maurer provable prime generation, then the known factor q of p-1 can be used to help certify p, and the one for q-1 to help with q in turn; if you switch to probabilistic prime generation then you still get an RSA key with the right structure, except that every time the definition says 'prime factor' you just append '(probably)'. (The probabilistic version of this procedure is described as 'Gordon's algorithm' in HoAC section 4.4.2.)
2020-03-02 06:52:09 +00:00
if typename == "boolean":
return "true" if arg else "false"
New test system for mp_int and cryptography. I've written a new standalone test program which incorporates all of PuTTY's crypto code, including the mp_int and low-level elliptic curve layers but also going all the way up to the implementations of the MAC, hash, cipher, public key and kex abstractions. The test program itself, 'testcrypt', speaks a simple line-oriented protocol on standard I/O in which you write the name of a function call followed by some inputs, and it gives you back a list of outputs preceded by a line telling you how many there are. Dynamically allocated objects are assigned string ids in the protocol, and there's a 'free' function that tells testcrypt when it can dispose of one. It's possible to speak that protocol by hand, but cumbersome. I've also provided a Python module that wraps it, by running testcrypt as a persistent subprocess and gatewaying all the function calls into things that look reasonably natural to call from Python. The Python module and testcrypt.c both read a carefully formatted header file testcrypt.h which contains the name and signature of every exported function, so it costs minimal effort to expose a given function through this test API. In a few cases it's necessary to write a wrapper in testcrypt.c that makes the function look more friendly, but mostly you don't even need that. (Though that is one of the motivations between a lot of API cleanups I've done recently!) I considered doing Python integration in the more obvious way, by linking parts of the PuTTY code directly into a native-code .so Python module. I decided against it because this way is more flexible: I can run the testcrypt program on its own, or compile it in a way that Python wouldn't play nicely with (I bet compiling just that .so with Leak Sanitiser wouldn't do what you wanted when Python loaded it!), or attach a debugger to it. I can even recompile testcrypt for a different CPU architecture (32- vs 64-bit, or even running it on a different machine over ssh or under emulation) and still layer the nice API on top of that via the local Python interpreter. All I need is a bidirectional data channel.
2019-01-01 19:08:37 +00:00
if typename in {
Merge the ssh1_cipher type into ssh2_cipher. The aim of this reorganisation is to make it easier to test all the ciphers in PuTTY in a uniform way. It was inconvenient that there were two separate vtable systems for the ciphers used in SSH-1 and SSH-2 with different functionality. Now there's only one type, called ssh_cipher. But really it's the old ssh2_cipher, just renamed: I haven't made any changes to the API on the SSH-2 side. Instead, I've removed ssh1_cipher completely, and adapted the SSH-1 BPP to use the SSH-2 style API. (The relevant differences are that ssh1_cipher encapsulated both the sending and receiving directions in one object - so now ssh1bpp has to make a separate cipher instance per direction - and that ssh1_cipher automatically initialised the IV to all zeroes, which ssh1bpp now has to do by hand.) The previous ssh1_cipher vtable for single-DES has been removed completely, because when converted into the new API it became identical to the SSH-2 single-DES vtable; so now there's just one vtable for DES-CBC which works in both protocols. The other two SSH-1 ciphers each had to stay separate, because 3DES is completely different between SSH-1 and SSH-2 (three layers of CBC structure versus one), and Blowfish varies in endianness and key length between the two. (Actually, while I'm here, I've only just noticed that the SSH-1 Blowfish cipher mis-describes itself in log messages as Blowfish-128. In fact it passes the whole of the input key buffer, which has length SSH1_SESSION_KEY_LENGTH == 32 bytes == 256 bits. So it's actually Blowfish-256, and has been all along!)
2019-01-17 18:06:08 +00:00
"hashalg", "macalg", "keyalg", "cipheralg",
Add implementation of the Argon2 password hash. This is going to be used in the new version of the PPK file format. It was the winner of the Password Hashing Context, which I think makes it a reasonable choice. Argon2 comes in three flavours: one with no data dependency in its memory addressing, one with _deliberate_ data dependency (intended to serialise computation, to hinder parallel brute-forcing), and a hybrid form that starts off data-independent and then switches over to the dependent version once the sensitive input data has been adequately mixed around. I test all three in the test suite; the side-channel tester can only expect Argon2i to pass; and, following the spec's recommendation, I'll be using Argon2id for the actual key file encryption.
2021-02-13 17:30:12 +00:00
"dh_group", "ecdh_alg", "rsaorder", "primegenpolicy",
New post-quantum kex: ML-KEM, and three hybrids of it. As standardised by NIST in FIPS 203, this is a lattice-based post-quantum KEM. Very vaguely, the idea of it is that your public key is a matrix A and vector t, and the private key is the knowledge of how to decompose t into two vectors with all their coefficients small, one transformed by A relative to the other. Encryption of a binary secret starts by turning each bit into one of two maximally separated residues mod a prime q, and then adding 'noise' based on the public key in the form of small increments and decrements mod q, again with some of the noise transformed by A relative to the rest. Decryption uses the knowledge of t's decomposition to align the two sets of noise so that the _large_ changes (which masked the secret from an eavesdropper) cancel out, leaving only a collection of small changes to the original secret vector. Then the vector of input bits can be recovered by assuming that those accumulated small pieces of noise haven't concentrated in any particular residue enough to push it more than half way to the other of its possible starting values. A weird feature of it is that decryption is not a true mathematical inverse of encryption. The assumption that the noise doesn't get large enough to flip any bit of the secret is only probabilistically valid, not a hard guarantee. In other words, key agreement can fail, simply by getting particularly unlucky with the distribution of your random noise! However, the probability of a failure is very low - less than 2^-138 even for ML-KEM-512, and gets even smaller with the larger variants. An awkward feature for our purposes is that the matrix A, containing a large number of residues mod the prime q=3329, is required to be constructed by a process of rejection sampling, i.e. generating random 12-bit values and throwing away the out-of-range ones. That would be a real pain for our side-channel testing system, which generally handles rejection sampling badly (since it necessarily involves data-dependent control flow and timing variation). Fortunately, the matrix and the random seed it was made from are both public: the matrix seed is transmitted as part of the public key, so it's not necessary to try to hide it. Accordingly, I was able to get the implementation to pass testsc by means of not varying the matrix seed between runs, which is justified by the principle of testsc that you vary the _secrets_ to ensure timing is independent of them - and the matrix seed isn't a secret, so you're allowed to keep it the same. The three hybrid algorithms, defined by the current Internet-Draft draft-kampanakis-curdle-ssh-pq-ke, include one hybrid of ML-KEM-768 with Curve25519 in exactly the same way we were already hybridising NTRU Prime with Curve25519, and two more hybrids of ML-KEM with ECDH over a NIST curve. The former hybrid interoperates with the implementation in OpenSSH 9.9; all three interoperate with the fork 'openssh-oqs' at github.com/open-quantum-safe/openssh, and also with the Python library AsyncSSH.
2024-12-07 19:33:39 +00:00
"argon2flavour", "fptype", "httpdigesthash", "mlkem_params"}:
Migrate all Python scripts to Python 3. Most of them are now _mandatory_ P3 scripts, because I'm tired of maintaining everything to be compatible with both versions. The current exceptions are gdb.py (which has to live with whatever gdb gives it), and kh2reg.py (which is actually designed for other people to use, and some of them might still be stuck on P2 for the moment).
2020-03-04 21:23:49 +00:00
arg = coerce_to_bytes(arg)
New test system for mp_int and cryptography. I've written a new standalone test program which incorporates all of PuTTY's crypto code, including the mp_int and low-level elliptic curve layers but also going all the way up to the implementations of the MAC, hash, cipher, public key and kex abstractions. The test program itself, 'testcrypt', speaks a simple line-oriented protocol on standard I/O in which you write the name of a function call followed by some inputs, and it gives you back a list of outputs preceded by a line telling you how many there are. Dynamically allocated objects are assigned string ids in the protocol, and there's a 'free' function that tells testcrypt when it can dispose of one. It's possible to speak that protocol by hand, but cumbersome. I've also provided a Python module that wraps it, by running testcrypt as a persistent subprocess and gatewaying all the function calls into things that look reasonably natural to call from Python. The Python module and testcrypt.c both read a carefully formatted header file testcrypt.h which contains the name and signature of every exported function, so it costs minimal effort to expose a given function through this test API. In a few cases it's necessary to write a wrapper in testcrypt.c that makes the function look more friendly, but mostly you don't even need that. (Though that is one of the motivations between a lot of API cleanups I've done recently!) I considered doing Python integration in the more obvious way, by linking parts of the PuTTY code directly into a native-code .so Python module. I decided against it because this way is more flexible: I can run the testcrypt program on its own, or compile it in a way that Python wouldn't play nicely with (I bet compiling just that .so with Leak Sanitiser wouldn't do what you wanted when Python loaded it!), or attach a debugger to it. I can even recompile testcrypt for a different CPU architecture (32- vs 64-bit, or even running it on a different machine over ssh or under emulation) and still layer the nice API on top of that via the local Python interpreter. All I need is a bidirectional data channel.
2019-01-01 19:08:37 +00:00
if isinstance(arg, bytes) and b" " not in arg:
testcrypt: introduce and use 'checkenum' protocol query. This allows the Python side of testcrypt to check in advance if a given string is a valid element of an enumeration, and if not, cleanly throw a Python-level exception without terminating the testcrypt subprocess. Should be useful in both manual use (when I'm trying something out by hand and make a typo or misremember a spelling), and automated use (if I make the same kind of error in cryptsuite.py then the exception dump will make more sense). In order to do this, the new handle_checkenum() function has to recognise all the enumerated types by name and match them up to their lookup functions - which is just the kind of thing that can now be done easily be reincluding testcrypt-enum.h with different #defines.
2021-11-22 18:39:45 +00:00
dictkey = (typename, arg)
if dictkey not in checked_enum_values:
retwords = childprocess.funcall("checkenum", [typename, arg])
assert len(retwords) == 1
checked_enum_values[dictkey] = (retwords[0] == b"ok")
if checked_enum_values[dictkey]:
return arg
New 'Pockle' object, for verifying primality. This implements an extended form of primality verification using certificates based on Pocklington's theorem. You make a Pockle object, and then try to convince it that one number after another is prime, by means of providing it with a list of prime factors of p-1 and a primitive root. (Or just by saying 'this prime is small enough for you to check yourself'.) Pocklington's theorem requires you to have factors of p-1 whose product is at least the square root of p. I've extended that to support factorisations only as big as the cube root, via an extension of the theorem given in Maurer's paper on generating provable primes. The Pockle object is more or less write-only: it has no methods for reading out its contents. Its only output channel is the return value when you try to insert a prime into it: if it isn't sufficiently convinced that your prime is prime, it will return an error code. So anything for which it returns POCKLE_OK you can be confident of. I'm going to use this for provable prime generation. But exposing this part of the system as an object in its own right means I can write a set of unit tests for this specifically. My negative tests exercise all the different ways a certification can be erroneous or inadequate; the positive tests include proofs of primality of various primes used in elliptic-curve crypto. The Poly1305 proof in particular is taken from a proof in DJB's paper, which has exactly the form of a Pocklington certificate only written in English.
2020-02-23 15:16:30 +00:00
if typename == "mpint_list":
sublist = [make_argword(len(arg), ("uint", False),
testcrypt.py: use parameter names in diagnostics. Making a virtue of the necessity of adding parameter names to testcrypt.h a couple of commits ago, we can now use those names to improve diagnostics, so that if you use the wrong type in a Python function call the error message will tell you the name as well as the index of the offending argument. Also, the repr() text for the function itself will now print a full prototype (albeit in a nasty hybrid of C, Python and testcrypt.h syntax) which shows all the parameter names. That should be handy when trying to remember the order of arguments at the REPL prompt.
2021-11-21 13:21:01 +00:00
fnname, argindex, argname, to_preserve)]
New 'Pockle' object, for verifying primality. This implements an extended form of primality verification using certificates based on Pocklington's theorem. You make a Pockle object, and then try to convince it that one number after another is prime, by means of providing it with a list of prime factors of p-1 and a primitive root. (Or just by saying 'this prime is small enough for you to check yourself'.) Pocklington's theorem requires you to have factors of p-1 whose product is at least the square root of p. I've extended that to support factorisations only as big as the cube root, via an extension of the theorem given in Maurer's paper on generating provable primes. The Pockle object is more or less write-only: it has no methods for reading out its contents. Its only output channel is the return value when you try to insert a prime into it: if it isn't sufficiently convinced that your prime is prime, it will return an error code. So anything for which it returns POCKLE_OK you can be confident of. I'm going to use this for provable prime generation. But exposing this part of the system as an object in its own right means I can write a set of unit tests for this specifically. My negative tests exercise all the different ways a certification can be erroneous or inadequate; the positive tests include proofs of primality of various primes used in elliptic-curve crypto. The Poly1305 proof in particular is taken from a proof in DJB's paper, which has exactly the form of a Pocklington certificate only written in English.
2020-02-23 15:16:30 +00:00
for val in arg:
sublist.append(make_argword(val, ("val_mpint", False),
testcrypt.py: use parameter names in diagnostics. Making a virtue of the necessity of adding parameter names to testcrypt.h a couple of commits ago, we can now use those names to improve diagnostics, so that if you use the wrong type in a Python function call the error message will tell you the name as well as the index of the offending argument. Also, the repr() text for the function itself will now print a full prototype (albeit in a nasty hybrid of C, Python and testcrypt.h syntax) which shows all the parameter names. That should be handy when trying to remember the order of arguments at the REPL prompt.
2021-11-21 13:21:01 +00:00
fnname, argindex, argname, to_preserve))
Migrate all Python scripts to Python 3. Most of them are now _mandatory_ P3 scripts, because I'm tired of maintaining everything to be compatible with both versions. The current exceptions are gdb.py (which has to live with whatever gdb gives it), and kh2reg.py (which is actually designed for other people to use, and some of them might still be stuck on P2 for the moment).
2020-03-04 21:23:49 +00:00
return b" ".join(coerce_to_bytes(sub) for sub in sublist)
Implement OpenSSH 9.x's NTRU Prime / Curve25519 kex. This consists of DJB's 'Streamlined NTRU Prime' quantum-resistant cryptosystem, currently in round 3 of the NIST post-quantum key exchange competition; it's run in parallel with ordinary Curve25519, and generates a shared secret combining the output of both systems. (Hence, even if you don't trust this newfangled NTRU Prime thing at all, it's at least no _less_ secure than the kex you were using already.) As the OpenSSH developers point out, key exchange is the most urgent thing to make quantum-resistant, even before working quantum computers big enough to break crypto become available, because a break of the kex algorithm can be applied retroactively to recordings of your past sessions. By contrast, authentication is a real-time protocol, and can only be broken by a quantum computer if there's one available to attack you _already_. I've implemented both sides of the mechanism, so that PuTTY and Uppity both support it. In my initial testing, the two sides can both interoperate with the appropriate half of OpenSSH, and also (of course, but it would be embarrassing to mess it up) with each other.
2022-04-15 16:19:47 +00:00
if typename == "int16_list":
sublist = [make_argword(len(arg), ("uint", False),
fnname, argindex, argname, to_preserve)]
for val in arg:
sublist.append(make_argword(val & 0xFFFF, ("uint", False),
fnname, argindex, argname, to_preserve))
return b" ".join(coerce_to_bytes(sub) for sub in sublist)
New test system for mp_int and cryptography. I've written a new standalone test program which incorporates all of PuTTY's crypto code, including the mp_int and low-level elliptic curve layers but also going all the way up to the implementations of the MAC, hash, cipher, public key and kex abstractions. The test program itself, 'testcrypt', speaks a simple line-oriented protocol on standard I/O in which you write the name of a function call followed by some inputs, and it gives you back a list of outputs preceded by a line telling you how many there are. Dynamically allocated objects are assigned string ids in the protocol, and there's a 'free' function that tells testcrypt when it can dispose of one. It's possible to speak that protocol by hand, but cumbersome. I've also provided a Python module that wraps it, by running testcrypt as a persistent subprocess and gatewaying all the function calls into things that look reasonably natural to call from Python. The Python module and testcrypt.c both read a carefully formatted header file testcrypt.h which contains the name and signature of every exported function, so it costs minimal effort to expose a given function through this test API. In a few cases it's necessary to write a wrapper in testcrypt.c that makes the function look more friendly, but mostly you don't even need that. (Though that is one of the motivations between a lot of API cleanups I've done recently!) I considered doing Python integration in the more obvious way, by linking parts of the PuTTY code directly into a native-code .so Python module. I decided against it because this way is more flexible: I can run the testcrypt program on its own, or compile it in a way that Python wouldn't play nicely with (I bet compiling just that .so with Leak Sanitiser wouldn't do what you wanted when Python loaded it!), or attach a debugger to it. I can even recompile testcrypt for a different CPU architecture (32- vs 64-bit, or even running it on a different machine over ssh or under emulation) and still layer the nice API on top of that via the local Python interpreter. All I need is a bidirectional data channel.
2019-01-01 19:08:37 +00:00
raise TypeError(
testcrypt.py: use parameter names in diagnostics. Making a virtue of the necessity of adding parameter names to testcrypt.h a couple of commits ago, we can now use those names to improve diagnostics, so that if you use the wrong type in a Python function call the error message will tell you the name as well as the index of the offending argument. Also, the repr() text for the function itself will now print a full prototype (albeit in a nasty hybrid of C, Python and testcrypt.h syntax) which shows all the parameter names. That should be handy when trying to remember the order of arguments at the REPL prompt.
2021-11-21 13:21:01 +00:00
"Can't convert {}() argument #{:d} ({}) to {} (value was {!r})".format(
fnname, argindex, argname, typename, arg))
New test system for mp_int and cryptography. I've written a new standalone test program which incorporates all of PuTTY's crypto code, including the mp_int and low-level elliptic curve layers but also going all the way up to the implementations of the MAC, hash, cipher, public key and kex abstractions. The test program itself, 'testcrypt', speaks a simple line-oriented protocol on standard I/O in which you write the name of a function call followed by some inputs, and it gives you back a list of outputs preceded by a line telling you how many there are. Dynamically allocated objects are assigned string ids in the protocol, and there's a 'free' function that tells testcrypt when it can dispose of one. It's possible to speak that protocol by hand, but cumbersome. I've also provided a Python module that wraps it, by running testcrypt as a persistent subprocess and gatewaying all the function calls into things that look reasonably natural to call from Python. The Python module and testcrypt.c both read a carefully formatted header file testcrypt.h which contains the name and signature of every exported function, so it costs minimal effort to expose a given function through this test API. In a few cases it's necessary to write a wrapper in testcrypt.c that makes the function look more friendly, but mostly you don't even need that. (Though that is one of the motivations between a lot of API cleanups I've done recently!) I considered doing Python integration in the more obvious way, by linking parts of the PuTTY code directly into a native-code .so Python module. I decided against it because this way is more flexible: I can run the testcrypt program on its own, or compile it in a way that Python wouldn't play nicely with (I bet compiling just that .so with Leak Sanitiser wouldn't do what you wanted when Python loaded it!), or attach a debugger to it. I can even recompile testcrypt for a different CPU architecture (32- vs 64-bit, or even running it on a different machine over ssh or under emulation) and still layer the nice API on top of that via the local Python interpreter. All I need is a bidirectional data channel.
2019-01-01 19:08:37 +00:00
cmdgen: add a --dump option. Also spelled '-O text', this takes a public or private key as input, and produces on standard output a dump of all the actual numbers involved in the key: the exponent and modulus for RSA, the p,q,g,y parameters for DSA, the affine x and y coordinates of the public elliptic curve point for ECC keys, and all the extra bits and pieces in the private keys too. Partly I expect this to be useful to me for debugging: I've had to paste key files a few too many times through base64 decoders and hex dump tools, then manually decode SSH marshalling and paste the result into the Python REPL to get an integer object. Now I should be able to get _straight_ to text I can paste into Python. But also, it's a way that other applications can use the key generator: if you need to generate, say, an RSA key in some format I don't support (I've recently heard of an XML-based one, for example), then you can run 'puttygen -t rsa --dump' and have it print the elements of a freshly generated keypair on standard output, and then all you have to do is understand the output format.
2020-02-17 19:53:19 +00:00
def unpack_string(identifier):
retwords = childprocess.funcall("getstring", [identifier])
childprocess.funcall("free", [identifier])
return re.sub(b"%[0-9A-F][0-9A-F]",
Migrate all Python scripts to Python 3. Most of them are now _mandatory_ P3 scripts, because I'm tired of maintaining everything to be compatible with both versions. The current exceptions are gdb.py (which has to live with whatever gdb gives it), and kh2reg.py (which is actually designed for other people to use, and some of them might still be stuck on P2 for the moment).
2020-03-04 21:23:49 +00:00
lambda m: bytes([int(m.group(0)[1:], 16)]),
cmdgen: add a --dump option. Also spelled '-O text', this takes a public or private key as input, and produces on standard output a dump of all the actual numbers involved in the key: the exponent and modulus for RSA, the p,q,g,y parameters for DSA, the affine x and y coordinates of the public elliptic curve point for ECC keys, and all the extra bits and pieces in the private keys too. Partly I expect this to be useful to me for debugging: I've had to paste key files a few too many times through base64 decoders and hex dump tools, then manually decode SSH marshalling and paste the result into the Python REPL to get an integer object. Now I should be able to get _straight_ to text I can paste into Python. But also, it's a way that other applications can use the key generator: if you need to generate, say, an RSA key in some format I don't support (I've recently heard of an XML-based one, for example), then you can run 'puttygen -t rsa --dump' and have it print the elements of a freshly generated keypair on standard output, and then all you have to do is understand the output format.
2020-02-17 19:53:19 +00:00
retwords[0])
def unpack_mp(identifier):
retwords = childprocess.funcall("mp_dump", [identifier])
childprocess.funcall("free", [identifier])
return int(retwords[0], 16)
New test system for mp_int and cryptography. I've written a new standalone test program which incorporates all of PuTTY's crypto code, including the mp_int and low-level elliptic curve layers but also going all the way up to the implementations of the MAC, hash, cipher, public key and kex abstractions. The test program itself, 'testcrypt', speaks a simple line-oriented protocol on standard I/O in which you write the name of a function call followed by some inputs, and it gives you back a list of outputs preceded by a line telling you how many there are. Dynamically allocated objects are assigned string ids in the protocol, and there's a 'free' function that tells testcrypt when it can dispose of one. It's possible to speak that protocol by hand, but cumbersome. I've also provided a Python module that wraps it, by running testcrypt as a persistent subprocess and gatewaying all the function calls into things that look reasonably natural to call from Python. The Python module and testcrypt.c both read a carefully formatted header file testcrypt.h which contains the name and signature of every exported function, so it costs minimal effort to expose a given function through this test API. In a few cases it's necessary to write a wrapper in testcrypt.c that makes the function look more friendly, but mostly you don't even need that. (Though that is one of the motivations between a lot of API cleanups I've done recently!) I considered doing Python integration in the more obvious way, by linking parts of the PuTTY code directly into a native-code .so Python module. I decided against it because this way is more flexible: I can run the testcrypt program on its own, or compile it in a way that Python wouldn't play nicely with (I bet compiling just that .so with Leak Sanitiser wouldn't do what you wanted when Python loaded it!), or attach a debugger to it. I can even recompile testcrypt for a different CPU architecture (32- vs 64-bit, or even running it on a different machine over ssh or under emulation) and still layer the nice API on top of that via the local Python interpreter. All I need is a bidirectional data channel.
2019-01-01 19:08:37 +00:00
def make_retval(rettype, word, unpack_strings):
testcrypt: allow ssh2_cipher_new to return NULL. No cipher construction function _currently_ returns NULL, but one's about to start, so the testcrypt system will have to be able to cope. This is the first time a function in the testcrypt API has had an 'opt' type as its return value rather than an argument. But it works just the same in reverse: the wire protocol emits the special identifer "NULL" when the optional return value is absent, and the Python module catches that and rewrites it as Python 'None'.
2019-01-11 06:47:39 +00:00
if rettype.startswith("opt_"):
testcrypt / cryptsuite: another set of Python 3 fixes. One of these days I'll manage not to mess this up in every new test I add ... perhaps.
2019-01-23 23:40:32 +00:00
if word == b"NULL":
testcrypt: allow ssh2_cipher_new to return NULL. No cipher construction function _currently_ returns NULL, but one's about to start, so the testcrypt system will have to be able to cope. This is the first time a function in the testcrypt API has had an 'opt' type as its return value rather than an argument. But it works just the same in reverse: the wire protocol emits the special identifer "NULL" when the optional return value is absent, and the Python module catches that and rewrites it as Python 'None'.
2019-01-11 06:47:39 +00:00
return None
rettype = rettype[4:]
New test system for mp_int and cryptography. I've written a new standalone test program which incorporates all of PuTTY's crypto code, including the mp_int and low-level elliptic curve layers but also going all the way up to the implementations of the MAC, hash, cipher, public key and kex abstractions. The test program itself, 'testcrypt', speaks a simple line-oriented protocol on standard I/O in which you write the name of a function call followed by some inputs, and it gives you back a list of outputs preceded by a line telling you how many there are. Dynamically allocated objects are assigned string ids in the protocol, and there's a 'free' function that tells testcrypt when it can dispose of one. It's possible to speak that protocol by hand, but cumbersome. I've also provided a Python module that wraps it, by running testcrypt as a persistent subprocess and gatewaying all the function calls into things that look reasonably natural to call from Python. The Python module and testcrypt.c both read a carefully formatted header file testcrypt.h which contains the name and signature of every exported function, so it costs minimal effort to expose a given function through this test API. In a few cases it's necessary to write a wrapper in testcrypt.c that makes the function look more friendly, but mostly you don't even need that. (Though that is one of the motivations between a lot of API cleanups I've done recently!) I considered doing Python integration in the more obvious way, by linking parts of the PuTTY code directly into a native-code .so Python module. I decided against it because this way is more flexible: I can run the testcrypt program on its own, or compile it in a way that Python wouldn't play nicely with (I bet compiling just that .so with Leak Sanitiser wouldn't do what you wanted when Python loaded it!), or attach a debugger to it. I can even recompile testcrypt for a different CPU architecture (32- vs 64-bit, or even running it on a different machine over ssh or under emulation) and still layer the nice API on top of that via the local Python interpreter. All I need is a bidirectional data channel.
2019-01-01 19:08:37 +00:00
if rettype == "val_string" and unpack_strings:
cmdgen: add a --dump option. Also spelled '-O text', this takes a public or private key as input, and produces on standard output a dump of all the actual numbers involved in the key: the exponent and modulus for RSA, the p,q,g,y parameters for DSA, the affine x and y coordinates of the public elliptic curve point for ECC keys, and all the extra bits and pieces in the private keys too. Partly I expect this to be useful to me for debugging: I've had to paste key files a few too many times through base64 decoders and hex dump tools, then manually decode SSH marshalling and paste the result into the Python REPL to get an integer object. Now I should be able to get _straight_ to text I can paste into Python. But also, it's a way that other applications can use the key generator: if you need to generate, say, an RSA key in some format I don't support (I've recently heard of an XML-based one, for example), then you can run 'puttygen -t rsa --dump' and have it print the elements of a freshly generated keypair on standard output, and then all you have to do is understand the output format.
2020-02-17 19:53:19 +00:00
return unpack_string(word)
if rettype == "val_keycomponents":
kc = {}
retwords = childprocess.funcall("key_components_count", [word])
for i in range(int(retwords[0], 0)):
args = [word, "{:d}".format(i)]
retwords = childprocess.funcall("key_components_nth_name", args)
kc_key = unpack_string(retwords[0])
retwords = childprocess.funcall("key_components_nth_str", args)
if retwords[0] != b"NULL":
kc_value = unpack_string(retwords[0]).decode("ASCII")
else:
retwords = childprocess.funcall("key_components_nth_mp", args)
kc_value = unpack_mp(retwords[0])
kc[kc_key.decode("ASCII")] = kc_value
New test system for mp_int and cryptography. I've written a new standalone test program which incorporates all of PuTTY's crypto code, including the mp_int and low-level elliptic curve layers but also going all the way up to the implementations of the MAC, hash, cipher, public key and kex abstractions. The test program itself, 'testcrypt', speaks a simple line-oriented protocol on standard I/O in which you write the name of a function call followed by some inputs, and it gives you back a list of outputs preceded by a line telling you how many there are. Dynamically allocated objects are assigned string ids in the protocol, and there's a 'free' function that tells testcrypt when it can dispose of one. It's possible to speak that protocol by hand, but cumbersome. I've also provided a Python module that wraps it, by running testcrypt as a persistent subprocess and gatewaying all the function calls into things that look reasonably natural to call from Python. The Python module and testcrypt.c both read a carefully formatted header file testcrypt.h which contains the name and signature of every exported function, so it costs minimal effort to expose a given function through this test API. In a few cases it's necessary to write a wrapper in testcrypt.c that makes the function look more friendly, but mostly you don't even need that. (Though that is one of the motivations between a lot of API cleanups I've done recently!) I considered doing Python integration in the more obvious way, by linking parts of the PuTTY code directly into a native-code .so Python module. I decided against it because this way is more flexible: I can run the testcrypt program on its own, or compile it in a way that Python wouldn't play nicely with (I bet compiling just that .so with Leak Sanitiser wouldn't do what you wanted when Python loaded it!), or attach a debugger to it. I can even recompile testcrypt for a different CPU architecture (32- vs 64-bit, or even running it on a different machine over ssh or under emulation) and still layer the nice API on top of that via the local Python interpreter. All I need is a bidirectional data channel.
2019-01-01 19:08:37 +00:00
childprocess.funcall("free", [word])
cmdgen: add a --dump option. Also spelled '-O text', this takes a public or private key as input, and produces on standard output a dump of all the actual numbers involved in the key: the exponent and modulus for RSA, the p,q,g,y parameters for DSA, the affine x and y coordinates of the public elliptic curve point for ECC keys, and all the extra bits and pieces in the private keys too. Partly I expect this to be useful to me for debugging: I've had to paste key files a few too many times through base64 decoders and hex dump tools, then manually decode SSH marshalling and paste the result into the Python REPL to get an integer object. Now I should be able to get _straight_ to text I can paste into Python. But also, it's a way that other applications can use the key generator: if you need to generate, say, an RSA key in some format I don't support (I've recently heard of an XML-based one, for example), then you can run 'puttygen -t rsa --dump' and have it print the elements of a freshly generated keypair on standard output, and then all you have to do is understand the output format.
2020-02-17 19:53:19 +00:00
return kc
New test system for mp_int and cryptography. I've written a new standalone test program which incorporates all of PuTTY's crypto code, including the mp_int and low-level elliptic curve layers but also going all the way up to the implementations of the MAC, hash, cipher, public key and kex abstractions. The test program itself, 'testcrypt', speaks a simple line-oriented protocol on standard I/O in which you write the name of a function call followed by some inputs, and it gives you back a list of outputs preceded by a line telling you how many there are. Dynamically allocated objects are assigned string ids in the protocol, and there's a 'free' function that tells testcrypt when it can dispose of one. It's possible to speak that protocol by hand, but cumbersome. I've also provided a Python module that wraps it, by running testcrypt as a persistent subprocess and gatewaying all the function calls into things that look reasonably natural to call from Python. The Python module and testcrypt.c both read a carefully formatted header file testcrypt.h which contains the name and signature of every exported function, so it costs minimal effort to expose a given function through this test API. In a few cases it's necessary to write a wrapper in testcrypt.c that makes the function look more friendly, but mostly you don't even need that. (Though that is one of the motivations between a lot of API cleanups I've done recently!) I considered doing Python integration in the more obvious way, by linking parts of the PuTTY code directly into a native-code .so Python module. I decided against it because this way is more flexible: I can run the testcrypt program on its own, or compile it in a way that Python wouldn't play nicely with (I bet compiling just that .so with Leak Sanitiser wouldn't do what you wanted when Python loaded it!), or attach a debugger to it. I can even recompile testcrypt for a different CPU architecture (32- vs 64-bit, or even running it on a different machine over ssh or under emulation) and still layer the nice API on top of that via the local Python interpreter. All I need is a bidirectional data channel.
2019-01-01 19:08:37 +00:00
if rettype.startswith("val_"):
return Value(rettype, word)
testcrypt: handle int as a return type. I accepted both 'int' and 'uint' as function argument types, but hadn't previously noticed that only 'uint' is handled properly as a return type. Now both are.
2020-01-06 19:57:51 +00:00
elif rettype == "int" or rettype == "uint":
New test system for mp_int and cryptography. I've written a new standalone test program which incorporates all of PuTTY's crypto code, including the mp_int and low-level elliptic curve layers but also going all the way up to the implementations of the MAC, hash, cipher, public key and kex abstractions. The test program itself, 'testcrypt', speaks a simple line-oriented protocol on standard I/O in which you write the name of a function call followed by some inputs, and it gives you back a list of outputs preceded by a line telling you how many there are. Dynamically allocated objects are assigned string ids in the protocol, and there's a 'free' function that tells testcrypt when it can dispose of one. It's possible to speak that protocol by hand, but cumbersome. I've also provided a Python module that wraps it, by running testcrypt as a persistent subprocess and gatewaying all the function calls into things that look reasonably natural to call from Python. The Python module and testcrypt.c both read a carefully formatted header file testcrypt.h which contains the name and signature of every exported function, so it costs minimal effort to expose a given function through this test API. In a few cases it's necessary to write a wrapper in testcrypt.c that makes the function look more friendly, but mostly you don't even need that. (Though that is one of the motivations between a lot of API cleanups I've done recently!) I considered doing Python integration in the more obvious way, by linking parts of the PuTTY code directly into a native-code .so Python module. I decided against it because this way is more flexible: I can run the testcrypt program on its own, or compile it in a way that Python wouldn't play nicely with (I bet compiling just that .so with Leak Sanitiser wouldn't do what you wanted when Python loaded it!), or attach a debugger to it. I can even recompile testcrypt for a different CPU architecture (32- vs 64-bit, or even running it on a different machine over ssh or under emulation) and still layer the nice API on top of that via the local Python interpreter. All I need is a bidirectional data channel.
2019-01-01 19:08:37 +00:00
return int(word, 0)
elif rettype == "boolean":
assert word == b"true" or word == b"false"
return word == b"true"
Add some tests of Miller-Rabin to cryptsuite. I'm about to rewrite the Miller-Rabin testing code, so let's start by introducing a test suite that the old version passes, and then I can make sure the new one does too.
2021-08-27 16:43:40 +00:00
elif rettype in {"pocklestatus", "mr_result"}:
New 'Pockle' object, for verifying primality. This implements an extended form of primality verification using certificates based on Pocklington's theorem. You make a Pockle object, and then try to convince it that one number after another is prime, by means of providing it with a list of prime factors of p-1 and a primitive root. (Or just by saying 'this prime is small enough for you to check yourself'.) Pocklington's theorem requires you to have factors of p-1 whose product is at least the square root of p. I've extended that to support factorisations only as big as the cube root, via an extension of the theorem given in Maurer's paper on generating provable primes. The Pockle object is more or less write-only: it has no methods for reading out its contents. Its only output channel is the return value when you try to insert a prime into it: if it isn't sufficiently convinced that your prime is prime, it will return an error code. So anything for which it returns POCKLE_OK you can be confident of. I'm going to use this for provable prime generation. But exposing this part of the system as an object in its own right means I can write a set of unit tests for this specifically. My negative tests exercise all the different ways a certification can be erroneous or inadequate; the positive tests include proofs of primality of various primes used in elliptic-curve crypto. The Poly1305 proof in particular is taken from a proof in DJB's paper, which has exactly the form of a Pocklington certificate only written in English.
2020-02-23 15:16:30 +00:00
return word.decode("ASCII")
Implement OpenSSH 9.x's NTRU Prime / Curve25519 kex. This consists of DJB's 'Streamlined NTRU Prime' quantum-resistant cryptosystem, currently in round 3 of the NIST post-quantum key exchange competition; it's run in parallel with ordinary Curve25519, and generates a shared secret combining the output of both systems. (Hence, even if you don't trust this newfangled NTRU Prime thing at all, it's at least no _less_ secure than the kex you were using already.) As the OpenSSH developers point out, key exchange is the most urgent thing to make quantum-resistant, even before working quantum computers big enough to break crypto become available, because a break of the kex algorithm can be applied retroactively to recordings of your past sessions. By contrast, authentication is a real-time protocol, and can only be broken by a quantum computer if there's one available to attack you _already_. I've implemented both sides of the mechanism, so that PuTTY and Uppity both support it. In my initial testing, the two sides can both interoperate with the appropriate half of OpenSSH, and also (of course, but it would be embarrassing to mess it up) with each other.
2022-04-15 16:19:47 +00:00
elif rettype == "int16_list":
return list(map(int, word.split(b',')))
New test system for mp_int and cryptography. I've written a new standalone test program which incorporates all of PuTTY's crypto code, including the mp_int and low-level elliptic curve layers but also going all the way up to the implementations of the MAC, hash, cipher, public key and kex abstractions. The test program itself, 'testcrypt', speaks a simple line-oriented protocol on standard I/O in which you write the name of a function call followed by some inputs, and it gives you back a list of outputs preceded by a line telling you how many there are. Dynamically allocated objects are assigned string ids in the protocol, and there's a 'free' function that tells testcrypt when it can dispose of one. It's possible to speak that protocol by hand, but cumbersome. I've also provided a Python module that wraps it, by running testcrypt as a persistent subprocess and gatewaying all the function calls into things that look reasonably natural to call from Python. The Python module and testcrypt.c both read a carefully formatted header file testcrypt.h which contains the name and signature of every exported function, so it costs minimal effort to expose a given function through this test API. In a few cases it's necessary to write a wrapper in testcrypt.c that makes the function look more friendly, but mostly you don't even need that. (Though that is one of the motivations between a lot of API cleanups I've done recently!) I considered doing Python integration in the more obvious way, by linking parts of the PuTTY code directly into a native-code .so Python module. I decided against it because this way is more flexible: I can run the testcrypt program on its own, or compile it in a way that Python wouldn't play nicely with (I bet compiling just that .so with Leak Sanitiser wouldn't do what you wanted when Python loaded it!), or attach a debugger to it. I can even recompile testcrypt for a different CPU architecture (32- vs 64-bit, or even running it on a different machine over ssh or under emulation) and still layer the nice API on top of that via the local Python interpreter. All I need is a bidirectional data channel.
2019-01-01 19:08:37 +00:00
raise TypeError("Can't deal with return value {!r} of type {!r}"
testcrypt: fix malformatted error message. I managed to get two format parameters reversed in the message when a return type is unhandled.
2020-01-06 19:58:01 +00:00
.format(word, rettype))
New test system for mp_int and cryptography. I've written a new standalone test program which incorporates all of PuTTY's crypto code, including the mp_int and low-level elliptic curve layers but also going all the way up to the implementations of the MAC, hash, cipher, public key and kex abstractions. The test program itself, 'testcrypt', speaks a simple line-oriented protocol on standard I/O in which you write the name of a function call followed by some inputs, and it gives you back a list of outputs preceded by a line telling you how many there are. Dynamically allocated objects are assigned string ids in the protocol, and there's a 'free' function that tells testcrypt when it can dispose of one. It's possible to speak that protocol by hand, but cumbersome. I've also provided a Python module that wraps it, by running testcrypt as a persistent subprocess and gatewaying all the function calls into things that look reasonably natural to call from Python. The Python module and testcrypt.c both read a carefully formatted header file testcrypt.h which contains the name and signature of every exported function, so it costs minimal effort to expose a given function through this test API. In a few cases it's necessary to write a wrapper in testcrypt.c that makes the function look more friendly, but mostly you don't even need that. (Though that is one of the motivations between a lot of API cleanups I've done recently!) I considered doing Python integration in the more obvious way, by linking parts of the PuTTY code directly into a native-code .so Python module. I decided against it because this way is more flexible: I can run the testcrypt program on its own, or compile it in a way that Python wouldn't play nicely with (I bet compiling just that .so with Leak Sanitiser wouldn't do what you wanted when Python loaded it!), or attach a debugger to it. I can even recompile testcrypt for a different CPU architecture (32- vs 64-bit, or even running it on a different machine over ssh or under emulation) and still layer the nice API on top of that via the local Python interpreter. All I need is a bidirectional data channel.
2019-01-01 19:08:37 +00:00
def make_retvals(rettypes, retwords, unpack_strings=True):
assert len(rettypes) == len(retwords) # FIXME: better exception
return [make_retval(rettype, word, unpack_strings)
for rettype, word in zip(rettypes, retwords)]
class Function(object):
testcrypt.py: use parameter names in diagnostics. Making a virtue of the necessity of adding parameter names to testcrypt.h a couple of commits ago, we can now use those names to improve diagnostics, so that if you use the wrong type in a Python function call the error message will tell you the name as well as the index of the offending argument. Also, the repr() text for the function itself will now print a full prototype (albeit in a nasty hybrid of C, Python and testcrypt.h syntax) which shows all the parameter names. That should be handy when trying to remember the order of arguments at the REPL prompt.
2021-11-21 13:21:01 +00:00
def __init__(self, fnname, rettypes, retnames, argtypes, argnames):
New test system for mp_int and cryptography. I've written a new standalone test program which incorporates all of PuTTY's crypto code, including the mp_int and low-level elliptic curve layers but also going all the way up to the implementations of the MAC, hash, cipher, public key and kex abstractions. The test program itself, 'testcrypt', speaks a simple line-oriented protocol on standard I/O in which you write the name of a function call followed by some inputs, and it gives you back a list of outputs preceded by a line telling you how many there are. Dynamically allocated objects are assigned string ids in the protocol, and there's a 'free' function that tells testcrypt when it can dispose of one. It's possible to speak that protocol by hand, but cumbersome. I've also provided a Python module that wraps it, by running testcrypt as a persistent subprocess and gatewaying all the function calls into things that look reasonably natural to call from Python. The Python module and testcrypt.c both read a carefully formatted header file testcrypt.h which contains the name and signature of every exported function, so it costs minimal effort to expose a given function through this test API. In a few cases it's necessary to write a wrapper in testcrypt.c that makes the function look more friendly, but mostly you don't even need that. (Though that is one of the motivations between a lot of API cleanups I've done recently!) I considered doing Python integration in the more obvious way, by linking parts of the PuTTY code directly into a native-code .so Python module. I decided against it because this way is more flexible: I can run the testcrypt program on its own, or compile it in a way that Python wouldn't play nicely with (I bet compiling just that .so with Leak Sanitiser wouldn't do what you wanted when Python loaded it!), or attach a debugger to it. I can even recompile testcrypt for a different CPU architecture (32- vs 64-bit, or even running it on a different machine over ssh or under emulation) and still layer the nice API on top of that via the local Python interpreter. All I need is a bidirectional data channel.
2019-01-01 19:08:37 +00:00
self.fnname = fnname
self.rettypes = rettypes
testcrypt.py: use parameter names in diagnostics. Making a virtue of the necessity of adding parameter names to testcrypt.h a couple of commits ago, we can now use those names to improve diagnostics, so that if you use the wrong type in a Python function call the error message will tell you the name as well as the index of the offending argument. Also, the repr() text for the function itself will now print a full prototype (albeit in a nasty hybrid of C, Python and testcrypt.h syntax) which shows all the parameter names. That should be handy when trying to remember the order of arguments at the REPL prompt.
2021-11-21 13:21:01 +00:00
self.retnames = retnames
New test system for mp_int and cryptography. I've written a new standalone test program which incorporates all of PuTTY's crypto code, including the mp_int and low-level elliptic curve layers but also going all the way up to the implementations of the MAC, hash, cipher, public key and kex abstractions. The test program itself, 'testcrypt', speaks a simple line-oriented protocol on standard I/O in which you write the name of a function call followed by some inputs, and it gives you back a list of outputs preceded by a line telling you how many there are. Dynamically allocated objects are assigned string ids in the protocol, and there's a 'free' function that tells testcrypt when it can dispose of one. It's possible to speak that protocol by hand, but cumbersome. I've also provided a Python module that wraps it, by running testcrypt as a persistent subprocess and gatewaying all the function calls into things that look reasonably natural to call from Python. The Python module and testcrypt.c both read a carefully formatted header file testcrypt.h which contains the name and signature of every exported function, so it costs minimal effort to expose a given function through this test API. In a few cases it's necessary to write a wrapper in testcrypt.c that makes the function look more friendly, but mostly you don't even need that. (Though that is one of the motivations between a lot of API cleanups I've done recently!) I considered doing Python integration in the more obvious way, by linking parts of the PuTTY code directly into a native-code .so Python module. I decided against it because this way is more flexible: I can run the testcrypt program on its own, or compile it in a way that Python wouldn't play nicely with (I bet compiling just that .so with Leak Sanitiser wouldn't do what you wanted when Python loaded it!), or attach a debugger to it. I can even recompile testcrypt for a different CPU architecture (32- vs 64-bit, or even running it on a different machine over ssh or under emulation) and still layer the nice API on top of that via the local Python interpreter. All I need is a bidirectional data channel.
2019-01-01 19:08:37 +00:00
self.argtypes = argtypes
testcrypt.py: use parameter names in diagnostics. Making a virtue of the necessity of adding parameter names to testcrypt.h a couple of commits ago, we can now use those names to improve diagnostics, so that if you use the wrong type in a Python function call the error message will tell you the name as well as the index of the offending argument. Also, the repr() text for the function itself will now print a full prototype (albeit in a nasty hybrid of C, Python and testcrypt.h syntax) which shows all the parameter names. That should be handy when trying to remember the order of arguments at the REPL prompt.
2021-11-21 13:21:01 +00:00
self.argnames = argnames
New test system for mp_int and cryptography. I've written a new standalone test program which incorporates all of PuTTY's crypto code, including the mp_int and low-level elliptic curve layers but also going all the way up to the implementations of the MAC, hash, cipher, public key and kex abstractions. The test program itself, 'testcrypt', speaks a simple line-oriented protocol on standard I/O in which you write the name of a function call followed by some inputs, and it gives you back a list of outputs preceded by a line telling you how many there are. Dynamically allocated objects are assigned string ids in the protocol, and there's a 'free' function that tells testcrypt when it can dispose of one. It's possible to speak that protocol by hand, but cumbersome. I've also provided a Python module that wraps it, by running testcrypt as a persistent subprocess and gatewaying all the function calls into things that look reasonably natural to call from Python. The Python module and testcrypt.c both read a carefully formatted header file testcrypt.h which contains the name and signature of every exported function, so it costs minimal effort to expose a given function through this test API. In a few cases it's necessary to write a wrapper in testcrypt.c that makes the function look more friendly, but mostly you don't even need that. (Though that is one of the motivations between a lot of API cleanups I've done recently!) I considered doing Python integration in the more obvious way, by linking parts of the PuTTY code directly into a native-code .so Python module. I decided against it because this way is more flexible: I can run the testcrypt program on its own, or compile it in a way that Python wouldn't play nicely with (I bet compiling just that .so with Leak Sanitiser wouldn't do what you wanted when Python loaded it!), or attach a debugger to it. I can even recompile testcrypt for a different CPU architecture (32- vs 64-bit, or even running it on a different machine over ssh or under emulation) and still layer the nice API on top of that via the local Python interpreter. All I need is a bidirectional data channel.
2019-01-01 19:08:37 +00:00
def __repr__(self):
testcrypt.py: use parameter names in diagnostics. Making a virtue of the necessity of adding parameter names to testcrypt.h a couple of commits ago, we can now use those names to improve diagnostics, so that if you use the wrong type in a Python function call the error message will tell you the name as well as the index of the offending argument. Also, the repr() text for the function itself will now print a full prototype (albeit in a nasty hybrid of C, Python and testcrypt.h syntax) which shows all the parameter names. That should be handy when trying to remember the order of arguments at the REPL prompt.
2021-11-21 13:21:01 +00:00
return "<Function {}({}) -> ({})>".format(
self.fnname,
", ".join(("consumed " if c else "")+t+" "+n
for (t,c),n in zip(self.argtypes, self.argnames)),
", ".join((t+" "+n if n is not None else t)
for t,n in zip(self.rettypes, self.retnames)),
)
New test system for mp_int and cryptography. I've written a new standalone test program which incorporates all of PuTTY's crypto code, including the mp_int and low-level elliptic curve layers but also going all the way up to the implementations of the MAC, hash, cipher, public key and kex abstractions. The test program itself, 'testcrypt', speaks a simple line-oriented protocol on standard I/O in which you write the name of a function call followed by some inputs, and it gives you back a list of outputs preceded by a line telling you how many there are. Dynamically allocated objects are assigned string ids in the protocol, and there's a 'free' function that tells testcrypt when it can dispose of one. It's possible to speak that protocol by hand, but cumbersome. I've also provided a Python module that wraps it, by running testcrypt as a persistent subprocess and gatewaying all the function calls into things that look reasonably natural to call from Python. The Python module and testcrypt.c both read a carefully formatted header file testcrypt.h which contains the name and signature of every exported function, so it costs minimal effort to expose a given function through this test API. In a few cases it's necessary to write a wrapper in testcrypt.c that makes the function look more friendly, but mostly you don't even need that. (Though that is one of the motivations between a lot of API cleanups I've done recently!) I considered doing Python integration in the more obvious way, by linking parts of the PuTTY code directly into a native-code .so Python module. I decided against it because this way is more flexible: I can run the testcrypt program on its own, or compile it in a way that Python wouldn't play nicely with (I bet compiling just that .so with Leak Sanitiser wouldn't do what you wanted when Python loaded it!), or attach a debugger to it. I can even recompile testcrypt for a different CPU architecture (32- vs 64-bit, or even running it on a different machine over ssh or under emulation) and still layer the nice API on top of that via the local Python interpreter. All I need is a bidirectional data channel.
2019-01-01 19:08:37 +00:00
def __call__(self, *args):
if len(args) != len(self.argtypes):
raise TypeError(
"{}() takes exactly {} arguments ({} given)".format(
self.fnname, len(self.argtypes), len(args)))
to_preserve = []
retwords = childprocess.funcall(
self.fnname, [make_argword(args[i], self.argtypes[i],
testcrypt.py: use parameter names in diagnostics. Making a virtue of the necessity of adding parameter names to testcrypt.h a couple of commits ago, we can now use those names to improve diagnostics, so that if you use the wrong type in a Python function call the error message will tell you the name as well as the index of the offending argument. Also, the repr() text for the function itself will now print a full prototype (albeit in a nasty hybrid of C, Python and testcrypt.h syntax) which shows all the parameter names. That should be handy when trying to remember the order of arguments at the REPL prompt.
2021-11-21 13:21:01 +00:00
self.fnname, i, self.argnames[i],
to_preserve)
New test system for mp_int and cryptography. I've written a new standalone test program which incorporates all of PuTTY's crypto code, including the mp_int and low-level elliptic curve layers but also going all the way up to the implementations of the MAC, hash, cipher, public key and kex abstractions. The test program itself, 'testcrypt', speaks a simple line-oriented protocol on standard I/O in which you write the name of a function call followed by some inputs, and it gives you back a list of outputs preceded by a line telling you how many there are. Dynamically allocated objects are assigned string ids in the protocol, and there's a 'free' function that tells testcrypt when it can dispose of one. It's possible to speak that protocol by hand, but cumbersome. I've also provided a Python module that wraps it, by running testcrypt as a persistent subprocess and gatewaying all the function calls into things that look reasonably natural to call from Python. The Python module and testcrypt.c both read a carefully formatted header file testcrypt.h which contains the name and signature of every exported function, so it costs minimal effort to expose a given function through this test API. In a few cases it's necessary to write a wrapper in testcrypt.c that makes the function look more friendly, but mostly you don't even need that. (Though that is one of the motivations between a lot of API cleanups I've done recently!) I considered doing Python integration in the more obvious way, by linking parts of the PuTTY code directly into a native-code .so Python module. I decided against it because this way is more flexible: I can run the testcrypt program on its own, or compile it in a way that Python wouldn't play nicely with (I bet compiling just that .so with Leak Sanitiser wouldn't do what you wanted when Python loaded it!), or attach a debugger to it. I can even recompile testcrypt for a different CPU architecture (32- vs 64-bit, or even running it on a different machine over ssh or under emulation) and still layer the nice API on top of that via the local Python interpreter. All I need is a bidirectional data channel.
2019-01-01 19:08:37 +00:00
for i in range(len(args))])
retvals = make_retvals(self.rettypes, retwords)
if len(retvals) == 0:
return None
if len(retvals) == 1:
return retvals[0]
return tuple(retvals)
Rewrite the testcrypt.c macro system. Yesterday's commit 52ee636b092c199 which further extended the huge pile of arity-specific annoying wrapper macros pushed me over the edge and inspired me to give some harder thought to finding a way to handle all arities at once. And this time I found one! The new technique changes the syntax of the function specifications in testcrypt.h. In particular, they now have to specify a _name_ for each parameter as well as a type, because the macros generating the C marshalling wrappers will need a structure field for each parameter and cpp isn't flexible enough to generate names for those fields automatically. Rather than tediously name them arg1, arg2 etc, I've reused the names of the parameters from the prototypes or definitions of the underlying real functions (via a one-off auto-extraction process starting from the output of 'clang -Xclang -dump-ast' plus some manual polishing), which means testcrypt.h is now a bit more self-documenting. The testcrypt.py end of the mechanism is rewritten to eat the new format. Since it's got more complicated syntax and nested parens and things, I've written something a bit like a separated lexer/parser system in place of the previous crude regex matcher, which should enforce that the whole header file really does conform to the restricted syntax it has to fit into. The new system uses a lot less code in testcrypt.c, but I've made up for that by also writing a long comment explaining how it works, which was another thing the previous system lacked! Similarly, the new testcrypt.h has some long-overdue instructions at the top.
2021-11-21 10:27:30 +00:00
def _lex_testcrypt_header(header):
pat = re.compile(
# Skip any combination of whitespace and comments
'(?:{})*'.format('|'.join((
'[ \t\n]', # whitespace
'/\\*(?:.|\n)*?\\*/', # C90-style /* ... */ comment, ended eagerly
'//[^\n]*\n', # C99-style comment to end-of-line
))) +
# And then match a token
'({})'.format('|'.join((
# Punctuation
Suppress syntax warnings on Python 3.12. Python 3.12 has a new warning for backslash-character pairs that are not valid escape sequences at the level of string literals, as opposed to in some interior syntax such as regular expressions (https://docs.python.org/3/whatsnew/3.12.html#other-language-changes). Suppress it by using raw strings.
2024-08-01 14:18:57 +00:00
r'\(',
r'\)',
Rewrite the testcrypt.c macro system. Yesterday's commit 52ee636b092c199 which further extended the huge pile of arity-specific annoying wrapper macros pushed me over the edge and inspired me to give some harder thought to finding a way to handle all arities at once. And this time I found one! The new technique changes the syntax of the function specifications in testcrypt.h. In particular, they now have to specify a _name_ for each parameter as well as a type, because the macros generating the C marshalling wrappers will need a structure field for each parameter and cpp isn't flexible enough to generate names for those fields automatically. Rather than tediously name them arg1, arg2 etc, I've reused the names of the parameters from the prototypes or definitions of the underlying real functions (via a one-off auto-extraction process starting from the output of 'clang -Xclang -dump-ast' plus some manual polishing), which means testcrypt.h is now a bit more self-documenting. The testcrypt.py end of the mechanism is rewritten to eat the new format. Since it's got more complicated syntax and nested parens and things, I've written something a bit like a separated lexer/parser system in place of the previous crude regex matcher, which should enforce that the whole header file really does conform to the restricted syntax it has to fit into. The new system uses a lot less code in testcrypt.c, but I've made up for that by also writing a long comment explaining how it works, which was another thing the previous system lacked! Similarly, the new testcrypt.h has some long-overdue instructions at the top.
2021-11-21 10:27:30 +00:00
',',
# Identifier
'[A-Za-z_][A-Za-z0-9_]*',
# End of string
'$',
)))
)
pos = 0
end = len(header)
while pos < end:
m = pat.match(header, pos)
assert m is not None, (
Move some tests into the test subdirectory. Now testcrypt has _two_ header files, that's more files than I want at the top level, so I decided to move it. It has a good claim to live in either 'test' or 'crypto', but in the end I decided it wasn't quite specific enough to crypto (it already also tests things in keygen and proxy), and also, the Python half of the mechanism already lives in 'test', so it can live alongside that. Having done that, it seemed silly to leave testsc and testzlib at the top level: those have 'test' in the names as well, so they can go in the test subdir as well. While I'm renaming, also renamed testcrypt.h to testcrypt-func.h to distinguish it from the new testcrypt-enum.h.
2021-11-22 18:50:12 +00:00
"Failed to lex testcrypt-func.h at byte position {:d}".format(pos))
Rewrite the testcrypt.c macro system. Yesterday's commit 52ee636b092c199 which further extended the huge pile of arity-specific annoying wrapper macros pushed me over the edge and inspired me to give some harder thought to finding a way to handle all arities at once. And this time I found one! The new technique changes the syntax of the function specifications in testcrypt.h. In particular, they now have to specify a _name_ for each parameter as well as a type, because the macros generating the C marshalling wrappers will need a structure field for each parameter and cpp isn't flexible enough to generate names for those fields automatically. Rather than tediously name them arg1, arg2 etc, I've reused the names of the parameters from the prototypes or definitions of the underlying real functions (via a one-off auto-extraction process starting from the output of 'clang -Xclang -dump-ast' plus some manual polishing), which means testcrypt.h is now a bit more self-documenting. The testcrypt.py end of the mechanism is rewritten to eat the new format. Since it's got more complicated syntax and nested parens and things, I've written something a bit like a separated lexer/parser system in place of the previous crude regex matcher, which should enforce that the whole header file really does conform to the restricted syntax it has to fit into. The new system uses a lot less code in testcrypt.c, but I've made up for that by also writing a long comment explaining how it works, which was another thing the previous system lacked! Similarly, the new testcrypt.h has some long-overdue instructions at the top.
2021-11-21 10:27:30 +00:00
pos = m.end()
tok = m.group(1)
if len(tok) == 0:
assert pos == end, (
"Empty token should only be returned at end of string")
yield tok, m.start(1)
def _parse_testcrypt_header(tokens):
def is_id(tok):
return tok[0] in string.ascii_letters+"_"
def expect(what, why, eof_ok=False):
tok, pos = next(tokens)
if tok == '' and eof_ok:
return None
if hasattr(what, '__call__'):
description = lambda: ""
ok = what(tok)
elif isinstance(what, set):
description = lambda: " or ".join("'"+x+"' " for x in sorted(what))
ok = tok in what
else:
description = lambda: "'"+what+"' "
ok = tok == what
if not ok:
Move some tests into the test subdirectory. Now testcrypt has _two_ header files, that's more files than I want at the top level, so I decided to move it. It has a good claim to live in either 'test' or 'crypto', but in the end I decided it wasn't quite specific enough to crypto (it already also tests things in keygen and proxy), and also, the Python half of the mechanism already lives in 'test', so it can live alongside that. Having done that, it seemed silly to leave testsc and testzlib at the top level: those have 'test' in the names as well, so they can go in the test subdir as well. While I'm renaming, also renamed testcrypt.h to testcrypt-func.h to distinguish it from the new testcrypt-enum.h.
2021-11-22 18:50:12 +00:00
sys.exit("testcrypt-func.h:{:d}: expected {}{}".format(
Rewrite the testcrypt.c macro system. Yesterday's commit 52ee636b092c199 which further extended the huge pile of arity-specific annoying wrapper macros pushed me over the edge and inspired me to give some harder thought to finding a way to handle all arities at once. And this time I found one! The new technique changes the syntax of the function specifications in testcrypt.h. In particular, they now have to specify a _name_ for each parameter as well as a type, because the macros generating the C marshalling wrappers will need a structure field for each parameter and cpp isn't flexible enough to generate names for those fields automatically. Rather than tediously name them arg1, arg2 etc, I've reused the names of the parameters from the prototypes or definitions of the underlying real functions (via a one-off auto-extraction process starting from the output of 'clang -Xclang -dump-ast' plus some manual polishing), which means testcrypt.h is now a bit more self-documenting. The testcrypt.py end of the mechanism is rewritten to eat the new format. Since it's got more complicated syntax and nested parens and things, I've written something a bit like a separated lexer/parser system in place of the previous crude regex matcher, which should enforce that the whole header file really does conform to the restricted syntax it has to fit into. The new system uses a lot less code in testcrypt.c, but I've made up for that by also writing a long comment explaining how it works, which was another thing the previous system lacked! Similarly, the new testcrypt.h has some long-overdue instructions at the top.
2021-11-21 10:27:30 +00:00
pos, description(), why))
return tok
New test system for mp_int and cryptography. I've written a new standalone test program which incorporates all of PuTTY's crypto code, including the mp_int and low-level elliptic curve layers but also going all the way up to the implementations of the MAC, hash, cipher, public key and kex abstractions. The test program itself, 'testcrypt', speaks a simple line-oriented protocol on standard I/O in which you write the name of a function call followed by some inputs, and it gives you back a list of outputs preceded by a line telling you how many there are. Dynamically allocated objects are assigned string ids in the protocol, and there's a 'free' function that tells testcrypt when it can dispose of one. It's possible to speak that protocol by hand, but cumbersome. I've also provided a Python module that wraps it, by running testcrypt as a persistent subprocess and gatewaying all the function calls into things that look reasonably natural to call from Python. The Python module and testcrypt.c both read a carefully formatted header file testcrypt.h which contains the name and signature of every exported function, so it costs minimal effort to expose a given function through this test API. In a few cases it's necessary to write a wrapper in testcrypt.c that makes the function look more friendly, but mostly you don't even need that. (Though that is one of the motivations between a lot of API cleanups I've done recently!) I considered doing Python integration in the more obvious way, by linking parts of the PuTTY code directly into a native-code .so Python module. I decided against it because this way is more flexible: I can run the testcrypt program on its own, or compile it in a way that Python wouldn't play nicely with (I bet compiling just that .so with Leak Sanitiser wouldn't do what you wanted when Python loaded it!), or attach a debugger to it. I can even recompile testcrypt for a different CPU architecture (32- vs 64-bit, or even running it on a different machine over ssh or under emulation) and still layer the nice API on top of that via the local Python interpreter. All I need is a bidirectional data channel.
2019-01-01 19:08:37 +00:00
Rewrite the testcrypt.c macro system. Yesterday's commit 52ee636b092c199 which further extended the huge pile of arity-specific annoying wrapper macros pushed me over the edge and inspired me to give some harder thought to finding a way to handle all arities at once. And this time I found one! The new technique changes the syntax of the function specifications in testcrypt.h. In particular, they now have to specify a _name_ for each parameter as well as a type, because the macros generating the C marshalling wrappers will need a structure field for each parameter and cpp isn't flexible enough to generate names for those fields automatically. Rather than tediously name them arg1, arg2 etc, I've reused the names of the parameters from the prototypes or definitions of the underlying real functions (via a one-off auto-extraction process starting from the output of 'clang -Xclang -dump-ast' plus some manual polishing), which means testcrypt.h is now a bit more self-documenting. The testcrypt.py end of the mechanism is rewritten to eat the new format. Since it's got more complicated syntax and nested parens and things, I've written something a bit like a separated lexer/parser system in place of the previous crude regex matcher, which should enforce that the whole header file really does conform to the restricted syntax it has to fit into. The new system uses a lot less code in testcrypt.c, but I've made up for that by also writing a long comment explaining how it works, which was another thing the previous system lacked! Similarly, the new testcrypt.h has some long-overdue instructions at the top.
2021-11-21 10:27:30 +00:00
while True:
testcrypt.h: invent FUNC_WRAPPED. FUNC_WRAPPED is an alternative keyword to FUNC which you can use to introduce a function specification in testcrypt.h, indicating that the function is _not_ the one of the same name used in the main PuTTY code, but instead a wrapper on it in testcrypt.c whose API was reworked to be more friendly to translation into Python. There are a lot of those wrappers already, and previously they passed without comment in testcrypt.h, and were put into service by #defining over the top of each name before expanding the marshalling functions. Now, all those #defines are gone, because the use of FUNC_WRAPPED in testcrypt.h is enough to clue in the marshalling wrapper to be generated with a call to foo_wrapper() instead of foo(). Mostly the purpose of this is to make testcrypt.h a bit more self-documenting: if you see FUNC_WRAPPED, you know not to be confused by the Python and C function definitions totally failing to match.
2021-11-21 13:03:34 +00:00
tok = expect({"FUNC", "FUNC_WRAPPED"},
"at start of function specification", eof_ok=True)
Rewrite the testcrypt.c macro system. Yesterday's commit 52ee636b092c199 which further extended the huge pile of arity-specific annoying wrapper macros pushed me over the edge and inspired me to give some harder thought to finding a way to handle all arities at once. And this time I found one! The new technique changes the syntax of the function specifications in testcrypt.h. In particular, they now have to specify a _name_ for each parameter as well as a type, because the macros generating the C marshalling wrappers will need a structure field for each parameter and cpp isn't flexible enough to generate names for those fields automatically. Rather than tediously name them arg1, arg2 etc, I've reused the names of the parameters from the prototypes or definitions of the underlying real functions (via a one-off auto-extraction process starting from the output of 'clang -Xclang -dump-ast' plus some manual polishing), which means testcrypt.h is now a bit more self-documenting. The testcrypt.py end of the mechanism is rewritten to eat the new format. Since it's got more complicated syntax and nested parens and things, I've written something a bit like a separated lexer/parser system in place of the previous crude regex matcher, which should enforce that the whole header file really does conform to the restricted syntax it has to fit into. The new system uses a lot less code in testcrypt.c, but I've made up for that by also writing a long comment explaining how it works, which was another thing the previous system lacked! Similarly, the new testcrypt.h has some long-overdue instructions at the top.
2021-11-21 10:27:30 +00:00
if tok is None:
break
expect("(", "after FUNC")
rettype = expect(is_id, "return type")
expect(",", "after return type")
funcname = expect(is_id, "function name")
expect(",", "after function name")
args = []
firstargkind = expect({"ARG", "VOID"}, "at start of argument list")
if firstargkind == "VOID":
expect(")", "after VOID")
else:
while True:
# Every time we come back to the top of this loop, we've
# just seen 'ARG'
expect("(", "after ARG")
argtype = expect(is_id, "argument type")
expect(",", "after argument type")
argname = expect(is_id, "argument name")
args.append((argtype, argname))
expect(")", "at end of ARG")
punct = expect({",", ")"}, "after argument")
if punct == ")":
break
expect("ARG", "to begin next argument")
yield funcname, rettype, args
def _setup(scope):
New test system for mp_int and cryptography. I've written a new standalone test program which incorporates all of PuTTY's crypto code, including the mp_int and low-level elliptic curve layers but also going all the way up to the implementations of the MAC, hash, cipher, public key and kex abstractions. The test program itself, 'testcrypt', speaks a simple line-oriented protocol on standard I/O in which you write the name of a function call followed by some inputs, and it gives you back a list of outputs preceded by a line telling you how many there are. Dynamically allocated objects are assigned string ids in the protocol, and there's a 'free' function that tells testcrypt when it can dispose of one. It's possible to speak that protocol by hand, but cumbersome. I've also provided a Python module that wraps it, by running testcrypt as a persistent subprocess and gatewaying all the function calls into things that look reasonably natural to call from Python. The Python module and testcrypt.c both read a carefully formatted header file testcrypt.h which contains the name and signature of every exported function, so it costs minimal effort to expose a given function through this test API. In a few cases it's necessary to write a wrapper in testcrypt.c that makes the function look more friendly, but mostly you don't even need that. (Though that is one of the motivations between a lot of API cleanups I've done recently!) I considered doing Python integration in the more obvious way, by linking parts of the PuTTY code directly into a native-code .so Python module. I decided against it because this way is more flexible: I can run the testcrypt program on its own, or compile it in a way that Python wouldn't play nicely with (I bet compiling just that .so with Leak Sanitiser wouldn't do what you wanted when Python loaded it!), or attach a debugger to it. I can even recompile testcrypt for a different CPU architecture (32- vs 64-bit, or even running it on a different machine over ssh or under emulation) and still layer the nice API on top of that via the local Python interpreter. All I need is a bidirectional data channel.
2019-01-01 19:08:37 +00:00
valprefix = "val_"
outprefix = "out_"
testcrypt.py: look past 'opt_' prefix on argument types. When testcrypt.h lists a function argument as 'opt_val_foo', it means that the argument is optional in the sense that the C function can take a null pointer in place of a valid foo, and so the Python wrapper module should accept None in the corresponding argument slot from the client code and translate it into the special string "NULL" in the wire protocol. This works fine at argument translation time, but the code that reads testcrypt.h wasn't looking at it, so if you said 'opt_val_foo_suffix' in place of 'opt_val_foo' (indicating that that argument is optional _and_ the C function expects it in a translated form), then the initial pass over testcrypt.h wouldn't strip the _suffix, and would set up data structures with mismatched type names.
2019-01-10 19:21:33 +00:00
optprefix = "opt_"
New test system for mp_int and cryptography. I've written a new standalone test program which incorporates all of PuTTY's crypto code, including the mp_int and low-level elliptic curve layers but also going all the way up to the implementations of the MAC, hash, cipher, public key and kex abstractions. The test program itself, 'testcrypt', speaks a simple line-oriented protocol on standard I/O in which you write the name of a function call followed by some inputs, and it gives you back a list of outputs preceded by a line telling you how many there are. Dynamically allocated objects are assigned string ids in the protocol, and there's a 'free' function that tells testcrypt when it can dispose of one. It's possible to speak that protocol by hand, but cumbersome. I've also provided a Python module that wraps it, by running testcrypt as a persistent subprocess and gatewaying all the function calls into things that look reasonably natural to call from Python. The Python module and testcrypt.c both read a carefully formatted header file testcrypt.h which contains the name and signature of every exported function, so it costs minimal effort to expose a given function through this test API. In a few cases it's necessary to write a wrapper in testcrypt.c that makes the function look more friendly, but mostly you don't even need that. (Though that is one of the motivations between a lot of API cleanups I've done recently!) I considered doing Python integration in the more obvious way, by linking parts of the PuTTY code directly into a native-code .so Python module. I decided against it because this way is more flexible: I can run the testcrypt program on its own, or compile it in a way that Python wouldn't play nicely with (I bet compiling just that .so with Leak Sanitiser wouldn't do what you wanted when Python loaded it!), or attach a debugger to it. I can even recompile testcrypt for a different CPU architecture (32- vs 64-bit, or even running it on a different machine over ssh or under emulation) and still layer the nice API on top of that via the local Python interpreter. All I need is a bidirectional data channel.
2019-01-01 19:08:37 +00:00
consprefix = "consumed_"
def trim_argtype(arg):
testcrypt.py: look past 'opt_' prefix on argument types. When testcrypt.h lists a function argument as 'opt_val_foo', it means that the argument is optional in the sense that the C function can take a null pointer in place of a valid foo, and so the Python wrapper module should accept None in the corresponding argument slot from the client code and translate it into the special string "NULL" in the wire protocol. This works fine at argument translation time, but the code that reads testcrypt.h wasn't looking at it, so if you said 'opt_val_foo_suffix' in place of 'opt_val_foo' (indicating that that argument is optional _and_ the C function expects it in a translated form), then the initial pass over testcrypt.h wouldn't strip the _suffix, and would set up data structures with mismatched type names.
2019-01-10 19:21:33 +00:00
if arg.startswith(optprefix):
return optprefix + trim_argtype(arg[len(optprefix):])
New test system for mp_int and cryptography. I've written a new standalone test program which incorporates all of PuTTY's crypto code, including the mp_int and low-level elliptic curve layers but also going all the way up to the implementations of the MAC, hash, cipher, public key and kex abstractions. The test program itself, 'testcrypt', speaks a simple line-oriented protocol on standard I/O in which you write the name of a function call followed by some inputs, and it gives you back a list of outputs preceded by a line telling you how many there are. Dynamically allocated objects are assigned string ids in the protocol, and there's a 'free' function that tells testcrypt when it can dispose of one. It's possible to speak that protocol by hand, but cumbersome. I've also provided a Python module that wraps it, by running testcrypt as a persistent subprocess and gatewaying all the function calls into things that look reasonably natural to call from Python. The Python module and testcrypt.c both read a carefully formatted header file testcrypt.h which contains the name and signature of every exported function, so it costs minimal effort to expose a given function through this test API. In a few cases it's necessary to write a wrapper in testcrypt.c that makes the function look more friendly, but mostly you don't even need that. (Though that is one of the motivations between a lot of API cleanups I've done recently!) I considered doing Python integration in the more obvious way, by linking parts of the PuTTY code directly into a native-code .so Python module. I decided against it because this way is more flexible: I can run the testcrypt program on its own, or compile it in a way that Python wouldn't play nicely with (I bet compiling just that .so with Leak Sanitiser wouldn't do what you wanted when Python loaded it!), or attach a debugger to it. I can even recompile testcrypt for a different CPU architecture (32- vs 64-bit, or even running it on a different machine over ssh or under emulation) and still layer the nice API on top of that via the local Python interpreter. All I need is a bidirectional data channel.
2019-01-01 19:08:37 +00:00
if (arg.startswith(valprefix) and
"_" in arg[len(valprefix):]):
# Strip suffixes like val_string_asciz
arg = arg[:arg.index("_", len(valprefix))]
return arg
Move some tests into the test subdirectory. Now testcrypt has _two_ header files, that's more files than I want at the top level, so I decided to move it. It has a good claim to live in either 'test' or 'crypto', but in the end I decided it wasn't quite specific enough to crypto (it already also tests things in keygen and proxy), and also, the Python half of the mechanism already lives in 'test', so it can live alongside that. Having done that, it seemed silly to leave testsc and testzlib at the top level: those have 'test' in the names as well, so they can go in the test subdir as well. While I'm renaming, also renamed testcrypt.h to testcrypt-func.h to distinguish it from the new testcrypt-enum.h.
2021-11-22 18:50:12 +00:00
with open(os.path.join(putty_srcdir, "test", "testcrypt-func.h")) as f:
Rewrite the testcrypt.c macro system. Yesterday's commit 52ee636b092c199 which further extended the huge pile of arity-specific annoying wrapper macros pushed me over the edge and inspired me to give some harder thought to finding a way to handle all arities at once. And this time I found one! The new technique changes the syntax of the function specifications in testcrypt.h. In particular, they now have to specify a _name_ for each parameter as well as a type, because the macros generating the C marshalling wrappers will need a structure field for each parameter and cpp isn't flexible enough to generate names for those fields automatically. Rather than tediously name them arg1, arg2 etc, I've reused the names of the parameters from the prototypes or definitions of the underlying real functions (via a one-off auto-extraction process starting from the output of 'clang -Xclang -dump-ast' plus some manual polishing), which means testcrypt.h is now a bit more self-documenting. The testcrypt.py end of the mechanism is rewritten to eat the new format. Since it's got more complicated syntax and nested parens and things, I've written something a bit like a separated lexer/parser system in place of the previous crude regex matcher, which should enforce that the whole header file really does conform to the restricted syntax it has to fit into. The new system uses a lot less code in testcrypt.c, but I've made up for that by also writing a long comment explaining how it works, which was another thing the previous system lacked! Similarly, the new testcrypt.h has some long-overdue instructions at the top.
2021-11-21 10:27:30 +00:00
header = f.read()
tokens = _lex_testcrypt_header(header)
for function, rettype, arglist in _parse_testcrypt_header(tokens):
rettypes = []
testcrypt.py: use parameter names in diagnostics. Making a virtue of the necessity of adding parameter names to testcrypt.h a couple of commits ago, we can now use those names to improve diagnostics, so that if you use the wrong type in a Python function call the error message will tell you the name as well as the index of the offending argument. Also, the repr() text for the function itself will now print a full prototype (albeit in a nasty hybrid of C, Python and testcrypt.h syntax) which shows all the parameter names. That should be handy when trying to remember the order of arguments at the REPL prompt.
2021-11-21 13:21:01 +00:00
retnames = []
Rewrite the testcrypt.c macro system. Yesterday's commit 52ee636b092c199 which further extended the huge pile of arity-specific annoying wrapper macros pushed me over the edge and inspired me to give some harder thought to finding a way to handle all arities at once. And this time I found one! The new technique changes the syntax of the function specifications in testcrypt.h. In particular, they now have to specify a _name_ for each parameter as well as a type, because the macros generating the C marshalling wrappers will need a structure field for each parameter and cpp isn't flexible enough to generate names for those fields automatically. Rather than tediously name them arg1, arg2 etc, I've reused the names of the parameters from the prototypes or definitions of the underlying real functions (via a one-off auto-extraction process starting from the output of 'clang -Xclang -dump-ast' plus some manual polishing), which means testcrypt.h is now a bit more self-documenting. The testcrypt.py end of the mechanism is rewritten to eat the new format. Since it's got more complicated syntax and nested parens and things, I've written something a bit like a separated lexer/parser system in place of the previous crude regex matcher, which should enforce that the whole header file really does conform to the restricted syntax it has to fit into. The new system uses a lot less code in testcrypt.c, but I've made up for that by also writing a long comment explaining how it works, which was another thing the previous system lacked! Similarly, the new testcrypt.h has some long-overdue instructions at the top.
2021-11-21 10:27:30 +00:00
if rettype != "void":
rettypes.append(trim_argtype(rettype))
testcrypt.py: use parameter names in diagnostics. Making a virtue of the necessity of adding parameter names to testcrypt.h a couple of commits ago, we can now use those names to improve diagnostics, so that if you use the wrong type in a Python function call the error message will tell you the name as well as the index of the offending argument. Also, the repr() text for the function itself will now print a full prototype (albeit in a nasty hybrid of C, Python and testcrypt.h syntax) which shows all the parameter names. That should be handy when trying to remember the order of arguments at the REPL prompt.
2021-11-21 13:21:01 +00:00
retnames.append(None)
Rewrite the testcrypt.c macro system. Yesterday's commit 52ee636b092c199 which further extended the huge pile of arity-specific annoying wrapper macros pushed me over the edge and inspired me to give some harder thought to finding a way to handle all arities at once. And this time I found one! The new technique changes the syntax of the function specifications in testcrypt.h. In particular, they now have to specify a _name_ for each parameter as well as a type, because the macros generating the C marshalling wrappers will need a structure field for each parameter and cpp isn't flexible enough to generate names for those fields automatically. Rather than tediously name them arg1, arg2 etc, I've reused the names of the parameters from the prototypes or definitions of the underlying real functions (via a one-off auto-extraction process starting from the output of 'clang -Xclang -dump-ast' plus some manual polishing), which means testcrypt.h is now a bit more self-documenting. The testcrypt.py end of the mechanism is rewritten to eat the new format. Since it's got more complicated syntax and nested parens and things, I've written something a bit like a separated lexer/parser system in place of the previous crude regex matcher, which should enforce that the whole header file really does conform to the restricted syntax it has to fit into. The new system uses a lot less code in testcrypt.c, but I've made up for that by also writing a long comment explaining how it works, which was another thing the previous system lacked! Similarly, the new testcrypt.h has some long-overdue instructions at the top.
2021-11-21 10:27:30 +00:00
argtypes = []
testcrypt.py: use parameter names in diagnostics. Making a virtue of the necessity of adding parameter names to testcrypt.h a couple of commits ago, we can now use those names to improve diagnostics, so that if you use the wrong type in a Python function call the error message will tell you the name as well as the index of the offending argument. Also, the repr() text for the function itself will now print a full prototype (albeit in a nasty hybrid of C, Python and testcrypt.h syntax) which shows all the parameter names. That should be handy when trying to remember the order of arguments at the REPL prompt.
2021-11-21 13:21:01 +00:00
argnames = []
Rewrite the testcrypt.c macro system. Yesterday's commit 52ee636b092c199 which further extended the huge pile of arity-specific annoying wrapper macros pushed me over the edge and inspired me to give some harder thought to finding a way to handle all arities at once. And this time I found one! The new technique changes the syntax of the function specifications in testcrypt.h. In particular, they now have to specify a _name_ for each parameter as well as a type, because the macros generating the C marshalling wrappers will need a structure field for each parameter and cpp isn't flexible enough to generate names for those fields automatically. Rather than tediously name them arg1, arg2 etc, I've reused the names of the parameters from the prototypes or definitions of the underlying real functions (via a one-off auto-extraction process starting from the output of 'clang -Xclang -dump-ast' plus some manual polishing), which means testcrypt.h is now a bit more self-documenting. The testcrypt.py end of the mechanism is rewritten to eat the new format. Since it's got more complicated syntax and nested parens and things, I've written something a bit like a separated lexer/parser system in place of the previous crude regex matcher, which should enforce that the whole header file really does conform to the restricted syntax it has to fit into. The new system uses a lot less code in testcrypt.c, but I've made up for that by also writing a long comment explaining how it works, which was another thing the previous system lacked! Similarly, the new testcrypt.h has some long-overdue instructions at the top.
2021-11-21 10:27:30 +00:00
argsconsumed = []
for arg, argname in arglist:
if arg.startswith(outprefix):
rettypes.append(trim_argtype(arg[len(outprefix):]))
testcrypt.py: use parameter names in diagnostics. Making a virtue of the necessity of adding parameter names to testcrypt.h a couple of commits ago, we can now use those names to improve diagnostics, so that if you use the wrong type in a Python function call the error message will tell you the name as well as the index of the offending argument. Also, the repr() text for the function itself will now print a full prototype (albeit in a nasty hybrid of C, Python and testcrypt.h syntax) which shows all the parameter names. That should be handy when trying to remember the order of arguments at the REPL prompt.
2021-11-21 13:21:01 +00:00
retnames.append(argname)
Rewrite the testcrypt.c macro system. Yesterday's commit 52ee636b092c199 which further extended the huge pile of arity-specific annoying wrapper macros pushed me over the edge and inspired me to give some harder thought to finding a way to handle all arities at once. And this time I found one! The new technique changes the syntax of the function specifications in testcrypt.h. In particular, they now have to specify a _name_ for each parameter as well as a type, because the macros generating the C marshalling wrappers will need a structure field for each parameter and cpp isn't flexible enough to generate names for those fields automatically. Rather than tediously name them arg1, arg2 etc, I've reused the names of the parameters from the prototypes or definitions of the underlying real functions (via a one-off auto-extraction process starting from the output of 'clang -Xclang -dump-ast' plus some manual polishing), which means testcrypt.h is now a bit more self-documenting. The testcrypt.py end of the mechanism is rewritten to eat the new format. Since it's got more complicated syntax and nested parens and things, I've written something a bit like a separated lexer/parser system in place of the previous crude regex matcher, which should enforce that the whole header file really does conform to the restricted syntax it has to fit into. The new system uses a lot less code in testcrypt.c, but I've made up for that by also writing a long comment explaining how it works, which was another thing the previous system lacked! Similarly, the new testcrypt.h has some long-overdue instructions at the top.
2021-11-21 10:27:30 +00:00
else:
consumed = False
if arg.startswith(consprefix):
arg = arg[len(consprefix):]
consumed = True
arg = trim_argtype(arg)
argtypes.append((arg, consumed))
testcrypt.py: use parameter names in diagnostics. Making a virtue of the necessity of adding parameter names to testcrypt.h a couple of commits ago, we can now use those names to improve diagnostics, so that if you use the wrong type in a Python function call the error message will tell you the name as well as the index of the offending argument. Also, the repr() text for the function itself will now print a full prototype (albeit in a nasty hybrid of C, Python and testcrypt.h syntax) which shows all the parameter names. That should be handy when trying to remember the order of arguments at the REPL prompt.
2021-11-21 13:21:01 +00:00
argnames.append(argname)
func = Function(function, rettypes, retnames,
argtypes, argnames)
Rewrite the testcrypt.c macro system. Yesterday's commit 52ee636b092c199 which further extended the huge pile of arity-specific annoying wrapper macros pushed me over the edge and inspired me to give some harder thought to finding a way to handle all arities at once. And this time I found one! The new technique changes the syntax of the function specifications in testcrypt.h. In particular, they now have to specify a _name_ for each parameter as well as a type, because the macros generating the C marshalling wrappers will need a structure field for each parameter and cpp isn't flexible enough to generate names for those fields automatically. Rather than tediously name them arg1, arg2 etc, I've reused the names of the parameters from the prototypes or definitions of the underlying real functions (via a one-off auto-extraction process starting from the output of 'clang -Xclang -dump-ast' plus some manual polishing), which means testcrypt.h is now a bit more self-documenting. The testcrypt.py end of the mechanism is rewritten to eat the new format. Since it's got more complicated syntax and nested parens and things, I've written something a bit like a separated lexer/parser system in place of the previous crude regex matcher, which should enforce that the whole header file really does conform to the restricted syntax it has to fit into. The new system uses a lot less code in testcrypt.c, but I've made up for that by also writing a long comment explaining how it works, which was another thing the previous system lacked! Similarly, the new testcrypt.h has some long-overdue instructions at the top.
2021-11-21 10:27:30 +00:00
scope[function] = func
if len(argtypes) > 0:
t = argtypes[0][0]
testcrypt: permit multiple OO function prefixes for a type. This means if I have functions like foo_subfoo_bar and foo_baz that both operate on a foo, the Python testcrypt system can translate both into .bar() and .baz() methods on the object, even though they don't start with the same prefix.
2022-04-15 16:18:58 +00:00
if t in method_prefixes:
for prefix in method_prefixes[t]:
if function.startswith(prefix):
methodname = function[len(prefix):]
method_lists[t].append((methodname, func))
break
New test system for mp_int and cryptography. I've written a new standalone test program which incorporates all of PuTTY's crypto code, including the mp_int and low-level elliptic curve layers but also going all the way up to the implementations of the MAC, hash, cipher, public key and kex abstractions. The test program itself, 'testcrypt', speaks a simple line-oriented protocol on standard I/O in which you write the name of a function call followed by some inputs, and it gives you back a list of outputs preceded by a line telling you how many there are. Dynamically allocated objects are assigned string ids in the protocol, and there's a 'free' function that tells testcrypt when it can dispose of one. It's possible to speak that protocol by hand, but cumbersome. I've also provided a Python module that wraps it, by running testcrypt as a persistent subprocess and gatewaying all the function calls into things that look reasonably natural to call from Python. The Python module and testcrypt.c both read a carefully formatted header file testcrypt.h which contains the name and signature of every exported function, so it costs minimal effort to expose a given function through this test API. In a few cases it's necessary to write a wrapper in testcrypt.c that makes the function look more friendly, but mostly you don't even need that. (Though that is one of the motivations between a lot of API cleanups I've done recently!) I considered doing Python integration in the more obvious way, by linking parts of the PuTTY code directly into a native-code .so Python module. I decided against it because this way is more flexible: I can run the testcrypt program on its own, or compile it in a way that Python wouldn't play nicely with (I bet compiling just that .so with Leak Sanitiser wouldn't do what you wanted when Python loaded it!), or attach a debugger to it. I can even recompile testcrypt for a different CPU architecture (32- vs 64-bit, or even running it on a different machine over ssh or under emulation) and still layer the nice API on top of that via the local Python interpreter. All I need is a bidirectional data channel.
2019-01-01 19:08:37 +00:00
_setup(globals())
del _setup
Reference in New Issue Copy Permalink