If we build out of tree and then run `make clean`, we hit an infinite loop where libtommath tries to enter subdirs that don't exist and run `make clean`.