Stop requiring the RSA_FLAG_SIGN_VER

You can set custom sign and verify handlers on an RSA method (wihch is
used to create RSA private and public key handles). However, even if you
set them explicitly with RSA_meth_set_{sign,verify}(3), these handlers
aren't used for the sake of "backward compatibility" (with what?). In order
to use them, you need to opt your objects into using the custom methods
you set by setting the RSA_FLAG_SIGN_VER flag.

OpenSSL 1.1 dropped this requirement and therefore nobody sets this flag
anyore. Like most of the mechanically added accessors, almost nothing
uses them, but, as found by kn, the yubco-piv-tool does. This resulted
in a public key being passed to rsa_private_encrypt(), which of course
doesn't end well.

So follow OpenSSL 1.1 and drop this muppetry. This makes kn's problem
with yubico-piv-tool go away.

ok jsing kn

2 files changed, 5 insertions, 7 deletions

diff --git a/src/lib/libcrypto/rsa/rsa_local.h b/src/lib/libcrypto/rsa/rsa_local.h
index 31172093c4..3f88b952a2 100644
--- a/src/lib/libcrypto/rsa/rsa_local.h
+++ b/src/lib/libcrypto/rsa/rsa_local.h
@@ -1,4 +1,4 @@
-/* $OpenBSD: rsa_local.h,v 1.9 2024/11/29 07:42:35 tb Exp $ */
+/* $OpenBSD: rsa_local.h,v 1.10 2025/01/05 15:39:12 tb Exp $ */
 /* Copyright (C) 1995-1998 Eric Young (eay@cryptsoft.com)
  * All rights reserved.
  *
@@ -81,9 +81,7 @@ struct rsa_meth_st {
 /* New sign and verify functions: some libraries don't allow arbitrary data
  * to be signed/verified: this allows them to be used. Note: for this to work
  * the RSA_public_decrypt() and RSA_private_encrypt() should *NOT* be used
- * RSA_sign(), RSA_verify() should be used instead. Note: for backwards
- * compatibility this functionality is only enabled if the RSA_FLAG_SIGN_VER
- * option is set in 'flags'.
+ * RSA_sign(), RSA_verify() should be used instead.
  */
         int (*rsa_sign)(int type, const unsigned char *m, unsigned int m_length,
             unsigned char *sigret, unsigned int *siglen, const RSA *rsa);
diff --git a/src/lib/libcrypto/rsa/rsa_sign.c b/src/lib/libcrypto/rsa/rsa_sign.c
index 5356768615..6edd20626d 100644
--- a/src/lib/libcrypto/rsa/rsa_sign.c
+++ b/src/lib/libcrypto/rsa/rsa_sign.c
@@ -1,4 +1,4 @@
-/* $OpenBSD: rsa_sign.c,v 1.36 2023/07/08 12:26:45 beck Exp $ */
+/* $OpenBSD: rsa_sign.c,v 1.37 2025/01/05 15:39:12 tb Exp $ */
 /* Copyright (C) 1995-1998 Eric Young (eay@cryptsoft.com)
  * All rights reserved.
  *
@@ -130,7 +130,7 @@ RSA_sign(int type, const unsigned char *m, unsigned int m_len,
         unsigned char *tmps = NULL;
         int encrypt_len, encoded_len = 0, ret = 0;
 
-        if ((rsa->flags & RSA_FLAG_SIGN_VER) && rsa->meth->rsa_sign != NULL)
+        if (rsa->meth->rsa_sign != NULL)
                 return rsa->meth->rsa_sign(type, m, m_len, sigret, siglen, rsa);
 
         /* Compute the encoded digest. */
@@ -271,7 +271,7 @@ int
 RSA_verify(int dtype, const unsigned char *m, unsigned int m_len,
     const unsigned char *sigbuf, unsigned int siglen, RSA *rsa)
 {
-        if ((rsa->flags & RSA_FLAG_SIGN_VER) && rsa->meth->rsa_verify)
+        if (rsa->meth->rsa_verify != NULL)
                 return rsa->meth->rsa_verify(dtype, m, m_len, sigbuf, siglen,
                     rsa);