Fix a strange check in the auto DH codepath

The code assumes that the server certificate has an RSA key and bases
the calculation of the size of the ephemeral DH key on this assumption.
So instead of checking whether we have any key by inspecting the dh
part of the union, let's check that we actually have an RSA key.
While here, make sure that its length is non-negative.

ok jsing

1 files changed, 5 insertions, 3 deletions

diff --git a/src/lib/libssl/ssl_lib.c b/src/lib/libssl/ssl_lib.c
index b6882e7b12..662013378e 100644
--- a/src/lib/libssl/ssl_lib.c
+++ b/src/lib/libssl/ssl_lib.c
@@ -1,4 +1,4 @@
-/* $OpenBSD: ssl_lib.c,v 1.278 2021/11/08 18:19:22 bcook Exp $ */
+/* $OpenBSD: ssl_lib.c,v 1.279 2021/11/14 22:31:29 tb Exp $ */
 /* Copyright (C) 1995-1998 Eric Young (eay@cryptsoft.com)
  * All rights reserved.
  *
@@ -2335,9 +2335,11 @@ ssl_get_auto_dh(SSL *s)
         } else {
                 if ((cpk = ssl_get_server_send_pkey(s)) == NULL)
                         return (NULL);
-                if (cpk->privatekey == NULL || cpk->privatekey->pkey.dh == NULL)
+                if (cpk->privatekey == NULL ||
+                    EVP_PKEY_get0_RSA(cpk->privatekey) == NULL)
+                        return (NULL);
+                if ((keylen = EVP_PKEY_bits(cpk->privatekey)) <= 0)
                         return (NULL);
-                keylen = EVP_PKEY_bits(cpk->privatekey);
         }
 
         if ((dhp = DH_new()) == NULL)