Fix signed integer overflow in bnrand()

If more bits than INT_MAX - 7 are requested, the calculation of number
of bytes required to store the bignum triggers undefined behavior due to
signed integer overflow. This will typically result in bytes becoming
negative which will then make malloc() fail. If the ulimit should be
high enough to make malloc() succeed, there is a bad out of bounds write
in case bottom is set (an odd number was requested).

On jsing's request this does not deal with another bug which we could
catch with a similar check due to BN_bn2bin() failing later on as the
number of words in a BIGNUM is some fraction of INT_MAX.

ok jsing

1 files changed, 6 insertions, 1 deletions

diff --git a/src/lib/libcrypto/bn/bn_rand.c b/src/lib/libcrypto/bn/bn_rand.c
index a5b163c820..9cfcd8e2c0 100644
--- a/src/lib/libcrypto/bn/bn_rand.c
+++ b/src/lib/libcrypto/bn/bn_rand.c
@@ -1,4 +1,4 @@
-/* $OpenBSD: bn_rand.c,v 1.29 2023/08/03 18:53:55 tb Exp $ */
+/* $OpenBSD: bn_rand.c,v 1.30 2024/03/16 20:42:33 tb Exp $ */
 /* Copyright (C) 1995-1998 Eric Young (eay@cryptsoft.com)
  * All rights reserved.
  *
@@ -109,6 +109,7 @@
  *
  */
 
+#include <limits.h>
 #include <stdio.h>
 #include <stdlib.h>
 #include <string.h>
@@ -133,6 +134,10 @@ bnrand(int pseudorand, BIGNUM *rnd, int bits, int top, int bottom)
                 BNerror(BN_R_BITS_TOO_SMALL);
                 return (0);
         }
+        if (bits > INT_MAX - 7) {
+                BNerror(BN_R_BIGNUM_TOO_LONG);
+                return (0);
+        }
 
         if (bits == 0) {
                 BN_zero(rnd);