Correctly detect b < a in BN_usub().

BN_usub() requires that a >= b and should return an error in the case that
b < a. This is currently only detected by checking the number of words in
a versus b - if they have the same number of words, the top word is not
checked and b < a, which then succeeds and produces an incorrect result.

Fix this by checking for the case where a and b have an equal number of
words, yet there is a borrow returned from bn_sub_words().

ok miod@ tb@

1 files changed, 5 insertions, 1 deletions

diff --git a/src/lib/libcrypto/bn/bn_add.c b/src/lib/libcrypto/bn/bn_add.c
index c5bc024f3f..cfc04fd032 100644
--- a/src/lib/libcrypto/bn/bn_add.c
+++ b/src/lib/libcrypto/bn/bn_add.c
@@ -1,4 +1,4 @@
-/* $OpenBSD: bn_add.c,v 1.19 2023/01/23 10:34:21 jsing Exp $ */
+/* $OpenBSD: bn_add.c,v 1.20 2023/01/31 05:16:52 jsing Exp $ */
 /* Copyright (C) 1995-1998 Eric Young (eay@cryptsoft.com)
  * All rights reserved.
  *
@@ -291,6 +291,10 @@ BN_usub(BIGNUM *r, const BIGNUM *a, const BIGNUM *b)
         rp = r->d;
 
         borrow = bn_sub_words(rp, ap, bp, min);
+        if (dif == 0 && borrow > 0) {
+                BNerror(BN_R_ARG2_LT_ARG3);
+                return 0;
+        }
         ap += min;
         rp += min;