Fix return values bug in BN_ucmp().

BN_ucmp() is supposed to return -1/0/1 on a < b, a == b and a > b, however
it currently returns other negative and positive values when the top of
a and b differ. Correct this.

ok tb@

1 files changed, 6 insertions, 4 deletions

diff --git a/src/lib/libcrypto/bn/bn_lib.c b/src/lib/libcrypto/bn/bn_lib.c
index 0ac3977c19..df43da5db6 100644
--- a/src/lib/libcrypto/bn/bn_lib.c
+++ b/src/lib/libcrypto/bn/bn_lib.c
@@ -1,4 +1,4 @@
-/* $OpenBSD: bn_lib.c,v 1.64 2022/11/30 01:47:19 jsing Exp $ */
+/* $OpenBSD: bn_lib.c,v 1.65 2022/11/30 02:52:25 jsing Exp $ */
 /* Copyright (C) 1995-1998 Eric Young (eay@cryptsoft.com)
  * All rights reserved.
  *
@@ -691,9 +691,11 @@ BN_ucmp(const BIGNUM *a, const BIGNUM *b)
         BN_ULONG t1, t2, *ap, *bp;
 
 
-        i = a->top - b->top;
-        if (i != 0)
-                return (i);
+        if (a->top < b->top)
+                return -1;
+        if (a->top > b->top)
+                return 1;
+
         ap = a->d;
         bp = b->d;
         for (i = a->top - 1; i >= 0; i--) {