Fix an off-by-one bug in BN_GF2m_poly2arr(3).

If the last argument, the size of the output array, is too small to
contain all degrees present in the input polynomial plus one for the
terminating -1, the function is documented to return the size of the
output array that would be needed (in comments in the source code, in
the new manual page, and by the way how the function is used by other
functions in the same file).  However, in case of overflow, the existing
code failed to include the element needed for the terminating -1 in the
return value, wrongly indicating success if everything but the -1 did
fit and reporting failure with a size that was still too small otherwise.

According to tb@, this is very unlikely to cause vulnerabilities in
practical applications because there is no real reason to pick a
reducing polynomial longer than a pentanomial, because all known
callers use either fixed size arrays of size 6 or dynamic allocation,
because use of GF(2^m) is rare in practice, and GF(2^m) with custom
reducing polynomials even more so.

OK tb@

1 files changed, 3 insertions, 4 deletions

diff --git a/src/lib/libcrypto/bn/bn_gf2m.c b/src/lib/libcrypto/bn/bn_gf2m.c
index 8562b3f87e..1fd7105a31 100644
--- a/src/lib/libcrypto/bn/bn_gf2m.c
+++ b/src/lib/libcrypto/bn/bn_gf2m.c
@@ -1,4 +1,4 @@
-/* $OpenBSD: bn_gf2m.c,v 1.23 2017/01/29 17:49:22 beck Exp $ */
+/* $OpenBSD: bn_gf2m.c,v 1.24 2022/11/20 22:23:43 schwarze Exp $ */
 /* ====================================================================
  * Copyright 2002 Sun Microsystems, Inc. ALL RIGHTS RESERVED.
  *
@@ -1291,10 +1291,9 @@ BN_GF2m_poly2arr(const BIGNUM *a, int p[], int max)
                 }
         }
 
-        if (k < max) {
+        if (k < max)
                 p[k] = -1;
-                k++;
-        }
+        k++;
 
         return k;
 }