Fix a surprising quirk in BN_GF2m_mod(3).

All other wrappers in the same file that use a temporary array of
degrees size that array dynamically, such that they are able to
handle reducing polynomials of arbitrary lengths.  BN_GF2m_mod(3)
was the only one that used a static array of size 6 instead, limiting
it to trinomials and pentanomials and causing it to fail for longer
reducing polynomials.

Make this more uniform and less surprising by using exactly the
same code as in all the other wrappers, such that BN_GF2m_mod(3)
works with reducing polynomials of arbitrary length, too, just like
the others.

Again, tb@ points out this quirk is very unlikely to cause
vulnerabilities in practice because cryptographic applications do
not use longer reducing polynomials.

This patch is not expected to significantly impact performance
because the relevant caller, BN_GF2m_mod_div(3), already uses dynamic
allocation via BN_GF2m_mod_mul(3).

OK tb@

1 files changed, 11 insertions, 5 deletions

diff --git a/src/lib/libcrypto/bn/bn_gf2m.c b/src/lib/libcrypto/bn/bn_gf2m.c
index 1fd7105a31..b9e3ba8566 100644
--- a/src/lib/libcrypto/bn/bn_gf2m.c
+++ b/src/lib/libcrypto/bn/bn_gf2m.c
@@ -1,4 +1,4 @@
-/* $OpenBSD: bn_gf2m.c,v 1.24 2022/11/20 22:23:43 schwarze Exp $ */
+/* $OpenBSD: bn_gf2m.c,v 1.25 2022/11/20 23:35:00 schwarze Exp $ */
 /* ====================================================================
  * Copyright 2002 Sun Microsystems, Inc. ALL RIGHTS RESERVED.
  *
@@ -464,17 +464,23 @@ int
 BN_GF2m_mod(BIGNUM *r, const BIGNUM *a, const BIGNUM *p)
 {
         int ret = 0;
-        int arr[6];
+        const int max = BN_num_bits(p) + 1;
+        int *arr = NULL;
 
         bn_check_top(a);
         bn_check_top(p);
-        ret = BN_GF2m_poly2arr(p, arr, sizeof(arr) / sizeof(arr[0]));
-        if (!ret || ret > (int)(sizeof(arr) / sizeof(arr[0]))) {
+        if ((arr = reallocarray(NULL, max, sizeof(int))) == NULL)
+                goto err;
+        ret = BN_GF2m_poly2arr(p, arr, max);
+        if (!ret || ret > max) {
                 BNerror(BN_R_INVALID_LENGTH);
-                return 0;
+                goto err;
         }
         ret = BN_GF2m_mod_arr(r, a, arr);
         bn_check_top(r);
+
+ err:
+        free(arr);
         return ret;
 }