Bound cofactor in EC_GROUP_set_generator()

Instead of bounding only bounding the group order, also bound the
cofactor using Hasse's theorem. This could probably be made a lot
tighter since all curves of cryptographic interest have small
cofactors,  but for now this is good enough.

A timeout found by oss-fuzz creates a "group" with insane parameters
over a 40-bit field: the order is 14464, and the cofactor has 4196223
bits (which is obviously impossible by Hasse's theorem). These led to
running an expensive loop in ec_GFp_simple_mul_ct() millions of times.

Fixes oss-fuzz #46056

Diagnosed and fix joint with jsing

ok inoguchi jsing (previous version)

1 files changed, 7 insertions, 1 deletions

diff --git a/src/lib/libcrypto/ec/ec_lib.c b/src/lib/libcrypto/ec/ec_lib.c
index 455d44a942..888f1edfcf 100644
--- a/src/lib/libcrypto/ec/ec_lib.c
+++ b/src/lib/libcrypto/ec/ec_lib.c
@@ -1,4 +1,4 @@
-/* $OpenBSD: ec_lib.c,v 1.43 2022/03/29 13:48:40 tb Exp $ */
+/* $OpenBSD: ec_lib.c,v 1.44 2022/03/29 14:03:12 tb Exp $ */
 /*
  * Originally written by Bodo Moeller for the OpenSSL project.
  */
@@ -385,6 +385,12 @@ EC_GROUP_set_generator(EC_GROUP *group, const EC_POINT *generator,
         } else if (!ec_guess_cofactor(group))
                 return 0;
 
+        /* Use Hasse's theorem to bound the cofactor. */
+        if (BN_num_bits(&group->cofactor) > BN_num_bits(&group->field) + 1) {
+                ECerror(EC_R_INVALID_GROUP_ORDER);
+                return 0;
+        }
+
         return 1;
 }