Avoid infinite loop for custom curves of order 1

If a private key encoded with EC parameters happens to have
order 1 and is used for ECDSA signatures, this causes an
infinite loop since a random integer x in the interval [0,1)
will be 0, so do ... while (x == 0); will loop indefinitely.

Found and reported with a reproducer by Hanno Boeck.
Helpful comments and analysis from David Benjamin.

ok beck jsing

1 files changed, 6 insertions, 1 deletions

diff --git a/src/lib/libcrypto/ecdsa/ecs_ossl.c b/src/lib/libcrypto/ecdsa/ecs_ossl.c
index 2429e36b59..0203b01bb5 100644
--- a/src/lib/libcrypto/ecdsa/ecs_ossl.c
+++ b/src/lib/libcrypto/ecdsa/ecs_ossl.c
@@ -1,4 +1,4 @@
-/* $OpenBSD: ecs_ossl.c,v 1.23 2022/01/20 11:03:48 inoguchi Exp $ */
+/* $OpenBSD: ecs_ossl.c,v 1.24 2022/04/07 17:37:25 tb Exp $ */
 /*
  * Written by Nils Larsch for the OpenSSL project
  */
@@ -163,6 +163,11 @@ ecdsa_sign_setup(EC_KEY *eckey, BN_CTX *ctx_in, BIGNUM **kinvp, BIGNUM **rp)
                 goto err;
         }
 
+        if (BN_cmp(order, BN_value_one()) <= 0) {
+                ECDSAerror(EC_R_INVALID_GROUP_ORDER);
+                goto err;
+        }
+
         /* Preallocate space. */
         order_bits = BN_num_bits(order);
         if (!BN_set_bit(k, order_bits) ||