On systems where we do not have BN_ULLONG defined (most 64-bit systems),

BN_mod_word() can return incorrect results if the supplied modulus is
too big, so we need to fall back to BN_div_word.

Now that BN_mod_word may fail, handle errors properly update the man page.

Thanks to Brian Smith for pointing out these fixes from BoringSSL:

https://boringssl.googlesource.com/boringssl/+/67cb49d045f04973ddba0f92fe8a8ad483c7da89
https://boringssl.googlesource.com/boringssl/+/44bedc348d9491e63c7ed1438db100a4b8a830be

ok beck@

1 files changed, 5 insertions, 1 deletions

diff --git a/src/lib/libcrypto/dh/dh_check.c b/src/lib/libcrypto/dh/dh_check.c
index 93e1003bd6..a6010f0a6d 100644
--- a/src/lib/libcrypto/dh/dh_check.c
+++ b/src/lib/libcrypto/dh/dh_check.c
@@ -1,4 +1,4 @@
-/* $OpenBSD: dh_check.c,v 1.15 2015/02/07 13:19:15 doug Exp $ */
+/* $OpenBSD: dh_check.c,v 1.16 2016/07/05 02:54:35 bcook Exp $ */
 /* Copyright (C) 1995-1998 Eric Young (eay@cryptsoft.com)
  * All rights reserved.
  *
@@ -89,10 +89,14 @@ DH_check(const DH *dh, int *ret)
 
         if (BN_is_word(dh->g, DH_GENERATOR_2)) {
                 l = BN_mod_word(dh->p, 24);
+                if (l == (BN_ULONG)-1)
+                        goto err;
                 if (l != 11)
                         *ret |= DH_NOT_SUITABLE_GENERATOR;
         } else if (BN_is_word(dh->g, DH_GENERATOR_5)) {
                 l = BN_mod_word(dh->p, 10);
+                if (l == (BN_ULONG)-1)
+                        goto err;
                 if (l != 3 && l != 7)
                         *ret |= DH_NOT_SUITABLE_GENERATOR;
         } else