Remove a bogus memcmp in range_should_be_prefix()

range_should_be_prefix() currently always fails. The reason for this
is that OpenSSL commit 42d7d7dd incorrectly moved a memcmp() out of
an assertion.  As a consequence, the library emits and accepts
incorrectly encoded ipAddrBlock extensions since it will never detect
ranges that MUST be encoded as a prefix according to RFC 3779, 2.2.3.7.

The return -1 from this memcmp() indicates to the callers that the
range should be expressed as a range, so callers must check beforehand
that min <= max to be able to fail. Thus, remove this memcmp() and
add a check to make_addressRange(), the only caller that didn't already
ensure that min <= max.

This fixes the noisy output in regress/lib/libcrypto/x509/rfc3779.

ok inoguchi jsing

1 files changed, 6 insertions, 3 deletions

diff --git a/src/lib/libcrypto/x509/x509_addr.c b/src/lib/libcrypto/x509/x509_addr.c
index 705fc7df32..c6eac91aaa 100644
--- a/src/lib/libcrypto/x509/x509_addr.c
+++ b/src/lib/libcrypto/x509/x509_addr.c
@@ -1,4 +1,4 @@
-/*      $OpenBSD: x509_addr.c,v 1.61 2022/01/05 07:37:01 tb Exp $ */
+/*      $OpenBSD: x509_addr.c,v 1.62 2022/01/05 07:47:15 tb Exp $ */
 /*
  * Contributed to the OpenSSL Project by the American Registry for
  * Internet Numbers ("ARIN").
@@ -755,6 +755,8 @@ v6IPAddressOrRange_cmp(const IPAddressOrRange *const *a,
 /*
  * Calculate whether a range collapses to a prefix.
  * See last paragraph of RFC 3779 2.2.3.7.
+ *
+ * It's the caller's responsibility to ensure that min <= max.
  */
 static int
 range_should_be_prefix(const unsigned char *min, const unsigned char *max,
@@ -763,8 +765,6 @@ range_should_be_prefix(const unsigned char *min, const unsigned char *max,
         unsigned char mask;
         int i, j;
 
-        if (memcmp(min, max, length) <= 0)
-                return -1;
         for (i = 0; i < length && min[i] == max[i]; i++)
                 continue;
         for (j = length - 1; j >= 0 && min[j] == 0x00 && max[j] == 0xff; j--)
@@ -863,6 +863,9 @@ make_addressRange(IPAddressOrRange **result, unsigned char *min,
         IPAddressOrRange *aor;
         int i, prefix_len;
 
+        if (memcmp(min, max, length) > 0)
+                return 0;
+
         if ((prefix_len = range_should_be_prefix(min, max, length)) >= 0)
                 return make_addressPrefix(result, min, afi, prefix_len);