Make truncation in ASN1_BIT_STRING_set_bit() explicit

Instead of relying on i2c_ASN1_BIT_STRING() to determine the "unused"
bits on encoding, set them explicitly in abs->flags via a call to
asn1_abs_set_unused_bits(). This means ASN1_STRING_FLAGS_BITS_LEFT is
now set on a bit string, which was previously explicitly cleared.

This also means that the encoding of a non-zero ASN1_BIT_STRING
populated by setting the bits individually will now go through the
if (a->flags & ASN1_STRING_FLAG_BITS_LEFT) path in i2c_ASN1_BIT_STRING().

The most prominent usage of this function is in X.509 for the keyUsage
extension or the CRL reason codes. There's also the NS cert type, TS
PKIFailureInfo and general BITLIST config strings.

The reason for the truncation logic comes from the DER for NamedBitLists
X.690, 11.2.2 below:

  X.680, 22.7:

   When a "NamedBitList" is used in defining a bitstring type ASN.1
   encoding rules are free to add (or remove) arbitrarily any trailing 0
   bits to (or from) values that are being encoded or decoded. Application
   designers should therefore ensure that different semantics are not
   associated with such values which differ only in the number of trailing
   0 bits.

  X.690, 11.2.2

   Where ITU-T Rec. X.680 | ISO/IEC 8824-1, 22.7, applies, the bitstring
   shall have all trailing 0 bits removed before it is encoded.

   Note 1 - In the case where a size constraint has been applied, the
   abstract value delivered by a decoder to the application will be one of
   those satisfying the size constraint and differing from the transmitted
   value only in the number of trailing zero bits.

   Note 2 - If a bitstring value has no 1 bits, then an encoder shall
   encode the value with a length of 1 and an initial octet set to 0.

ok kenjiro (on an earlier version) jsing

1 files changed, 43 insertions, 9 deletions

diff --git a/src/lib/libcrypto/asn1/a_bitstr.c b/src/lib/libcrypto/asn1/a_bitstr.c
index e656c43f0c..d84ecb025e 100644
--- a/src/lib/libcrypto/asn1/a_bitstr.c
+++ b/src/lib/libcrypto/asn1/a_bitstr.c
@@ -1,4 +1,4 @@
-/* $OpenBSD: a_bitstr.c,v 1.48 2026/01/04 09:54:23 tb Exp $ */
+/* $OpenBSD: a_bitstr.c,v 1.49 2026/02/08 10:27:00 tb Exp $ */
 /* Copyright (C) 1995-1998 Eric Young (eay@cryptsoft.com)
  * All rights reserved.
  *
@@ -108,6 +108,39 @@ asn1_abs_set_unused_bits(ASN1_BIT_STRING *abs, uint8_t unused_bits)
         return 1;
 }
 
+/*
+ * X.690, 11.2.2: [When a "NamedBitList" is used] [...], the bitstring shall
+ * have all trailing 0 bits removed before it is encoded.
+ */
+static int
+asn1_abs_trim_trailing_zero_bits(ASN1_BIT_STRING *abs)
+{
+        int unused_bits = 0;
+
+        /* Remove trailing zero octets. */
+        while (abs->length > 0 && abs->data[abs->length - 1] == 0)
+                abs->length--;
+
+        /* Remove trailing zero bits by setting the unused bits octet. */
+        if (abs->length > 0) {
+                uint8_t u8 = abs->data[abs->length - 1];
+
+                /* u8 != 0. Only keep least significant bit. */
+                u8 &= 0x100 - u8;
+
+                /* Count trailing zero bits. */
+                unused_bits = 7;
+                if ((u8 & 0x0f) != 0)
+                        unused_bits -= 4;
+                if ((u8 & 0x33) != 0)
+                        unused_bits -= 2;
+                if ((u8 & 0x55) != 0)
+                        unused_bits -= 1;
+        }
+
+        return asn1_abs_set_unused_bits(abs, unused_bits);
+}
+
 int
 ASN1_BIT_STRING_set(ASN1_BIT_STRING *x, unsigned char *d, int len)
 {
@@ -133,12 +166,14 @@ ASN1_BIT_STRING_set_bit(ASN1_BIT_STRING *a, int n, int value)
         if (value == 0)
                 v = 0;
 
-        asn1_abs_clear_unused_bits(a);
-
         if (a->length < w + 1 || a->data == NULL) {
-                /* Don't expand if there's no bit to set. */
+                /*
+                 * Don't expand if there's no bit to set.
+                 * XXX - switch back to return 1 here when we drop
+                 * ASN1_STRING_FLAG_BITS_LEFT?
+                 */
                 if (value == 0)
-                        return 1;
+                        return asn1_abs_trim_trailing_zero_bits(a);
                 if ((c = recallocarray(a->data, a->length, w + 1, 1)) == NULL) {
                         ASN1error(ERR_R_MALLOC_FAILURE);
                         return 0;
@@ -147,11 +182,9 @@ ASN1_BIT_STRING_set_bit(ASN1_BIT_STRING *a, int n, int value)
                 a->length = w + 1;
         }
 
-        a->data[w] = ((a->data[w]) & iv) | v;
-        while (a->length > 0 && a->data[a->length - 1] == 0)
-                a->length--;
+        a->data[w] = (a->data[w] & iv) | v;
 
-        return 1;
+        return asn1_abs_trim_trailing_zero_bits(a);
 }
 LCRYPTO_ALIAS(ASN1_BIT_STRING_set_bit);
 
@@ -189,6 +222,7 @@ i2c_ASN1_BIT_STRING(ASN1_BIT_STRING *a, unsigned char **pp)
                 if (a->flags & ASN1_STRING_FLAG_BITS_LEFT) {
                         bits = (int)a->flags & 0x07;
                 } else {
+                        /* XXX - dedup with asn1_abs_trim_trailing_zero_bits? */
                         j = 0;
                         for (; len > 0; len--) {
                                 if (a->data[len - 1])