Make it possible to signal an error from an i2c_* function.

In asn1_i2d_ex_primitive(), asn1_ex_i2c() returning -1 is used to indicate
that the object is optional and should be skipped, while -2 is used to
indicate that indefinite length encoding should be used. Any other negative
value was treated as success, resulting in the out pointer being walked
backwards. Avoid this by treating any negative value (aside from -1 and -2)
as a failure, propagating it up the stack.

Additionally, check the return value of the second asn1_ex_i2c() call to
ensure that it matches the value returned by the first call. This makes
sure that the length of the encoded object is correct, plus it detects the
case where a failure occurs during the second call.

Discussed with tb@ (who also flagged the negative value issue).

1 files changed, 7 insertions, 2 deletions

diff --git a/src/lib/libcrypto/asn1/tasn_enc.c b/src/lib/libcrypto/asn1/tasn_enc.c
index ee2320f856..d42c5ea369 100644
--- a/src/lib/libcrypto/asn1/tasn_enc.c
+++ b/src/lib/libcrypto/asn1/tasn_enc.c
@@ -1,4 +1,4 @@
-/* $OpenBSD: tasn_enc.c,v 1.24 2022/01/07 11:13:54 tb Exp $ */
+/* $OpenBSD: tasn_enc.c,v 1.25 2022/08/20 17:55:08 jsing Exp $ */
 /* Written by Dr Stephen N Henson (steve@openssl.org) for the OpenSSL
  * project 2000.
  */
@@ -529,6 +529,10 @@ asn1_i2d_ex_primitive(ASN1_VALUE **pval, unsigned char **out,
                 len = 0;
         }
 
+        /* Treat any other negative value as an error. */
+        if (len < 0)
+                return -1;
+
         /* If not implicitly tagged get tag from underlying type */
         if (tag == -1)
                 tag = utype;
@@ -537,7 +541,8 @@ asn1_i2d_ex_primitive(ASN1_VALUE **pval, unsigned char **out,
         if (out) {
                 if (usetag)
                         ASN1_put_object(out, ndef, len, tag, aclass);
-                asn1_ex_i2c(pval, *out, &utype, it);
+                if (asn1_ex_i2c(pval, *out, &utype, it) != len)
+                        return -1;
                 if (ndef)
                         ASN1_put_eoc(out);
                 else