Fix some return checks in ecdh_cms_encrypt()

i2d functions return <= 0 on error, so check for that instead of == 0.

The issue with CMS_SharedInfo_encode() was found by Niels Dossche.
OpenSSL review overlooked that they had turned penclen into a size_t.

In principle the issue with i2d_X509_ALGOR() is purely cosmetic. Why do
a strange check when there is an idiomatic check? Then again this is CMS...

ok jsing

1 files changed, 3 insertions, 3 deletions

diff --git a/src/lib/libcrypto/ec/ec_ameth.c b/src/lib/libcrypto/ec/ec_ameth.c
index d9216cc451..a3ac989e6f 100644
--- a/src/lib/libcrypto/ec/ec_ameth.c
+++ b/src/lib/libcrypto/ec/ec_ameth.c
@@ -1,4 +1,4 @@
-/* $OpenBSD: ec_ameth.c,v 1.36 2022/11/26 16:08:52 tb Exp $ */
+/* $OpenBSD: ec_ameth.c,v 1.37 2023/03/06 08:37:24 tb Exp $ */
 /* Written by Dr Stephen N Henson (steve@openssl.org) for the OpenSSL
  * project 2006.
  */
@@ -976,7 +976,7 @@ ecdh_cms_encrypt(CMS_RecipientInfo *ri)
                 goto err;
 
         penclen = CMS_SharedInfo_encode(&penc, wrap_alg, ukm, keylen);
-        if (!penclen)
+        if (penclen <= 0)
                 goto err;
 
         if (EVP_PKEY_CTX_set0_ecdh_kdf_ukm(pctx, penc, penclen) <= 0)
@@ -988,7 +988,7 @@ ecdh_cms_encrypt(CMS_RecipientInfo *ri)
          * of another AlgorithmIdentifier.
          */
         penclen = i2d_X509_ALGOR(wrap_alg, &penc);
-        if (!penc || !penclen)
+        if (penclen <= 0)
                 goto err;
         wrap_str = ASN1_STRING_new();
         if (wrap_str == NULL)