Fix EVP_CIPHER_CTX_iv_length()

In today's episode of "curly nonsense from EVP land" we deal with a quite
harmless oversight and a not too bad suboptimal fix, relatively speaking.

At some point EVP_CIPHER_{CCM,GCM}_SET_IVLEN was added. It modified some
object hanging off of EVP_CIPHER. However, EVP_CIPHER_CTX_iv_length() wasn't
taught about this and kept returning the hardcoded default value on the
EVP_CIPHER. Once it transpired that a doc fix isn't going to cut it, this
was fixed. And of course it's easy to fix: you only have to dive through
about three layers of EVP, test and set a flag and handle a control in a
couple methods.

The upstream fix was done poorly and we begrudgingly have to match the API:
the caller is expected to pass a raw pointer next to a 0 length along with
EVP_CIPHER_GET_IV_LENGTH and the control handler goes *(int *)ptr = length
in full YOLO mode. That's never going to be an issue because of course the
caller will always pass a properly aligned pointer backing a sufficient
amount of memory. Yes, unlikely to be a real issue, but it could have been
done with proper semantics and checks without complicating the code. But
why do I even bother to complain? We're used to this.

Of note here is that there was some pushback painting other corners of a
bikeshed until the reviewer gave up with a resigned

  That kind of changes the semantics and is one extra complexity level,
  but [shrug] ok...

Anyway, the reason this matters now after so many years is that rust-openssl
has an assert, notably added in a +758 -84 commit with the awesome message
"Docs" that gets triggered by recent tests added to py-cryptography.

Thanks to Alex Gaynor for reporting this. Let me take the opportunity to
point out that pyca contributed to improve rust-openssl, in particular its
libressl support, quite a bit. That's much appreciated and very noticeable.

Regress coverage to follow in subsequent commits.

Based on OpenSSL PR #9499 and issue #8330.

ok beck jsing

PS: A few macros were kept internal for now to avoid impact on the release
cycle that is about to finish. They will be exposed after release.

1 files changed, 12 insertions, 3 deletions

diff --git a/src/lib/libcrypto/evp/e_aes.c b/src/lib/libcrypto/evp/e_aes.c
index 3d3b1a9d6c..3d357f0119 100644
--- a/src/lib/libcrypto/evp/e_aes.c
+++ b/src/lib/libcrypto/evp/e_aes.c
@@ -1,4 +1,4 @@
-/* $OpenBSD: e_aes.c,v 1.53 2023/07/07 19:37:53 beck Exp $ */
+/* $OpenBSD: e_aes.c,v 1.54 2023/09/28 11:29:10 tb Exp $ */
 /* ====================================================================
  * Copyright (c) 2001-2011 The OpenSSL Project.  All rights reserved.
  *
@@ -1305,7 +1305,11 @@ aes_gcm_ctrl(EVP_CIPHER_CTX *c, int type, int arg, void *ptr)
                 gctx->tls_aad_len = -1;
                 return 1;
 
-        case EVP_CTRL_GCM_SET_IVLEN:
+        case EVP_CTRL_AEAD_GET_IVLEN:
+                *(int *)ptr = gctx->ivlen;
+                return 1;
+
+        case EVP_CTRL_AEAD_SET_IVLEN:
                 if (arg <= 0)
                         return 0;
                 /* Allocate memory for IV if needed */
@@ -1631,6 +1635,7 @@ aes_gcm_cipher(EVP_CIPHER_CTX *ctx, unsigned char *out,
 
 #define CUSTOM_FLAGS \
     ( EVP_CIPH_FLAG_DEFAULT_ASN1 | EVP_CIPH_CUSTOM_IV | \
+      EVP_CIPH_FLAG_CUSTOM_IV_LENGTH | \
       EVP_CIPH_FLAG_CUSTOM_CIPHER | EVP_CIPH_ALWAYS_CALL_INIT | \
       EVP_CIPH_CTRL_INIT | EVP_CIPH_CUSTOM_COPY )
 
@@ -1968,7 +1973,11 @@ aes_ccm_ctrl(EVP_CIPHER_CTX *c, int type, int arg, void *ptr)
                 cctx->len_set = 0;
                 return 1;
 
-        case EVP_CTRL_CCM_SET_IVLEN:
+        case EVP_CTRL_AEAD_GET_IVLEN:
+                *(int *)ptr = 15 - cctx->L;
+                return 1;
+
+        case EVP_CTRL_AEAD_SET_IVLEN:
                 arg = 15 - arg;
 
         case EVP_CTRL_CCM_SET_L: