Disallow ciphers with EVP_CIPH_FLAG_CUSTOM_CIPHER in CMAC

These are usually AEAD ciphers, for which CMAC makes little sense (if you
need a MAC and all you have is an AEAD, you don't need CMAC, you can just
use a zero length cipher text). Also, since the CMAC implementation only
allows 64 and 128 bit block sizes, the AEADs would error out later anyway.

The only family of ciphers this effectively excludes is AES key wrap, for
which CMAC makes little sense.

One notable side effect of doing this is that the EVP_Cipher() return
value checks in the CMAC code magically become correct. EVP. What's not to
love about it.

ok jsing

1 files changed, 8 insertions, 1 deletions

diff --git a/src/lib/libcrypto/cmac/cmac.c b/src/lib/libcrypto/cmac/cmac.c
index f653219b8f..0df40277bc 100644
--- a/src/lib/libcrypto/cmac/cmac.c
+++ b/src/lib/libcrypto/cmac/cmac.c
@@ -1,4 +1,4 @@
-/* $OpenBSD: cmac.c,v 1.16 2023/11/29 21:35:57 tb Exp $ */
+/* $OpenBSD: cmac.c,v 1.17 2023/12/15 13:45:05 tb Exp $ */
 /* Written by Dr Stephen N Henson (steve@openssl.org) for the OpenSSL
  * project.
  */
@@ -191,6 +191,13 @@ CMAC_Init(CMAC_CTX *ctx, const void *key, size_t keylen,
 
         /* Initialise context. */
         if (cipher != NULL) {
+                /*
+                 * Disallow ciphers for which EVP_Cipher() behaves differently.
+                 * These are AEAD ciphers (or AES keywrap) for which the CMAC
+                 * construction makes little sense.
+                 */
+                if ((cipher->flags & EVP_CIPH_FLAG_CUSTOM_CIPHER) != 0)
+                        return 0;
                 if (!EVP_EncryptInit_ex(&ctx->cctx, cipher, NULL, NULL, NULL))
                         return 0;
         }