Close some major gaps in the documentation of EVP_Cipher(3),

and describe EVP_CIPHER_CTX_init(3) in a less misleading way.
Joint work with and OK tb@.

1 files changed, 65 insertions, 10 deletions

diff --git a/src/lib/libcrypto/man/EVP_CIPHER_CTX_init.3 b/src/lib/libcrypto/man/EVP_CIPHER_CTX_init.3
index 3bb40018f5..50df2e764d 100644
--- a/src/lib/libcrypto/man/EVP_CIPHER_CTX_init.3
+++ b/src/lib/libcrypto/man/EVP_CIPHER_CTX_init.3
@@ -1,11 +1,11 @@
-.\" $OpenBSD: EVP_CIPHER_CTX_init.3,v 1.1 2023/12/01 10:40:21 schwarze Exp $
+.\" $OpenBSD: EVP_CIPHER_CTX_init.3,v 1.2 2023/12/26 19:09:08 schwarze Exp $
 .\" full merge up to:
 .\" OpenSSL EVP_EncryptInit.pod 0874d7f2 Oct 11 13:13:47 2022 +0100
 .\"
 .\" This file is a derived work.
 .\" The changes are covered by the following Copyright and license:
 .\"
-.\" Copyright (c) 2018, 2019 Ingo Schwarze <schwarze@openbsd.org>
+.\" Copyright (c) 2018, 2019, 2023 Ingo Schwarze <schwarze@openbsd.org>
 .\"
 .\" Permission to use, copy, modify, and distribute this software for any
 .\" purpose with or without fee is hereby granted, provided that the above
@@ -68,7 +68,7 @@
 .\" ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED
 .\" OF THE POSSIBILITY OF SUCH DAMAGE.
 .\"
-.Dd $Mdocdate: December 1 2023 $
+.Dd $Mdocdate: December 26 2023 $
 .Dt EVP_CIPHER_CTX_INIT 3
 .Os
 .Sh NAME
@@ -95,11 +95,18 @@
 .Fc
 .Sh DESCRIPTION
 .Fn EVP_CIPHER_CTX_init
-is a deprecated function to clear a cipher context on the stack
-before use.
-Do not use it on a cipher context returned from
+is a deprecated function that could be used to clear a cipher context
+on the stack before
+.Vt EVP_CIPHER_CTX
+was made opaque.
+Calling it on a cipher context just returned from
 .Xr EVP_CIPHER_CTX_new 3
-or one that was already used.
+has no effect.
+Calling it on a cipher context that was already used leaks memory.
+Instead, use
+.Xr EVP_CIPHER_CTX_reset 3
+or
+.Xr EVP_CIPHER_CTX_free 3 .
 .Pp
 .Fn EVP_CIPHER_CTX_cleanup
 is a deprecated alias for
@@ -111,7 +118,41 @@ and frees all allocated memory associated with it, except the
 object itself.
 .Pp
 .Fn EVP_Cipher
-encrypts or decrypts aligned blocks of data
+exposes implementation details of the functions
+.Xr EVP_CipherUpdate 3
+and
+.Xr EVP_CipherFinal 3
+that should never have become part of the public API.
+.Pp
+If the flag
+.Dv EVP_CIPH_FLAG_CUSTOM_CIPHER
+is set for the cipher used by
+.Fa ctx ,
+behaviour depends on
+.Fa in .
+If that argument is
+.Dv NULL
+and
+.Fa inl
+is 0, behaviour is similar to
+.Xr EVP_CipherFinal 3 ;
+if
+.Fa inl
+is not 0, behaviour is undefined.
+If
+.Fa in
+is not
+.Dv NULL ,
+behaviour is similar to
+.Xr EVP_CipherUpdate 3 .
+In both cases, the exceptions to the similarity are that arguments
+and return values differ.
+.Pp
+If the flag
+.Dv EVP_CIPH_FLAG_CUSTOM_CIPHER
+is not set for the cipher used by
+.Fa ctx ,
+it encrypts or decrypts aligned blocks of data
 whose lengths match the cipher block size.
 It requires that the previous encryption or decryption operation
 using the same
@@ -133,9 +174,17 @@ the overhead incurred by using
 is minimal.
 .Sh RETURN VALUES
 .Fn EVP_CIPHER_CTX_cleanup
-and
+returns 1 for success or 0 for failure.
+.Pp
+With
+.Dv EVP_CIPH_FLAG_CUSTOM_CIPHER ,
 .Fn EVP_Cipher
-return 1 for success or 0 for failure.
+returns the number of bytes written to
+.Fa out
+for success or \-1 for failure.
+Without
+.Dv EVP_CIPH_FLAG_CUSTOM_CIPHER ,
+it returns 1 for success or 0 for failure.
 .Sh SEE ALSO
 .Xr evp 3 ,
 .Xr EVP_EncryptInit 3
@@ -148,3 +197,9 @@ first appeared in SSLeay 0.8.0.
 first appeared in SSLeay 0.9.0.
 All these functions have been available since
 .Ox 2.4 .
+.Sh CAVEATS
+Checking the return value of
+.Fn EVP_Cipher
+requires unusual caution: zero signals success if
+.Dv EVP_CIPH_FLAG_CUSTOM_CIPHER
+is set or failure otherwise.