Rewrite RSA_get_ex_new_index(3) and CRYPTO_set_ex_data(3) from scratch.

The defects of the old pages were too numerous to list in full but included
vagueness, gaps, misleading statements, bad ordering, and duplication.
Use my Copyright since none of the text we inherited from OpenSSL remains.

Without doing a thorough review, tb@ thinks he likes the new pages
after quickly reading through both of them.

1 files changed, 315 insertions, 222 deletions

diff --git a/src/lib/libcrypto/man/RSA_get_ex_new_index.3 b/src/lib/libcrypto/man/RSA_get_ex_new_index.3
index ee1e0e82f7..51a8f24cd8 100644
--- a/src/lib/libcrypto/man/RSA_get_ex_new_index.3
+++ b/src/lib/libcrypto/man/RSA_get_ex_new_index.3
@@ -1,65 +1,27 @@
-.\"     $OpenBSD: RSA_get_ex_new_index.3,v 1.11 2022/03/31 17:27:17 naddy Exp $
-.\"     OpenSSL 35cb565a Nov 19 15:49:30 2015 -0500
+.\" $OpenBSD: RSA_get_ex_new_index.3,v 1.12 2023/09/18 14:49:43 schwarze Exp $
 .\"
-.\" This file was written by Ulf Moeller <ulf@openssl.org> and
-.\" Dr. Stephen Henson <steve@openssl.org>.
-.\" Copyright (c) 2000, 2006 The OpenSSL Project.  All rights reserved.
+.\" Copyright (c) 2023 Ingo Schwarze <schwarze@openbsd.org>
 .\"
-.\" Redistribution and use in source and binary forms, with or without
-.\" modification, are permitted provided that the following conditions
-.\" are met:
+.\" Permission to use, copy, modify, and distribute this software for any
+.\" purpose with or without fee is hereby granted, provided that the above
+.\" copyright notice and this permission notice appear in all copies.
 .\"
-.\" 1. Redistributions of source code must retain the above copyright
-.\"    notice, this list of conditions and the following disclaimer.
+.\" THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
+.\" WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
+.\" MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
+.\" ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
+.\" WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
+.\" ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
+.\" OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
 .\"
-.\" 2. Redistributions in binary form must reproduce the above copyright
-.\"    notice, this list of conditions and the following disclaimer in
-.\"    the documentation and/or other materials provided with the
-.\"    distribution.
-.\"
-.\" 3. All advertising materials mentioning features or use of this
-.\"    software must display the following acknowledgment:
-.\"    "This product includes software developed by the OpenSSL Project
-.\"    for use in the OpenSSL Toolkit. (http://www.openssl.org/)"
-.\"
-.\" 4. The names "OpenSSL Toolkit" and "OpenSSL Project" must not be used to
-.\"    endorse or promote products derived from this software without
-.\"    prior written permission. For written permission, please contact
-.\"    openssl-core@openssl.org.
-.\"
-.\" 5. Products derived from this software may not be called "OpenSSL"
-.\"    nor may "OpenSSL" appear in their names without prior written
-.\"    permission of the OpenSSL Project.
-.\"
-.\" 6. Redistributions of any form whatsoever must retain the following
-.\"    acknowledgment:
-.\"    "This product includes software developed by the OpenSSL Project
-.\"    for use in the OpenSSL Toolkit (http://www.openssl.org/)"
-.\"
-.\" THIS SOFTWARE IS PROVIDED BY THE OpenSSL PROJECT ``AS IS'' AND ANY
-.\" EXPRESSED OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
-.\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
-.\" PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL THE OpenSSL PROJECT OR
-.\" ITS CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
-.\" SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
-.\" NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
-.\" LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
-.\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
-.\" STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
-.\" ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED
-.\" OF THE POSSIBILITY OF SUCH DAMAGE.
-.\"
-.Dd $Mdocdate: March 31 2022 $
+.Dd $Mdocdate: September 18 2023 $
 .Dt RSA_GET_EX_NEW_INDEX 3
 .Os
 .Sh NAME
 .Nm RSA_get_ex_new_index ,
 .Nm RSA_set_ex_data ,
-.Nm RSA_get_ex_data ,
-.Nm CRYPTO_EX_new ,
-.Nm CRYPTO_EX_dup ,
-.Nm CRYPTO_EX_free
-.Nd add application specific data to RSA structures
+.Nm RSA_get_ex_data
+.Nd add application specific data to RSA objects
 .Sh SYNOPSIS
 .In openssl/rsa.h
 .Ft int
@@ -72,194 +34,157 @@
 .Fc
 .Ft int
 .Fo RSA_set_ex_data
-.Fa "RSA *r"
+.Fa "RSA *rsa"
 .Fa "int idx"
-.Fa "void *arg"
+.Fa "void *data"
 .Fc
 .Ft void *
 .Fo RSA_get_ex_data
-.Fa "RSA *r"
-.Fa "int idx"
-.Fc
-.In openssl/crypto.h
-.Ft typedef int
-.Fo CRYPTO_EX_new
-.Fa "void *parent"
-.Fa "void *ptr"
-.Fa "CRYPTO_EX_DATA *ad"
+.Fa "RSA *rsa"
 .Fa "int idx"
-.Fa "long argl"
-.Fa "void *argp"
-.Fc
-.Ft typedef void
-.Fo CRYPTO_EX_free
-.Fa "void *parent"
-.Fa "void *ptr"
-.Fa "CRYPTO_EX_DATA *ad"
-.Fa "int idx"
-.Fa "long argl"
-.Fa "void *argp"
-.Fc
-.Ft typedef int
-.Fo CRYPTO_EX_dup
-.Fa "CRYPTO_EX_DATA *to"
-.Fa "CRYPTO_EX_DATA *from"
-.Fa "void *from_d"
-.Fa "int idx"
-.Fa "long argl"
-.Fa "void *argp"
 .Fc
 .Sh DESCRIPTION
-Several OpenSSL structures can have application specific data attached
-to them.
-This has several potential uses: it can be used to cache data associated
-with a structure (for example the hash of some part of the structure) or
-some additional data (for example a handle to the data in an external
-library).
+The following parent objects can have application specific data called
+.Dq ex_data
+attached to them:
+.Vt BIO , DH , DSA , EC_KEY , ENGINE , RSA ,
+.Vt SSL , SSL_CTX , SSL_SESSION , UI , X509 , X509_STORE ,
+and
+.Vt X509_STORE_CTX .
+.\" CRYPTO_EX_INDEX_APP and CRYPTO_EX_INDEX_UI_METHOD are unused.
+The present manual page documents the related API functions taking the
+.Vt RSA
+object type as an example.
+The functions for the other object types work in exactly the same way:
+just replace the string
+.Qq RSA
+with the name of the respective object type
+throughout the rest of this manual page.
 .Pp
-Since the application data can be anything at all, it is passed and
-retrieved as a
+By default, each individual
+.Vt RSA
+object can store one
 .Vt void *
-type.
+pointing to application specific data.
+That specific pointer is identified by an
+.Fa idx
+argument of 0.
 .Pp
-The
-.Fn RSA_get_ex_new_index
-function is initially called to "register" some new application specific
-data.
-It takes three optional function pointers which are called when the
-parent structure (in this case an RSA structure) is initially created,
-when it is copied and when it is freed up.
-If any or all of these function pointer arguments are not used, they
-should be set to
-.Dv NULL .
-The precise manner in which these function pointers are called is
-described in more detail below.
 .Fn RSA_get_ex_new_index
-also takes additional long and pointer parameters which will be passed
-to the supplied functions but which otherwise have no special meaning.
-It returns an index which should be stored (typically in a static
-variable) and passed as the
+reserves the next consecutive
 .Fa idx
-parameter in the remaining functions.
-Each successful call to
-.Fn RSA_get_ex_new_index
-will return an index greater than any previously returned.
-This is
-important because the optional functions are called in order of
-increasing index value.
-.Pp
-.Fn RSA_set_ex_data
-is used to set application specific data.
-The data is supplied in the
-.Fa arg
-parameter and its precise meaning is up to the application.
-.Pp
-.Fn RSA_get_ex_data
-is used to retrieve application specific data.
-The data is returned to the application, which will be the same value as
-supplied to a previous
-.Fn RSA_set_ex_data
-call.
-.Pp
-.Fa new_func
-is called when a structure is initially allocated (for example with
-.Xr RSA_new 3 .
-The parent structure members will not have any meaningful values at this
-point.
-This function will typically be used to allocate any application
-specific structure.
-.Pp
-.Fa free_func
-is called when a structure is being freed up.
-The dynamic parent structure members should not be accessed because they
-will be freed up when this function is called.
-.Pp
-.Fa new_func
-and
-.Fa free_func
-take the same parameters.
-.Fa parent
-is a pointer to the parent
+argument, enabling storage of one additional
+.Vt void *
+per
 .Vt RSA
-structure.
-.Fa ptr
-is the application specific data (this won't be of much use in
-.Fa new_func ) .
-.Fa ad
-is a pointer to the
-.Vt CRYPTO_EX_DATA
-structure from the parent
+object.
+It is typically called at program startup.
+It can be called more than once if some
 .Vt RSA
-structure: the functions
-.Fn CRYPTO_get_ex_data
+objects need to store more than two application specific pointers.
+Reserving an additional index for one parent object type, for example for
+.Vt RSA ,
+does not change the numbers of indices that can be used
+with any other parent object type.
+.Pp
+It is strongly recommended to always pass three
+.Dv NULL
+pointers for the arguments
+.Fa new_func ,
+.Fa dup_func ,
 and
-.Fn CRYPTO_set_ex_data
-can be called to manipulate it.
-The
-.Fa idx
-parameter is the index: this will be the same value returned by
-.Fn RSA_get_ex_new_index
-when the functions were initially registered.
-Finally the
+.Fa free_func .
+When following this recommendation, the arguments
 .Fa argl
 and
 .Fa argp
-parameters are the values originally passed to the same corresponding
-parameters when
-.Fn RSA_get_ex_new_index
-was called.
+are ignored; conventionally, passing 0 and
+.Dv NULL
+is recommended.
+Because using them is discouraged, the three function callback types
+are only documented in the low-level
+.Xr CRYPTO_EX_new 3
+manual page.
 .Pp
-.Fa dup_func
-is called when a structure is being copied.
-Pointers to the destination and source
-.Vt CRYPTO_EX_DATA
-structures are passed in the
-.Fa to
-and
-.Fa from
-parameters, respectively.
-The
-.Fa from_d
-parameter is passed a pointer to the source application data when the
-function is called.
-When the function returns, the value is copied to the destination:
-the application can thus modify the data pointed to by
-.Fa from_d
-and have different values in the source and destination.
-The
-.Fa idx ,
-.Fa argl ,
-and
-.Fa argp
-parameters are the same as those in
-.Fa new_func
-and
-.Fa free_func .
+.Fn RSA_set_ex_data
+stores the
+.Fa data
+pointer as application specific data at the given
+.Fa idx
+in the given
+.Fa rsa
+object.
+The meaning of the data pointed to is up to the application.
+The caller retains ownership of the
+.Fa data
+and is responsible for freeing it when neither the caller nor the
+.Fa rsa
+object need it any longer.
+Any other pointer that was previously stored at the same
+.Fa idx
+in the same
+.Fa rsa
+object is silently overwritten.
+Passing a
+.Dv NULL
+pointer for the
+.Fa data
+argument is valid and indicates that no application specific data
+currently needs to be stored at the given
+.Fa idx .
+.Pp
+.Fn RSA_get_ex_data
+retrieves the last pointer that was stored using
+.Fn RSA_set_ex_data
+at the given
+.Fa idx
+in the given
+.Fa rsa
+object.
 .Sh RETURN VALUES
 .Fn RSA_get_ex_new_index
-returns a new index or -1 on failure.
-Note that 0 is a valid index value.
+returns a new index equal to or greater than 1
+or \-1 if memory allocation fails.
 .Pp
 .Fn RSA_set_ex_data
-returns 1 on success or 0 on failure.
+returns 1 on success or 0 if memory allocation fails.
 .Pp
 .Fn RSA_get_ex_data
-returns the application data or
-.Dv NULL
-on failure.
+returns the application specific data or
 .Dv NULL
-may also be valid application data, but currently it can only fail if
-given an invalid
-.Fa idx
-parameter.
-.Pp
-.Fa new_func
+if
+.Fa rsa
+does not contain application specific data at the given
+.Fa idx .
+.Sh ERRORS
+After failure of
+.Fn RSA_get_ex_new_index
+or
+.Fn RSA_set_ex_data ,
+the following diagnostic can be retrieved with
+.Xr ERR_get_error 3 ,
+.Xr ERR_GET_REASON 3 ,
 and
-.Fa dup_func
-should return 0 for failure and 1 for success.
+.Xr ERR_reason_error_string 3 :
+.Bl -tag -width Ds
+.It Dv ERR_R_MALLOC_FAILURE Qq "malloc failure"
+Memory allocation failed.
+.El
+.Pp
+In a few unusual failure cases,
+.Xr ERR_get_error 3
+may report different errors caused by
+.Xr OPENSSL_init_crypto 3
+or even none at all.
 .Pp
-On failure an error code can be obtained from
-.Xr ERR_get_error 3 .
+.Fn RSA_get_ex_data
+does not distinguish success from failure.
+Consequently, after
+.Fn RSA_get_ex_data
+returns
+.Dv NULL ,
+.Xr ERR_get_error 3
+returns 0 unless there is still an earlier error in the queue.
 .Sh SEE ALSO
 .Xr BIO_set_ex_data 3 ,
 .Xr CRYPTO_set_ex_data 3 ,
@@ -275,15 +200,183 @@ On failure an error code can be obtained from
 These functions first appeared in SSLeay 0.9.0
 and have been available since
 .Ox 2.4 .
-.Sh BUGS
-.Fa dup_func
-is currently never called.
+.Sh CAVEATS
+A relatively small minority of application programs
+attempt to change the API contract such that
+.Fn RSA_set_ex_data
+transfers ownership of the
+.Fa data
+to the
+.Fa rsa
+object.
+They do this by providing a
+.Fa free_func
+that calls
+.Xr free 3
+or higher-level
+.Fn *_free
+functions on the
+.Fa data
+and sometimes also attempt additional cleanup work as a side effect.
 .Pp
-The return value of
-.Fa new_func
-is ignored.
+This practice is discouraged for several reasons:
+.Bl -enum
+.It
+Due to a massive design mistake in the low-level API function
+.Xr CRYPTO_free_ex_data 3 ,
+this practice creates a possibility that
+.Xr RSA_free 3
+may fail due to memory allocation failure, consequently leaking the
+memory containing the application specific data and silently skipping
+any additional cleanup work the
+.Fa free_func
+was supposed to do, leaving the application in an undetectably
+inconsistent state.
+Arguably, leaking additional memory while trying to free some
+is most unfortunate especially when the program
+is already starved for memory.
+.It
+This practice introduces a risk of use-after-free and double-free
+bugs in case the
+.Fa rsa
+object gets destructed while a caller of
+.Fn RSA_set_ex_data
+or
+.Fn RSA_get_ex_data
+still holds a
+.Fa data
+pointer.
+No such risk exists when no
+.Fa free_func
+is installed.
+.It
+Attempting additional cleanup work in
+.Fa free_func
+is an even worse idea because
+.Fa free_func
+is unable to report any issues it might detect while doing that work.
+Instead, if any additional cleanup work is needed, it is recommended
+that the calling code takes care of that before calling
+.Xr RSA_free 3 .
+.El
 .Pp
-The
+Even fewer application programs install a
+.Fa new_func
+that allocates memory and stores a pointer to it in the
+.Fa rsa
+object by calling
+.Xr CRYPTO_set_ex_data 3 .
+That is useless because
 .Fa new_func
-function isn't very useful because no meaningful values are present in
-the parent RSA structure when it is called.
+does not have access to any useful information it could store in such memory
+and because the default return value of
+.Dv NULL
+from
+.Fn RSA_get_ex_data
+is sufficient to indicate
+that no application specific data has been stored yet.
+In addition, allocating memory in
+.Fa new_func
+is also inadvisable because it introduces an additional responsibility
+for callers of
+.Fn RSA_set_ex_data
+to always call
+.Fn RSA_get_ex_data
+first, even when it is the first time the application wants to set
+application specific data in a particular
+.Fa rsa
+object, and to either modify whatever
+.Fn RSA_get_ex_data
+returns or to free it before calling
+.Fn RSA_set_ex_data .
+If that is forgotten, a memory leak results.
+.Pp
+Consequently, allocating any required memory
+is better left to the application code that calls
+.Fn RSA_set_ex_data .
+.Pp
+Installing a
+.Fa dup_func
+is often seen in combination with installing a
+.Fa free_func ,
+for obvious reasons.
+It is rarely useful because for most parent object types
+that support ex_data, including for
+.Vt RSA ,
+the library does not provide a copying API function in the first place, and
+even where copying functions exist, they tend to be fragile and error-prone.
+When a new object is needed, it is usually advisable to construct it from
+scratch whenever possible, rather than attempting a copy operation.
+.Pp
+On top of that, if
+.Fa dup_func
+fails, for example because of a memory allocation failure, the
+failure is neither reported nor detectable in any way, leaving the
+new parent object with incomplete data and potentially in an
+inconsistent state.
+.Sh BUGS
+If
+.Fn RSA_set_ex_data
+fails, recovery is very difficult.
+In particular, calling
+.Xr RSA_free 3
+on the parent
+.Fa rsa
+object right afterwards is likely to also hit a memory allocation
+failure, leaking all memory internally allocated by all earlier calls of
+.Fn RSA_set_ex_data
+on
+.Fa rsa
+rather than freeing that memory.
+In order to recover, the application program
+would have to free a sufficient amount of
+.Em other
+memory before calling
+.Xr RSA_free 3 ,
+which will rarely be feasible.
+Consequently, after a failure of
+.Fn RSA_set_ex_data ,
+terminating the program is likely the only reasonable option.
+.Pp
+If
+.Fn RSA_set_ex_data
+is called with an
+.Fa idx
+argument greater than the last one previously returned from
+.Fn RSA_get_ex_new_index ,
+it may still succeed, and though that is not guaranteed by the API,
+retrieving the
+.Fa data
+from such a bogus
+.Fa idx
+may even be possible with
+.Fn RSA_get_ex_data ,
+hiding the bug in the application program that caused passing the bogus
+.Fa idx
+to
+.Fn RSA_set_ex_data
+in the first place.
+.Pp
+If the bogus
+.Fa idx
+argument is large,
+.Fn RSA_set_ex_data
+may uselessly allocate a large amount of memory.
+Calling
+.Xr RSA_free 3
+on the parent
+.Fa rsa
+object is the only way to recover that memory.
+.Pp
+If the bogus
+.Fa idx
+argument is very large,
+.Fn RSA_set_ex_data
+is likely to cause a significant delay before eventually failing
+due to memory exhaustion.
+It is likely to return without releasing the memory already
+allocated, causing any subsequent attempt to allocate memory
+for other purposes to fail, too.
+In this situation, what was said above about failure of
+.Fn RSA_set_ex_data
+applies, so terminating the program is likely the only reasonable option.