Revamp malloc.3 by reordering the sections and rewriting parts.

The old man page had a lot of useful information, but it was all mixed
together which made it difficult to reference.  The main theme in this
commit is that the sections are more focused:

* DESCRIPTION describes the overall behavior
* RETURN VALUES describes what it may return (including implementation
  defined values)
* EXAMPLES shows why we recently started an audit on malloc and realloc
  usage in the tree.
* Added CAVEATS which describes what is implementation defined, gotchas
  and security implications of misusing these functions
* Added IDIOMS which describes how these functions should or
  should not be used

The MALLOC_OPTIONS section was left unchanged.  Function names were
added to DIAGNOSTICS and STANDARDS.  The MALLOC_OPTIONS and DIAGNOSTICS
sections were pushed down in the page so more pertinent information is
higher up.

This has gone through several revisions thanks to input from deraadt@
and schwarze@.  Ingo also helped with some of the mandoc formatting.

OK schwarze@ (as far as it is a good starting point and the code
snippets look ok)

1 files changed, 417 insertions, 185 deletions

diff --git a/src/lib/libc/stdlib/malloc.3 b/src/lib/libc/stdlib/malloc.3
index d738524c81..4185648c84 100644
--- a/src/lib/libc/stdlib/malloc.3
+++ b/src/lib/libc/stdlib/malloc.3
@@ -30,9 +30,9 @@
 .\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
 .\" SUCH DAMAGE.
 .\"
-.\"     $OpenBSD: malloc.3,v 1.78 2014/05/01 18:41:59 jmc Exp $
+.\"     $OpenBSD: malloc.3,v 1.79 2014/10/19 17:58:14 doug Exp $
 .\"
-.Dd $Mdocdate: May 1 2014 $
+.Dd $Mdocdate: October 19 2014 $
 .Dt MALLOC 3
 .Os
 .Sh NAME
@@ -65,58 +65,15 @@ The
 function allocates uninitialized space for an object whose
 size is specified by
 .Fa size .
-The
 .Fn malloc
-function maintains multiple lists of free blocks according to size, allocating
+maintains multiple lists of free blocks according to size, allocating
 space from the appropriate list.
-.Pp
-The allocated space is
-suitably aligned (after possible pointer
-coercion) for storage of any type of object.
+The allocated space is suitably aligned (after possible pointer coercion) for
+storage of any type of object.
 If the space is of
 .Em pagesize
 or larger, the memory returned will be page-aligned.
 .Pp
-Allocation of a zero size object returns a pointer to a zero size object.
-This zero size object is access protected, so any access to it will
-generate an exception (SIGSEGV).
-Many zero-sized objects can be placed consecutively in shared
-protected pages.
-The minimum size of the protection on each object is suitably aligned and
-sized as previously stated, but the protection may extend further depending
-on where in a protected zone the object lands.
-.Pp
-When using
-.Fn malloc
-be careful to avoid the following idiom:
-.Bd -literal -offset indent
-if ((p = malloc(num * size)) == NULL)
-        err(1, "malloc");
-.Ed
-.Pp
-The multiplication may lead to an integer overflow, which can
-be avoided using the extension
-.Fn reallocarray ,
-as follows:
-.Bd -literal -offset indent
-if ((p = reallocarray(NULL, num, size)) == NULL)
-        err(1, "malloc");
-.Ed
-.Pp
-Alternatively
-.Fn calloc
-is a more portable solution which comes with the cost of clearing memory.
-.Pp
-If
-.Fn malloc
-must be used, be sure to test for overflow:
-.Bd -literal -offset indent
-if (size && num > SIZE_MAX / size) {
-        errno = ENOMEM;
-        err(1, "overflow");
-}
-.Ed
-.Pp
 The
 .Fn calloc
 function allocates space for an array of
@@ -124,12 +81,35 @@ function allocates space for an array of
 objects, each of whose size is
 .Fa size .
 The space is initialized to zero.
-The use of
+.Pp
+The
+.Fn realloc
+function changes the size of the object pointed to by
+.Fa ptr
+to
+.Fa size
+bytes and returns a pointer to the (possibly moved) object.
+The contents of the object are unchanged up to the lesser
+of the new and old sizes.
+If the new size is larger, the value of the newly allocated portion
+of the object is indeterminate and uninitialized.
+If the space cannot be allocated, the object
+pointed to by
+.Fa ptr
+is unchanged.
+.Pp
+The
 .Fn reallocarray
-or
-.Fn calloc
-is strongly encouraged when allocating multiple sized objects
-in order to avoid possible integer overflows.
+function is similar to
+.Fn realloc
+except it operates on
+.Fa nmemb
+members of size
+.Fa size
+and checks for integer overflow in
+.Fa nmemb
+*
+.Fa size .
 .Pp
 The
 .Fn free
@@ -140,7 +120,17 @@ allocation or, if required, to be returned to the kernel using
 .Xr munmap 2 .
 If
 .Fa ptr
-is a null pointer, no action occurs.
+is a
+.Dv NULL
+pointer, no action occurs.
+If
+.Fa ptr
+was previously freed by
+.Fn free
+.Fn realloc ,
+or
+.Fn reallocarray ,
+the behavior is undefined and the double free is a security concern.
 .Pp
 A
 .Fn cfree
@@ -148,38 +138,135 @@ function is also provided for compatibility with old systems and other
 .Nm malloc
 libraries; it is simply an alias for
 .Fn free .
+.Sh RETURN VALUES
+If
+.Fn malloc ,
+.Fn calloc ,
+.Fn realloc ,
+or
+.Fn reallocarray
+is called with
+.Fa size
+or
+.Fa nmemb
+is equal to 0,
+a pointer to an access protected, zero sized object is returned.
+.Pp
+If
+.Fn malloc
+is called with
+.Fa size
+greater than 0, it returns a pointer to the allocated space if successful;
+otherwise, a
+.Dv NULL
+pointer is returned and
+.Va errno
+is set to
+.Er ENOMEM .
 .Pp
 The
-.Fn realloc
-function changes the size of the object pointed to by
-.Fa ptr
-to
+.Fn calloc
+function checks for integer overflow and returns
+.Dv NULL
+if
+.Fa nmemb
+*
 .Fa size
-bytes and returns a pointer to the (possibly moved) object.
-The contents of the object are unchanged up to the lesser
-of the new and old sizes.
-If the new size is larger, the value of the newly allocated portion
-of the object is indeterminate and uninitialized.
+will result in integer overflow.
 If
-.Fa ptr
-is a null pointer, the
+.Fa nmemb
+and
+.Fa size
+are greater than 0,
+.Fn calloc
+returns a pointer to the allocated space if successful; otherwise, a
+.Dv NULL
+pointer is returned and
+.Va errno
+is set to
+.Er ENOMEM .
+.Pp
+The
 .Fn realloc
-function behaves like the
+function behaves like
 .Fn malloc
-function for the specified size.
-If the space cannot be allocated, the object
-pointed to by
+for the specified
+.Fa size
+when
 .Fa ptr
-is unchanged.
+is
+.Dv NULL .
 If
 .Fa size
-is zero and
-.Fa ptr
-is not a null pointer, the object it points to is freed and a new zero size
-object is returned.
+is greater than 0,
+.Fn realloc
+returns a pointer to the allocated space if successful; otherwise, a
+.Dv NULL
+pointer is returned and
+.Va errno
+is set to
+.Er ENOMEM .
+.Pp
+The
+.Fn reallocarray
+function checks for integer overflow and returns
+.Dv NULL
+if
+.Fa nmemb
+*
+.Fa size
+will result in integer overflow.
+If
+.Fn reallocarray
+is called with
+.Fa size
+and
+.Fa nmemb
+greater than 0, it returns a pointer to the allocated space if successful;
+otherwise, a
+.Dv NULL
+pointer is returned and
+.Va errno
+is set to
+.Er ENOMEM .
+.Pp
+The
+.Fn free
+and
+.Fn cfree
+functions return no value.
+.Sh IDIOMS
+Consider
+.Fn calloc
+or the extension
+.Fn reallocarray
+when you have multiplication in the
+.Fa size
+argument of
+.Fn malloc
+or
+.Fn realloc .
+For example, avoid this common idiom as it may lead to integer overflow:
+.Bd -literal -offset indent
+if ((p = malloc(num * size)) == NULL)
+        err(1, "malloc");
+.Ed
+.Pp
+A drop-in replacement is the
+.Ox
+extension
+.Fn reallocarray :
+.Bd -literal -offset indent
+if ((p = reallocarray(NULL, num, size)) == NULL)
+        err(1, "reallocarray");
+.Ed
+.Pp
+Alternatively,
+.Fn calloc
+may be used at the cost of initialization overhead.
 .Pp
 When using
-.Fn realloc
+.Fn realloc ,
 be careful to avoid the following idiom:
 .Bd -literal -offset indent
 size += 50;
@@ -208,13 +295,187 @@ size = newsize;
 .Ed
 .Pp
 As with
-.Fn malloc
+.Fn malloc ,
 it is important to ensure the new size value will not overflow;
 i.e. avoid allocations like the following:
 .Bd -literal -offset indent
 if ((newp = realloc(p, num * size)) == NULL) {
         ...
 .Ed
+.Pp
+Instead, use
+.Fn reallocarray :
+.Bd -literal -offset indent
+if ((newp = reallocarray(p, num, size)) == NULL) {
+        ...
+.Ed
+.Pp
+Code designed for some ancient platforms avoided calling
+.Fn realloc
+with a
+.Dv NULL
+.Fa ptr .
+Such hacks are no longer necessary in modern code.  Instead of
+this idiom:
+.Bd -literal -offset indent
+if (p == NULL)
+        newp = malloc(newsize);
+else
+        newp = realloc(p, newsize);
+.Ed
+.Pp
+Use the following as calling
+.Fn realloc
+with
+.Dv NULL
+is equivalent to calling
+.Fn malloc :
+.Bd -literal -offset indent
+newp = realloc(p, newsize);
+.Ed
+.Sh ENVIRONMENT
+.Bl -tag -width Ev
+.It Ev MALLOC_OPTIONS
+See below.
+.El
+.Sh FILES
+.Bl -tag -width "/etc/malloc.conf"
+.It Pa /etc/malloc.conf
+symbolic link to filename containing option flags
+.El
+.Sh EXAMPLES
+If
+.Fn malloc
+must be used with multiplication, be sure to test for overflow:
+.Bd -literal -offset indent
+size_t size;
+size_t num;
+\&...
+
+/* Check for size_t overflow */
+if (size && num > SIZE_MAX / size) {
+        errno = EOVERFLOW;
+        err(1, "overflow");
+}
+if ((p = malloc(size * num)) == NULL)
+        err(1, "malloc");
+.Ed
+.Pp
+The above test is not sufficient in all cases.  For example, multiplying
+ints requires a different set of checks:
+.Bd -literal -offset indent
+int size;
+int num;
+\&...
+
+/* Avoid invalid requests */
+if (size < 0 || num < 0) {
+        errno = EOVERFLOW;
+        err(1, "overflow");
+}
+
+/* Check for signed int overflow */
+if (size && num > INT_MAX / size) {
+        errno = EOVERFLOW;
+        err(1, "overflow");
+}
+
+if ((p = malloc(size * num)) == NULL)
+        err(1, "malloc");
+.Ed
+.Pp
+Assuming the implementation checks for integer overflow as
+.Ox
+does, it is much easier to use
+.Fn calloc
+or
+.Fn reallocarray .
+.Pp
+The above examples could be simplified to:
+.Bd -literal -offset indent
+if ((p = reallocarray(NULL, num, size)) == NULL)
+        err(1, "reallocarray");
+.Ed
+.Pp
+or at the cost of initialization:
+.Bd -literal -offset indent
+if ((p = calloc(num, size)) == NULL)
+        err(1, "calloc");
+.Ed
+.Sh DIAGNOSTICS
+If
+.Fn malloc ,
+.Fn calloc ,
+.Fn realloc ,
+.Fn reallocarray ,
+or
+.Fn free
+detect an error condition,
+a message will be printed to file descriptor
+2 (not using stdio).
+Errors will result in the process being aborted,
+unless the
+.Cm a
+option has been specified.
+.Pp
+Here is a brief description of the error messages and what they mean:
+.Bl -tag -width Ds
+.It Dq out of memory
+If the
+.Cm X
+option is specified it is an error for
+.Fn malloc ,
+.Fn calloc ,
+.Fn realloc ,
+or
+.Fn reallocarray
+to return
+.Dv NULL .
+.It Dq malloc init mmap failed
+This is a rather weird condition that is most likely to indicate a
+seriously overloaded system or a ulimit restriction.
+.It Dq bogus pointer (double free?)
+An attempt to
+.Fn free ,
+.Fn realloc ,
+or
+.Fn reallocarray
+an unallocated pointer was made.
+.It Dq chunk is already free
+There was an attempt to free a chunk that had already been freed.
+.It Dq modified chunk-pointer
+The pointer passed to
+.Fn free ,
+.Fn realloc ,
+or
+.Fn reallocarray
+has been modified.
+.It Dq recursive call
+An attempt was made to call recursively into these functions, i.e., from a
+signal handler.
+This behavior is not supported.
+In particular, signal handlers should
+.Em not
+use any of the
+.Fn malloc
+functions nor utilize any other functions which may call
+.Fn malloc
+(e.g.,
+.Xr stdio 3
+routines).
+.It Dq unknown char in MALLOC_OPTIONS
+We found something we didn't understand.
+.It Dq malloc cache overflow/underflow
+The internal malloc page cache has been corrupted.
+.It Dq malloc free slot lost
+The internal malloc page cache has been corrupted.
+.It Dq guard size
+An inconsistent guard size was detected.
+.It any other error
+.Fn malloc
+detected an internal error;
+consult sources and/or wizards.
+.El
 .Sh MALLOC_OPTIONS
 Malloc will first look for a symbolic link called
 .Pa /etc/malloc.conf
@@ -335,122 +596,22 @@ are used,
 it is buggy.
 .Pp
 The default number of free pages cached is 64.
-.Sh RETURN VALUES
-The
-.Fn malloc ,
-.Fn reallocarray ,
-and
-.Fn calloc
-functions return a pointer to the allocated space if successful; otherwise,
-a null pointer is returned and
-.Va errno
-is set to
-.Er ENOMEM .
-.Pp
-The
-.Fn free
-and
-.Fn cfree
-functions return no value.
-.Pp
-The
-.Fn realloc
-function returns a pointer to the (possibly moved) allocated space
-if successful; otherwise, a null pointer is returned and
-.Va errno
-is set to
-.Er ENOMEM .
-.Sh ENVIRONMENT
-.Bl -tag -width Ev
-.It Ev MALLOC_OPTIONS
-See above.
-.El
-.Sh FILES
-.Bl -tag -width "/etc/malloc.conf"
-.It Pa /etc/malloc.conf
-symbolic link to filename containing option flags
-.El
-.Sh DIAGNOSTICS
-If
-.Fn malloc ,
-.Fn calloc ,
-.Fn realloc ,
-or
-.Fn free
-detect an error condition,
-a message will be printed to file descriptor
-2 (not using stdio).
-Errors will result in the process being aborted,
-unless the
-.Cm a
-option has been specified.
-.Pp
-Here is a brief description of the error messages and what they mean:
-.Bl -tag -width Ds
-.It Dq out of memory
-If the
-.Cm X
-option is specified it is an error for
-.Fn malloc ,
-.Fn calloc ,
-or
-.Fn realloc
-to return
-.Dv NULL .
-.It Dq malloc init mmap failed
-This is a rather weird condition that is most likely to indicate a
-seriously overloaded system or a ulimit restriction.
-.It Dq bogus pointer (double free?)
-An attempt to
-.Fn free
-or
-.Fn realloc
-an unallocated pointer was made.
-.It Dq chunk is already free
-There was an attempt to free a chunk that had already been freed.
-.It Dq modified chunk-pointer
-The pointer passed to
-.Fn free
-or
-.Fn realloc
-has been modified.
-.It Dq recursive call
-An attempt was made to call recursively into these functions, i.e., from a
-signal handler.
-This behavior is not supported.
-In particular, signal handlers should
-.Em not
-use any of the
-.Fn malloc
-functions nor utilize any other functions which may call
-.Fn malloc
-(e.g.,
-.Xr stdio 3
-routines).
-.It Dq unknown char in MALLOC_OPTIONS
-We found something we didn't understand.
-.It Dq malloc cache overflow/underflow
-The internal malloc page cache has been corrupted.
-.It Dq malloc free slot lost
-The internal malloc page cache has been corrupted.
-.It Dq guard size
-An inconsistent guard size was detected.
-.It any other error
-.Fn malloc
-detected an internal error;
-consult sources and/or wizards.
-.El
 .Sh SEE ALSO
 .Xr brk 2 ,
 .Xr mmap 2 ,
 .Xr munmap 2 ,
 .Xr alloca 3 ,
 .Xr getpagesize 3 ,
-.Xr posix_memalign 3
+.Xr posix_memalign 3 ,
+.Xr sysconf 3
 .Sh STANDARDS
 The
-.Fn malloc
-function conforms to
+.Fn malloc ,
+.Fn calloc ,
+.Fn realloc ,
+and
+.Fn free
+functions conform to
 .St -ansiC .
 .Sh HISTORY
 A
@@ -496,6 +657,77 @@ random.
 A rewrite by Otto Moerbeek introducing a new central data structure and more
 randomization appeared in
 .Ox 4.4 .
+.Pp
+The
 .Fn reallocarray
-appeared in
+function appeared in
 .Ox 5.6 .
+.Pp
+The
+.Fn cfree
+function appeared in SunOS 4.x.
+.Sh CAVEATS
+The
+.Fn calloc
+function checks for integer overflow in
+.Ox
+and most other modern platforms.
+Software targeting ancient platforms should not rely on this behavior.
+.Pp
+The
+.Fn malloc ,
+.Fn calloc ,
+and
+.Fn realloc
+functions have implementation defined behavior when
+.Fa size
+or
+.Fa nmemb
+are zero.
+.Pp
+Allocation of a zero size object returns a pointer to an access protected zero
+size object.
+Many zero-sized objects can be placed consecutively in shared
+protected pages.
+The minimum size of the protection on each object is suitably aligned and
+sized as previously stated, but the protection may extend further depending
+on where in a protected zone the object lands.
+Attempting to access these objects will generate a
+.Pq Dv SIGSEGV
+exception.
+.Pp
+When using
+.Fn malloc ,
+be wary of signed integer and
+.Vt size_t
+overflow especially when you
+have multiplication in the
+.Fa size
+argument.
+.Pp
+Signed integer overflow will cause undefined behavior which compilers
+typically handle by wrapping back around to negative numbers.
+Depending on the input, this can result in allocating more or less
+memory than you intended.
+.Pp
+An unsigned overflow has defined behavior which will wrap back around and you
+will receive less memory than you intended.
+.Pp
+A signed or unsigned integer overflow is a
+.Em security
+risk if you end up allocating less memory than you intended.
+Your code may corrupt the heap by writing beyond the memory that you
+were allocated.
+An attacker may be able to leverage this heap corruption to convince your
+program to execute arbitrary code.
+.Pp
+Consider using
+.Fn calloc
+or
+.Fn reallocarray
+instead of using multiplication in
+.Fn malloc
+and
+.Fn realloc
+to avoid these problems on
+.Ox .