Do not ignore SSL_TLSEXT_ERR_FATAL from the ALPN callback

As reported by Jeremy Harris, we inherited a strange behavior from
OpenSSL, in that we ignore the SSL_TLSEXT_ERR_FATAL return from the
ALPN callback. RFC 7301, 3.2 states: 'In the event that the server
supports no protocols that the client advertises, then the server
SHALL respond with a fatal "no_application_protocol" alert.'

Honor this requirement and succeed only on SSL_TLSEXT_ERR_{OK,NOACK}
which is the current behavior of OpenSSL. The documentation change
is taken from OpenSSL 1.1.1 as well.

As pointed out by jsing, there is more to be fixed here:
- ensure that the same protocol is selected on session resumption
- should the callback be called even if no ALPN extension was sent?
- ensure for TLSv1.2 and earlier that the SNI has already been processed

ok beck jsing

1 files changed, 2 insertions, 1 deletions

diff --git a/src/lib/libssl/ssl_err.c b/src/lib/libssl/ssl_err.c
index 8507690f79..9ea7cd469a 100644
--- a/src/lib/libssl/ssl_err.c
+++ b/src/lib/libssl/ssl_err.c
@@ -1,4 +1,4 @@
-/* $OpenBSD: ssl_err.c,v 1.38 2021/05/16 08:24:21 jsing Exp $ */
+/* $OpenBSD: ssl_err.c,v 1.39 2021/09/10 09:25:29 tb Exp $ */
 /* ====================================================================
  * Copyright (c) 1999-2011 The OpenSSL Project.  All rights reserved.
  *
@@ -294,6 +294,7 @@ static ERR_STRING_DATA SSL_str_reasons[]= {
         {ERR_REASON(SSL_R_MISSING_VERIFY_MESSAGE), "missing verify message"},
         {ERR_REASON(SSL_R_MULTIPLE_SGC_RESTARTS) , "multiple sgc restarts"},
         {ERR_REASON(SSL_R_NON_SSLV2_INITIAL_PACKET), "non sslv2 initial packet"},
+        {ERR_REASON(SSL_R_NO_APPLICATION_PROTOCOL), "no application protocol"},
         {ERR_REASON(SSL_R_NO_CERTIFICATES_RETURNED), "no certificates returned"},
         {ERR_REASON(SSL_R_NO_CERTIFICATE_ASSIGNED), "no certificate assigned"},
         {ERR_REASON(SSL_R_NO_CERTIFICATE_RETURNED), "no certificate returned"},