Improve SNI hostname validation.

For some time now we've validated the hostname provided to the server in
the SNI extension. Per RFC 6066, an IP literal is invalid as a hostname -
the current code rejects IPv6 literals, but allows IPv4 literals through.

Improve this check to explicitly detect both IPv4 and IPv6 literals. Some
software has been historically known to include IP literals in SNI, so
rather than rejecting this outright (and failing with a decode error),
pretend that the SNI extension does not exist (such that we do not break
some older clients).

ok inoguchi@ tb@

1 files changed, 2 insertions, 2 deletions

diff --git a/src/lib/libssl/ssl_tlsext.h b/src/lib/libssl/ssl_tlsext.h
index 8e0742aa2c..b4c135fdf1 100644
--- a/src/lib/libssl/ssl_tlsext.h
+++ b/src/lib/libssl/ssl_tlsext.h
@@ -1,4 +1,4 @@
-/* $OpenBSD: ssl_tlsext.h,v 1.26 2020/10/11 01:13:04 guenther Exp $ */
+/* $OpenBSD: ssl_tlsext.h,v 1.27 2021/11/01 16:37:17 jsing Exp $ */
 /*
  * Copyright (c) 2016, 2017 Joel Sing <jsing@openbsd.org>
  * Copyright (c) 2017 Doug Hogan <doug@openbsd.org>
@@ -60,7 +60,7 @@ int tlsext_sni_client_parse(SSL *s, uint16_t msg_type, CBS *cbs, int *alert);
 int tlsext_sni_server_needs(SSL *s, uint16_t msg_type);
 int tlsext_sni_server_build(SSL *s, uint16_t msg_type, CBB *cbb);
 int tlsext_sni_server_parse(SSL *s, uint16_t msg_type, CBS *cbs, int *alert);
-int tlsext_sni_is_valid_hostname(CBS *cbs);
+int tlsext_sni_is_valid_hostname(CBS *cbs, int *is_ip);
 
 int tlsext_supportedgroups_client_needs(SSL *s, uint16_t msg_type);
 int tlsext_supportedgroups_client_build(SSL *s, uint16_t msg_type, CBB *cbb);