Send illegal parameter alerts for various HelloRetryRequest violations.

Be more RFC compliant and send illegal parameter alerts when the client
receives a HelloRetryRequest that requests a group that we did not offer
or a group that we sent a key share for in the ClientHello. These were
annotated as missing, but not previously implemented.

Prompted by a report from the tlspuffin team.

ok tb@

1 files changed, 10 insertions, 6 deletions

diff --git a/src/lib/libssl/tls13_client.c b/src/lib/libssl/tls13_client.c
index 21d3960796..d650717856 100644
--- a/src/lib/libssl/tls13_client.c
+++ b/src/lib/libssl/tls13_client.c
@@ -1,4 +1,4 @@
-/* $OpenBSD: tls13_client.c,v 1.106 2025/12/04 21:16:17 beck Exp $ */
+/* $OpenBSD: tls13_client.c,v 1.107 2026/06/14 14:53:07 jsing Exp $ */
 /*
  * Copyright (c) 2018, 2019 Joel Sing <jsing@openbsd.org>
  *
@@ -450,12 +450,16 @@ tls13_client_hello_retry_send(struct tls13_ctx *ctx, CBB *cbb)
         /*
          * Ensure that the server supported group is one that we listed in our
          * supported groups and is not the same as the key share we previously
-         * offered.
+         * offered. See RFC 8446 section 4.2.8.
          */
-        if (!tls1_check_group(ctx->ssl, ctx->hs->tls13.server_group))
-                return 0; /* XXX alert */
-        if (ctx->hs->tls13.server_group == tls_key_share_group(ctx->hs->key_share))
-                return 0; /* XXX alert */
+        if (!tls1_check_group(ctx->ssl, ctx->hs->tls13.server_group)) {
+                ctx->alert = TLS13_ALERT_ILLEGAL_PARAMETER;
+                return 0;
+        }
+        if (ctx->hs->tls13.server_group == tls_key_share_group(ctx->hs->key_share)) {
+                ctx->alert = TLS13_ALERT_ILLEGAL_PARAMETER;
+                return 0;
+        }
 
         /* Switch to new key share. */
         tls_key_share_free(ctx->hs->key_share);