regresshion.txt

Qualys Security Advisory

regreSSHion: RCE in OpenSSH's server, on glibc-based Linux systems
(CVE-2024-6387)


========================================================================
Contents
========================================================================

Summary
SSH-2.0-OpenSSH_3.4p1 Debian 1:3.4p1-1.woody.3 (Debian 3.0r6, from 2005)
- Theory
- Practice
- Timing
SSH-2.0-OpenSSH_4.2p1 Debian-7ubuntu3 (Ubuntu 6.06.1, from 2006)
- Theory, take one
- Theory, take two
- Practice
- Timing
SSH-2.0-OpenSSH_9.2p1 Debian-2+deb12u2 (Debian 12.5.0, from 2024)
- Theory
- Practice
- Timing
Towards an amd64 exploit
Patches and mitigation
Acknowledgments
Timeline


========================================================================
Summary
========================================================================

    All it takes is a leap of faith
        -- The Interrupters, "Leap of Faith"

Preliminary note: OpenSSH is one of the most secure software in the
world; this vulnerability is one slip-up in an otherwise near-flawless
implementation. Its defense-in-depth design and code are a model and an
inspiration, and we thank OpenSSH's developers for their exemplary work.

We discovered a vulnerability (a signal handler race condition) in
OpenSSH's server (sshd): if a client does not authenticate within
LoginGraceTime seconds (120 by default, 600 in old OpenSSH versions),
then sshd's SIGALRM handler is called asynchronously, but this signal
handler calls various functions that are not async-signal-safe (for
example, syslog()). This race condition affects sshd in its default
configuration.

On investigation, we realized that this vulnerability is in fact a
regression of CVE-2006-5051 ("Signal handler race condition in OpenSSH
before 4.4 allows remote attackers to cause a denial of service (crash),
and possibly execute arbitrary code"), which was reported in 2006 by
Mark Dowd.

This regression was introduced in October 2020 (OpenSSH 8.5p1) by commit
752250c ("revised log infrastructure for OpenSSH"), which accidentally
removed an "#ifdef DO_LOG_SAFE_IN_SIGHAND" from sigdie(), a function
that is directly called by sshd's SIGALRM handler. In other words:

- OpenSSH < 4.4p1 is vulnerable to this signal handler race condition,
  if not backport-patched against CVE-2006-5051, or not patched against
  CVE-2008-4109, which was an incorrect fix for CVE-2006-5051;

- 4.4p1 <= OpenSSH < 8.5p1 is not vulnerable to this signal handler race
  condition (because the "#ifdef DO_LOG_SAFE_IN_SIGHAND" that was added
  to sigdie() by the patch for CVE-2006-5051 transformed this unsafe
  function into a safe _exit(1) call);

- 8.5p1 <= OpenSSH < 9.8p1 is vulnerable again to this signal handler
  race condition (because the "#ifdef DO_LOG_SAFE_IN_SIGHAND" was
  accidentally removed from sigdie()).

This vulnerability is exploitable remotely on glibc-based Linux systems,
where syslog() itself calls async-signal-unsafe functions (for example,
malloc() and free()): an unauthenticated remote code execution as root,
because it affects sshd's privileged code, which is not sandboxed and
runs with full privileges. We have not investigated any other libc or
operating system; but OpenBSD is notably not vulnerable, because its
SIGALRM handler calls syslog_r(), an async-signal-safer version of
syslog() that was invented by OpenBSD in 2001.

To exploit this vulnerability remotely (to the best of our knowledge,
CVE-2006-5051 has never been successfully exploited before), we drew
inspiration from a visionary paper, "Delivering Signals for Fun and
Profit", which was published in 2001 by Michal Zalewski:

  https://lcamtuf.coredump.cx/signals.txt

Nevertheless, we immediately faced three major problems:

- From a theoretical point of view, we must find a useful code path
  that, if interrupted at the right time by SIGALRM, leaves sshd in an
  inconsistent state, and we must then exploit this inconsistent state
  inside the SIGALRM handler.

- From a practical point of view, we must find a way to reach this
  useful code path in sshd, and maximize our chances of interrupting it
  at the right time.

- From a timing point of view, we must find a way to further increase
  our chances of interrupting this useful code path at the right time,
  remotely.

To focus on these three problems without having to immediately fight
against all the modern operating system protections (in particular, ASLR
and NX), we decided to exploit old OpenSSH versions first, on i386, and
then, based on this experience, recent versions:

- First, "SSH-2.0-OpenSSH_3.4p1 Debian 1:3.4p1-1.woody.3", from
  "debian-30r6-dvd-i386-binary-1_NONUS.iso": this is the first Debian
  version that has privilege separation enabled by default and that is
  patched against all the critical vulnerabilities of that era (in
  particular, CVE-2003-0693 and CVE-2002-0640).

  To remotely exploit this version, we interrupt a call to free() with
  SIGALRM (inside sshd's public-key parsing code), leave the heap in an
  inconsistent state, and exploit this inconsistent state during another
  call to free(), inside the SIGALRM handler.

  In our experiments, it takes ~10,000 tries on average to win this race
  condition; i.e., with 10 connections (MaxStartups) accepted per 600
  seconds (LoginGraceTime), it takes ~1 week on average to obtain a
  remote root shell.

- Second, "SSH-2.0-OpenSSH_4.2p1 Debian-7ubuntu3", from
  "ubuntu-6.06.1-server-i386.iso": this is the last Ubuntu version that
  is still vulnerable to CVE-2006-5051 ("Signal handler race condition
  in OpenSSH before 4.4").

  To remotely exploit this version, we interrupt a call to pam_start()
  with SIGALRM, leave one of PAM's structures in an inconsistent state,
  and exploit this inconsistent state during a call to pam_end(), inside
  the SIGALRM handler.

  In our experiments, it takes ~10,000 tries on average to win this race
  condition; i.e., with 10 connections (MaxStartups) accepted per 120
  seconds (LoginGraceTime), it takes ~1-2 days on average to obtain a
  remote root shell.

- Finally, "SSH-2.0-OpenSSH_9.2p1 Debian-2+deb12u2", from
  "debian-12.5.0-i386-DVD-1.iso": this is the current Debian stable
  version, and it is vulnerable to the regression of CVE-2006-5051.

  To remotely exploit this version, we interrupt a call to malloc() with
  SIGALRM (inside sshd's public-key parsing code), leave the heap in an
  inconsistent state, and exploit this inconsistent state during another
  call to malloc(), inside the SIGALRM handler (more precisely, inside
  syslog()).

  In our experiments, it takes ~10,000 tries on average to win this race
  condition, so ~3-4 hours with 100 connections (MaxStartups) accepted
  per 120 seconds (LoginGraceTime). Ultimately, it takes ~6-8 hours on
  average to obtain a remote root shell, because we can only guess the
  glibc's address correctly half of the time (because of ASLR).

This research is still a work in progress:

- we have targeted virtual machines only, not bare-metal servers, on a
  mostly stable network link (~10ms packet jitter);

- we are convinced that various aspects of our exploits can be greatly
  improved;

- we have started to work on an amd64 exploit, which is much harder
  because of the stronger ASLR.

A few days after we started our work on amd64, we noticed the following
bug report (in OpenSSH's public Bugzilla), about a deadlock in sshd's
SIGALRM handler:

  https://bugzilla.mindrot.org/show_bug.cgi?id=3690

We therefore decided to contact OpenSSH's developers immediately (to let
them know that this deadlock is caused by an exploitable vulnerability),
we put our amd64 work on hold, and we started to write this advisory.


========================================================================
SSH-2.0-OpenSSH_3.4p1 Debian 1:3.4p1-1.woody.3 (Debian 3.0r6, from 2005)
========================================================================

------------------------------------------------------------------------
Theory
------------------------------------------------------------------------

    But that's not like me, I'm breaking free
        -- The Interrupters, "Haven't Seen the Last of Me"

The SIGALRM handler of this OpenSSH version calls packet_close(), which
calls buffer_free(), which calls xfree() and hence free(), which is not
async-signal-safe:

------------------------------------------------------------------------
 302 grace_alarm_handler(int sig)
 303 {
 ...
 307         packet_close();
------------------------------------------------------------------------
 329 packet_close(void)
 330 {
 ...
 341         buffer_free(&input);
 342         buffer_free(&output);
 343         buffer_free(&outgoing_packet);
 344         buffer_free(&incoming_packet);
------------------------------------------------------------------------
 35 buffer_free(Buffer *buffer)
 36 {
 37         memset(buffer->buf, 0, buffer->alloc);
 38         xfree(buffer->buf);
 39 }
------------------------------------------------------------------------
 51 xfree(void *ptr)
 52 {
 53         if (ptr == NULL)
 54                 fatal("xfree: NULL pointer given as argument");
 55         free(ptr);
 56 }
------------------------------------------------------------------------

Consequently, we started to read the malloc code of this Debian's glibc
(2.2.5), to see if a first call to free() can be interrupted by SIGALRM
and exploited during a second call to free() inside the SIGALRM handler
(at lines 341-344, above). Because this glibc's malloc is not hardened
against the unlink() technique pioneered by Solar Designer in 2000, we
quickly spotted an interesting code path in chunk_free() (which is
called internally by free()):

------------------------------------------------------------------------
1028 struct malloc_chunk
1029 {
1030   INTERNAL_SIZE_T prev_size; /* Size of previous chunk (if free). */
1031   INTERNAL_SIZE_T size;      /* Size in bytes, including overhead. */
1032   struct malloc_chunk* fd;   /* double links -- used only if free. */
1033   struct malloc_chunk* bk;
1034 };
------------------------------------------------------------------------
2516 #define unlink(P, BK, FD)                                           \
2517 {                                                                   \
2518   BK = P->bk;                                                       \
2519   FD = P->fd;                                                       \
2520   FD->bk = BK;                                                      \
2521   BK->fd = FD;                                                      \
2522 }                                                                   \
------------------------------------------------------------------------
3160 chunk_free(arena *ar_ptr, mchunkptr p)
....
3164 {
3165   INTERNAL_SIZE_T hd = p->size; /* its head field */
....
3177   sz = hd & ~PREV_INUSE;
3178   next = chunk_at_offset(p, sz);
3179   nextsz = chunksize(next);
....
3230   if (!(inuse_bit_at_offset(next, nextsz)))   /* consolidate forward */
3231   {
....
3241       unlink(next, bck, fwd);
....
3244   }
3245   else
3246     set_head(next, nextsz);                  /* clear inuse bit */
....
3251     frontlink(ar_ptr, p, sz, idx, bck, fwd);
------------------------------------------------------------------------

To exploit this code path, we arrange for sshd's heap to have the
following layout (chunk_X, chunk_Y, and chunk_Z are malloc()ated chunks
of memory, and p, s, f, b are their prev_size, size, fd, and bk fields):

-----|---+---------------|---+---------------|---+---------------|-----
 ... |p|s|f|b|  chunk_X  |p|s|f|b|  chunk_Y  |p|s|f|b|  chunk_Z  | ...
-----|---+---------------|---+---------------|---+---------------|-----
                             |<------------->|
                                 user data

- First, if a call to free(chunk_Y) is interrupted by SIGALRM *after*
  line 3246 but *before* line 3251, then chunk_Y is already marked as
  free (because chunk_Z's PREV_INUSE bit is cleared at line 3246) but it
  is not yet linked into its doubly-linked list (at line 3251): in other
  words, chunk_Y's fd and bk pointers still contain user data (attacker-
  controlled data).

- Second, if (inside the SIGALRM handler) packet_close() calls
  free(chunk_X), then the code block at lines 3230-3244 is entered
  (because chunk_Y is marked as free) and chunk_Y is unlink()ed (at line
  3241): a so-called aa4bmo primitive (almost arbitrary 4 bytes mirrored
  overwrite), because chunk_Y's fd and bk pointers are still attacker-
  controlled. For more information on the unlink() technique and the
  aa4bmo primitive:

  https://www.openwall.com/articles/JPEG-COM-Marker-Vulnerability#exploit
  http://phrack.org/issues/61/6.html#article

- Last, with this aa4bmo primitive we overwrite the glibc's __free_hook
  function pointer (this old Debian version does not have ASLR, nor NX)
  with the address of our shellcode in the heap, thus achieving remote
  code execution during the next call to free() in packet_close().

------------------------------------------------------------------------
Practice
------------------------------------------------------------------------

    Now they're taking over and they got complete control
        -- The Interrupters, "Liberty"

To mount this attack against sshd, we interrupt a call to free() inside
sshd's parsing code of a DSA public key (i.e., line 144 below is our
free(chunk_Y)) and exploit it during one of the free() calls in
packet_close() (i.e., one of the lines 341-344 above is our
free(chunk_X)):

------------------------------------------------------------------------
136 buffer_get_bignum2(Buffer *buffer, BIGNUM *value)
137 {
138         u_int len;
139         u_char *bin = buffer_get_string(buffer, &len);
...
143         BN_bin2bn(bin, len, value);
144         xfree(bin);
145 }
------------------------------------------------------------------------

Initially, however, we were never able to win this race condition (i.e.,
interrupt the free() call at line 144 at the right time). Eventually, we
realized that we could greatly improve our chances of winning this race:
the DSA public-key parsing code allows us to call free() four times (at
lines 704-707 below), and furthermore sshd allows us to attempt six user
authentications (AUTH_FAIL_MAX); if any one of these 24 free() calls is
interrupted at the right time, then we later achieve remote code
execution inside the SIGALRM handler.

------------------------------------------------------------------------
678 key_from_blob(u_char *blob, int blen)
679 {
...
693         switch (type) {
...
702         case KEY_DSA:
703                 key = key_new(type);
704                 buffer_get_bignum2(&b, key->dsa->p);
705                 buffer_get_bignum2(&b, key->dsa->q);
706                 buffer_get_bignum2(&b, key->dsa->g);
707                 buffer_get_bignum2(&b, key->dsa->pub_key);
------------------------------------------------------------------------

With this improvement, we finally won the race condition after ~1 month:
we were happy (and did a root-shell dance), but we also felt that there
was still room for improvement.

------------------------------------------------------------------------
Timing
------------------------------------------------------------------------

    Don't worry, just wait and see
        -- The Interrupters, "Haven't Seen the Last of Me"

We therefore implemented the following threefold timing strategy:

- We do not wait until the last moment to send our (rather large) DSA
  public-key packet to sshd: instead, we send the entire packet minus
  one byte (the last byte) long before the LoginGraceTime, and send the
  very last byte at the very last moment, to minimize the effects of
  network delays. (And we disable the Nagle algorithm.)

- We keep track of the median round-trip time (by regularly sending
  packets that produce a response from sshd), and keep track of the
  difference between the moment we are expecting our connection to be
  closed by sshd (essentially the moment we receive the first byte of
  sshd's banner, plus LoginGraceTime) and the moment our connection is
  really closed by sshd, and accordingly adjust our timing (i.e., the
  moment when we send the last byte of our DSA packet).

  These time differences allow us to track clock skews and network
  delays, which show predictable patterns over time: we experimented
  with linear and spline regressions, but in the end, nothing worked
  better than simply re-using the most recent measurement. Possibly,
  deep learning might yield even better results; this is left as an
  exercise for the interested reader.

- More importantly, we further increase our chances of winning this race
  condition by slowly adjusting our timing through involuntary feedback
  from sshd:

  - if we receive a response (SSH2_MSG_USERAUTH_FAILURE) to our DSA
    public-key packet, then we sent it too early (sshd had the time to
    receive our packet in the unprivileged child, parse it, send it to
    the privileged child, parse it there, and send a response all the
    way back to us);

  - if we cannot even send the last byte of our DSA packet, then we
    waited too long (sshd already received the SIGALRM and closed our
    connection);

  - if we can send the last byte of our DSA packet, and receive no
    response before sshd closes our connection, then our timing was
    reasonably accurate.

  This feedback allows us to target what we call the "large" race
  window: hitting it does not guarantee that we win the race condition,
  but inside this large window are the 24 "small" race windows (inside
  the 24 free() calls) that, if hit, guarantee that we do win the race
  condition.

With these improvements, it takes ~10,000 tries on average to win this
race condition; i.e., with 10 connections (MaxStartups) accepted per 600
seconds (LoginGraceTime), it takes ~1 week on average to obtain a remote
root shell.


========================================================================
SSH-2.0-OpenSSH_4.2p1 Debian-7ubuntu3 (Ubuntu 6.06.1, from 2006)
========================================================================

------------------------------------------------------------------------
Theory, take one
------------------------------------------------------------------------

    I sleep when the sun starts to rise
        -- The Interrupters, "Alien"

The SIGALRM handler of this OpenSSH version does not call packet_close()
anymore; moreover, this Ubuntu's glibc (2.3.6) always takes a mandatory
lock when entering the functions of the malloc family (even if single-
threaded like sshd), which prevents us from interrupting a call to one
of the malloc functions and later exploiting it during another call to
these functions (they would always deadlock). We must find another
solution.

CVE-2006-5051 mentions a double-free in GSSAPI, but GSSAPI (or Kerberos)
is not enabled by default, so this does not sound very appealing. On the
other hand, PAM is enabled by default, and pam_end() is called by sshd's
SIGALRM handler (and is, of course, not async-signal-safe). We therefore
searched for a PAM function that, if interrupted by SIGALRM at the right
time, would leave PAM's internal structures in an inconsistent state,
exploitable during pam_end() in the SIGALRM handler. We found
pam_set_data():

------------------------------------------------------------------------
 33 int pam_set_data(
 34     pam_handle_t *pamh,
 ..
 37     void (*cleanup)(pam_handle_t *pamh, void *data, int error_status))
 38 {
 39     struct pam_data *data_entry;
 ..
 57     } else if ((data_entry = malloc(sizeof(*data_entry)))) {
 ..
 65         data_entry->next = pamh->data;
 66         pamh->data = data_entry;
 ..
 74     data_entry->cleanup = cleanup;
------------------------------------------------------------------------

If this function is interrupted by SIGALRM *after* line 66 but *before*
line 74, then data_entry is already linked into PAM's structures (pamh),
but its cleanup field (a function pointer) is not yet initialized (since
the malloc() at line 57 does not initialize its memory). If we are able
to control cleanup (through leftovers from previous heap allocations),
then we can execute arbitrary code when pam_end() (inside the SIGALRM
handler) calls _pam_free_data() (at line 118):

------------------------------------------------------------------------
104 void _pam_free_data(pam_handle_t *pamh, int status)
105 {
106     struct pam_data *last;
107     struct pam_data *data;
...
112     data = pamh->data;
113 
114     while (data) {
115         last = data;
116         data = data->next;
117         if (last->cleanup) {
118             last->cleanup(pamh, last->data, status);
------------------------------------------------------------------------

This would have been an extremely simple exploit; unfortunately, we
completely overlooked that pam_set_data() can only be called from PAM
modules: if we interrupt it with SIGALRM, then pamh->caller_is is still
_PAM_CALLED_FROM_MODULE, in which case pam_end() returns immediately,
without ever calling _pam_free_data(). Back to the drawing board.

------------------------------------------------------------------------
Theory, take two
------------------------------------------------------------------------

    Not giving up, it's not what we do
        -- The Interrupters, "Title Holder"

We noticed that, at line 601 below, sshd passes a pointer to its global
sshpam_handle pointer directly to pam_start() (which is called once per
connection):

------------------------------------------------------------------------
 202 static pam_handle_t *sshpam_handle = NULL;
------------------------------------------------------------------------
 584 sshpam_init(Authctxt *authctxt)
 585 {
 ...
 600         sshpam_err =
 601             pam_start(SSHD_PAM_SERVICE, user, &store_conv, &sshpam_handle);
------------------------------------------------------------------------

We therefore decided to look into pam_start() itself: if interrupted by
SIGALRM, it might leave the structure pointed to by sshpam_handle in an
inconsistent state, which could then be exploited inside the SIGALRM
handler, when "pam_end(sshpam_handle, sshpam_err)" is called.

------------------------------------------------------------------------
 18 int pam_start (
 ..
 22     pam_handle_t **pamh)
 23 {
 ..
 32     if ((*pamh = calloc(1, sizeof(**pamh))) == NULL) {
...
110     if ( _pam_init_handlers(*pamh) != PAM_SUCCESS ) {
------------------------------------------------------------------------
 319 int _pam_init_handlers(pam_handle_t *pamh)
 320 {
 ...
 398                 retval = _pam_parse_conf_file(pamh, f, pamh->service_name, PAM_T_ANY
------------------------------------------------------------------------
  66 static int _pam_parse_conf_file(pam_handle_t *pamh, FILE *f
  ..
  73 {
 ...
 252             res = _pam_add_handler(pamh, must_fail, other
------------------------------------------------------------------------
 581 int _pam_add_handler(pam_handle_t *pamh
 ...
 585 {
 ...
 755     the_handlers = (other) ? &pamh->handlers.other : &pamh->handlers.conf;
 ...
 767         handler_p = &the_handlers->authenticate;
 ...
 874     if ((*handler_p = malloc(sizeof(struct handler))) == NULL) {
 ...
 886     (*handler_p)->next = NULL;
------------------------------------------------------------------------

At line 32, pam_start() immediately sets sshd's sshpam_handle to a
calloc()ated chunk of memory; this is safe, because calloc() initializes
this memory to zero. On the other hand, if _pam_add_handler() (which is
called multiple times by pam_start()) is interrupted by SIGALRM *after*
line 874 but *before* line 886, then a malloc()ated structure is linked
into pamh, but its next field is not yet initialized. If we are able to
control next (through leftovers from previous heap allocations), then we
can pass an arbitrary pointer to free() during the call to pam_end()
(inside the SIGALRM handler), at line 1020 (and line 1017) below:

------------------------------------------------------------------------
 11 int pam_end(pam_handle_t *pamh, int pam_status)
 12 {
 ..
 31     if ((ret = _pam_free_handlers(pamh)) != PAM_SUCCESS) {
------------------------------------------------------------------------
 925 int _pam_free_handlers(pam_handle_t *pamh)
 926 {
 ...
 954     _pam_free_handlers_aux(&(pamh->handlers.conf.authenticate));
------------------------------------------------------------------------
1009 void _pam_free_handlers_aux(struct handler **hp)
1010 {
1011     struct handler *h = *hp;
1012     struct handler *last;
....
1015     while (h) {
1016         last = h;
1017         _pam_drop(h->argv);  /* This is all alocated in a single chunk */
1018         h = h->next;
1019         memset(last, 0, sizeof(*last));
1020         free(last);
1021     }
------------------------------------------------------------------------

Because the malloc of this Ubuntu's glibc is already hardened against
the old unlink() technique, we decided to transform our arbitrary free()
into the Malloc Maleficarum's House of Mind (fastbin version): we free()
our own NON_MAIN_ARENA chunk, point our fake arena to sshd's .got.plt
(this Ubuntu's sshd has ASLR but not PIE), and overwrite _exit()'s entry
with the address of our shellcode in the heap (this Ubuntu's heap is
still executable by default). For more information on the Malloc
Maleficarum:

  https://seclists.org/bugtraq/2005/Oct/118

------------------------------------------------------------------------
Practice
------------------------------------------------------------------------

    I learned everything the hard way
        -- The Interrupters, "The Hard Way"

To mount this attack against sshd, we initially faced three problems:

- The House of Mind requires us to store the pointer to our fake arena
  at address 0x08100000 in the heap; but are we able to store attacker-
  controlled data at such a high address? Because sshd calls pam_start()
  at the very beginning of the user authentication, we do not control
  anything except the user name itself; luckily, a user name of length
  ~128KB (shorter than DEFAULT_MMAP_THRESHOLD) allows us to store our
  own data at address 0x08100000.

- The size field of our fake NON_MAIN_ARENA chunk must not be too large
  (to pass free()'s security checks); i.e., it must contain null bytes.
  But our long user name is a null-terminated string that cannot contain
  null bytes; luckily we remembered that _pam_free_handlers_aux() zeroes
  the structures that it free()s (line 1019 above): we therefore "patch"
  the size field of our fake chunk with such a memset(0), and only then
  free() it.

- We must survive several calls to free() (at lines 1017 and 1020 above)
  before the free() of our fake NON_MAIN_ARENA chunk. We transform these
  free()s into no-ops by pointing them to fake IS_MMAPPED chunks: free()
  calls munmap_chunk(), which calls munmap(), which fails because these
  fake IS_MMAPPED chunks are misaligned; effectively a no-op, because
  assert()ion failures are not enforced in this Ubuntu's glibc.

Finally, our long user name also allows us to control the potentially
uninitialized next field of 20 different structures (through leftovers
from temporary copies of our long user name), because pam_start() calls
_pam_add_handler() multiple times; i.e., our large race window contains
20 small race windows.

------------------------------------------------------------------------
Timing
------------------------------------------------------------------------

    Same tricks they used before
        -- The Interrupters, "Divide Us"

For this attack against Ubuntu 6.06.1, we simply re-used the timing
strategy that we used against Debian 3.0r6: it takes ~10,000 tries on
average to win the race condition, and with 10 connections (MaxStartups)
accepted per 120 seconds (LoginGraceTime), it takes ~1-2 days on average
to obtain a remote root shell.

Note: because this Ubuntu's glibc always takes a mandatory lock when
entering the functions of the malloc family, an unlucky attacker might
deadlock all 10 MaxStartups connections before obtaining a root shell;
we have not tried to work around this problem because our ultimate goal
was to exploit a modern OpenSSH version anyway.


========================================================================
SSH-2.0-OpenSSH_9.2p1 Debian-2+deb12u2 (Debian 12.5.0, from 2024)
========================================================================

------------------------------------------------------------------------
Theory
------------------------------------------------------------------------

    Now you're ready, take the demons head on
        -- The Interrupters, "Be Gone"

The SIGALRM handler of this OpenSSH version does not call packet_close()
nor pam_end(); in fact it calls only one interesting function, syslog():

------------------------------------------------------------------------
 358 grace_alarm_handler(int sig)
 359 {
 ...
 370         sigdie("Timeout before authentication for %s port %d",
 371             ssh_remote_ipaddr(the_active_state),
 372             ssh_remote_port(the_active_state));
------------------------------------------------------------------------
 96 #define sigdie(...)             sshsigdie(__FILE__, __func__, __LINE__, 0, SYSLOG_LEVEL_ERROR, NULL, __VA_ARGS__)
------------------------------------------------------------------------
451 sshsigdie(const char *file, const char *func, int line, int showfunc,
452     LogLevel level, const char *suffix, const char *fmt, ...)
453 {
...
457         sshlogv(file, func, line, showfunc, SYSLOG_LEVEL_FATAL,
458             suffix, fmt, args);
------------------------------------------------------------------------
464 sshlogv(const char *file, const char *func, int line, int showfunc,
465     LogLevel level, const char *suffix, const char *fmt, va_list args)
466 {
...
489         do_log(level, forced, suffix, fmt2, args);
------------------------------------------------------------------------
337 do_log(LogLevel level, int force, const char *suffix, const char *fmt,
338     va_list args)
339 {
...
419                 syslog(pri, "%.500s", fmtbuf);
------------------------------------------------------------------------

Our two key questions, then, are: Does the syslog() of this Debian's
glibc (2.36) call async-signal-unsafe functions such as malloc() and
free()? And if yes, does this glibc still take a mandatory lock when
entering the functions of the malloc family?

- Luckily for us attackers, the answer to our first question is yes; if,
  and only if, the syslog() inside the SIGALRM handler is the very first
  call to syslog(), then __localtime64_r() (which is called by syslog())
  calls malloc(304) to allocate a FILE structure (at line 166) and calls
  malloc(4096) to allocate an internal read buffer (at line 186):

------------------------------------------------------------------------
 28 __localtime64_r (const __time64_t *t, struct tm *tp)
 29 {
 30   return __tz_convert (*t, 1, tp);
------------------------------------------------------------------------
567 __tz_convert (__time64_t timer, int use_localtime, struct tm *tp)
568 {
...
577   tzset_internal (tp == &_tmbuf && use_localtime);
------------------------------------------------------------------------
367 tzset_internal (int always)
368 {
...
405   __tzfile_read (tz, 0, NULL);
------------------------------------------------------------------------
105 __tzfile_read (const char *file, size_t extra, char **extrap)
106 {
...
109   FILE *f;
...
166   f = fopen (file, "rce");
...
186   if (__builtin_expect (__fread_unlocked ((void *) &tzhead, sizeof (tzhead),
187                                           1, f) != 1, 0)
------------------------------------------------------------------------

  Note: because we do not control anything about these malloc()ations
  (not their order, not their sizes, not their contents), we took the
  "rce" at line 166 as a much-needed good omen.

- And luckily for us, the answer to our second question is no; since
  October 2017, the glibc's malloc functions do not take any lock
  anymore, when single-threaded (like sshd):

  https://sourceware.org/git?p=glibc.git;a=commit;h=a15d53e2de4c7d83bda251469d92a3c7b49a90db
  https://sourceware.org/git?p=glibc.git;a=commit;h=3f6bb8a32e5f5efd78ac08c41e623651cc242a89
  https://sourceware.org/git?p=glibc.git;a=commit;h=905a7725e9157ea522d8ab97b4c8b96aeb23df54

Moreover, this Debian version suffers from the ASLR weakness described
in the following great blog posts (by Justin Miller and Mathias Krause,
respectively):

  https://zolutal.github.io/aslrnt/
  https://grsecurity.net/toolchain_necromancy_past_mistakes_haunting_aslr

Concretely, in the case of sshd on i386, every memory mapping is
randomized normally (sshd's PIE, the heap, most libraries, the stack),
but the glibc itself is always mapped either at address 0xb7200000 or at
address 0xb7400000; in other words, we can correctly guess the glibc's
address half of the time (a small price to pay for defeating ASLR). In
our exploit we assume that the glibc is mapped at address 0xb7400000,
because it is slightly more common than 0xb7200000.

Our next question is: which code paths inside the glibc's malloc
functions, if interrupted by SIGALRM at the right time, leave the heap
in an inconsistent state, exploitable during one of the malloc() calls
inside the SIGALRM handler?

We found several interesting (and surprising!) code paths, but the one
we chose involves only relative sizes, not absolute addresses (unlike
various code paths inside unlink_chunk(), for example); this difference
might prove crucial for a future amd64 exploit. This code path, inside
malloc(), splits a large free chunk (victim) into two smaller chunks;
the first chunk is returned to malloc()'s caller (at line 4345) and the
second chunk (remainder) is linked into an unsorted list of free chunks
(at lines 4324-4327):

------------------------------------------------------------------------
1449 #define set_head(p, s)       ((p)->mchunk_size = (s))
------------------------------------------------------------------------
3765 _int_malloc (mstate av, size_t bytes)
3766 {
....
3798   nb = checked_request2size (bytes);
....
4295               size = chunksize (victim);
....
4300               remainder_size = size - nb;
....
4316                   remainder = chunk_at_offset (victim, nb);
....
4320                   bck = unsorted_chunks (av);
4321                   fwd = bck->fd;
....
4324                   remainder->bk = bck;
4325                   remainder->fd = fwd;
4326                   bck->fd = remainder;
4327                   fwd->bk = remainder;
....
4337                   set_head (victim, nb | PREV_INUSE |
4338                             (av != &main_arena ? NON_MAIN_ARENA : 0));
4339                   set_head (remainder, remainder_size | PREV_INUSE);
....
4343               void *p = chunk2mem (victim);
....
4345               return p;
------------------------------------------------------------------------

- If this code path is interrupted by SIGALRM *after* line 4327 but
  *before* line 4339, then the remainder chunk of this split is already
  linked into the unsorted list of free chunks (lines 4324-4327), but
  its size field (mchunk_size) is not yet initialized (line 4339).

- If we are able to control its size field (through leftovers from
  previous heap allocations), then we can make this remainder chunk
  larger and overlap with other heap chunks, and therefore corrupt heap
  memory when this enlarged, overlapping remainder chunk is eventually
  malloc()ated and written to (inside the SIGALRM handler).

Our last question, then, is: given that we do not control anything about
the malloc() calls inside the SIGALRM handler, what can we overwrite in
the heap to achieve arbitrary code execution before sshd calls _exit()
(in sshsigdie())?

Because __tzfile_read() (inside the SIGALRM handler) malloc()ates a FILE
structure in the heap (at line 166 above), and because FILE structures
have a long history of abuse for arbitrary code execution, we decided to
aim our heap corruption at this FILE structure. This is, however, easier
said than done: our heap corruption is very limited, and FILE structures
have been significantly hardened over the years (by IO_validate_vtable()
and PTR_DEMANGLE(), for example).

Eventually, we devised the following technique (which seems to be
specific to the i386 glibc -- the amd64 glibc does not seem to use
_vtable_offset at all):

- with our limited heap corruption, we overwrite the _vtable_offset
  field (a single signed char) of __tzfile_read()'s FILE structure;

- the glibc's libio functions will therefore look for this FILE
  structure's vtable pointer (a pointer to an array of function
  pointers) at a non-zero offset (our overwritten _vtable_offset),
  instead of the default zero offset;

- we (attackers) can easily control this fake vtable pointer (through
  leftovers from previous heap allocations), because the FILE structure
  around this offset is not explicitly initialized by fopen();

- to pass the glibc's security checks, our fake vtable pointer must
  point somewhere into the __libc_IO_vtables section: we decided to
  point it to the vtable for wide-character streams, _IO_wfile_jumps
  (i.e., to 0xb761b740, since we assume that the glibc is mapped at
  address 0xb7400000);

- as a result, __fread_unlocked() (at line 186 above) calls
  _IO_wfile_underflow() (instead of _IO_file_underflow()), which calls a
  function pointer (__fct) that basically comes from a structure whose
  pointer (_codecvt) is yet another field of the FILE structure;

- we (attackers) can easily control this _codecvt pointer (through
  leftovers from previous heap allocations, because this field of the
  FILE structure is not explicitly initialized by fopen()), which also
  allows us to control the __fct function pointer.

In summary, by overwriting a single byte (_vtable_offset) of the FILE
structure malloc()ated by fopen(), we can call our own __fct function
pointer and execute arbitrary code during __fread_unlocked().

------------------------------------------------------------------------
Practice
------------------------------------------------------------------------

    I wanted it perfect, no wrinkles in it
        -- The Interrupters, "In the Mirror"

To mount this attack against sshd's privileged child, let us first
imagine the following heap layout (the "XXX"s are "barrier" chunks that
allow us to make holes in the heap; for example, small memory-leaked
chunks):

---|----------------------------------------------|---|------------|---
XXX|                  large hole                  |XXX| small hole |XXX
---|----------------------------------------------|---|------------|---
   |                     ~8KB                     |   |    320B    |

- shortly before sshd receives the SIGALRM, we malloc()ate a ~4KB chunk
  that splits the large ~8KB hole into two smaller chunks:

---|-----------------------|----------------------|---|------------|---
XXX| large allocated chunk | free remainder chunk |XXX| small hole |XXX
---|-----------------------|----------------------|---|------------|---
   |         ~4KB          |         ~4KB         |   |    320B    |

- but if this malloc() is interrupted by SIGALRM *after* line 4327 but
  *before* line 4339, then the remainder chunk of this split is already
  linked into the unsorted list of free chunks, but its size field is
  under our control (through leftovers from previous heap allocations),
  and this artificially enlarged remainder chunk overlaps with the
  following small hole:

---|-----------------------|----------------------|---|------------|---
XXX| large allocated chunk | real remainder chunk |XXX| small hole |XXX
---|-----------------------|----------------------|---|------------|---
   |         ~4KB          |<------------------------------------->|
                             artificially enlarged remainder chunk

- when the SIGALRM handler calls syslog() and hence __tzfile_read(),
  fopen() malloc()ates the small hole for its FILE structure, and
  __fread_unlocked() malloc()ates a 4KB read buffer, thereby splitting
  the enlarged remainder chunk in two (the 4KB read buffer and a small
  remainder chunk):

---|-----------------------|----------------------|---|------------|---
XXX| large allocated chunk |                      |XXX|    FILE    |XXX
---|-----------------------|----------------------|---|--|---------|---
   |         ~4KB          |<--------------------------->|<------->|
                                   4KB read buffer        remainder

- we therefore overwrite parts of the FILE structure with the internal
  header of this small remainder chunk: more precisely, we overwrite the
  FILE's _vtable_offset with the third byte of this header's bk field,
  which is a pointer to the unsorted list of free chunks, 0xb761d7f8
  (i.e., we overwrite _vtable_offset with 0x61);

- then, as explained in the "Theory" subsection, __fread_unlocked()
  calls _IO_wfile_underflow() (instead of _IO_file_underflow()), which
  calls our own __fct function pointer (through our own _codecvt
  pointer) and executes our arbitrary code.

  Note: we have not yet explained how to reliably go from a controlled
  _codecvt pointer to a controlled __fct function pointer; we will do
  so, but we must first solve a more pressing problem.

Indeed, we learned from our work on older OpenSSH versions that we will
never win this signal handler race condition if our large race window
contains only one small race window. Consequently, we implemented the
following strategy, based on the following heap layout:

---|------------|---|------------|---|------------|---|------------|---
XXX|large hole 1|XXX|small hole 1|XXX|large hole 2|XXX|small hole 2|...
---|------------|---|------------|---|------------|---|------------|---
   |    ~8KB    |   |    320B    |   |    ~8KB    |   |    320B    |

The last packet that we send to sshd (shortly before the delivery of
SIGALRM) forces sshd to perform the following sequence of malloc()
calls: malloc(~4KB), malloc(304), malloc(~4KB), malloc(304), etc.

1/ Our first malloc(~4KB) splits the large hole 1 in two:

- if this first split is interrupted by SIGALRM at the right time, then
  the fopen() inside the SIGALRM handler malloc()ates the small hole 1
  for its FILE structure, and we achieve arbitrary code execution as
  explained above;

- if not, then we malloc()ate the small hole 1 ourselves with our first
  malloc(304), and:

2/ Our second malloc(~4KB) splits the large hole 2 in two:

- if this second split is interrupted by SIGALRM at the right time, then
  the fopen() inside the SIGALRM handler malloc()ates the small hole 2
  for its FILE structure, and we achieve arbitrary code execution as
  explained above;

- if not, then we malloc()ate the small hole 2 ourselves with our second
  malloc(304), etc.

We were able to make 27 pairs of such large and small holes in sshd's
heap (28 would exceed PACKET_MAX_SIZE, 256KB): our large race window now
contains 27 small race windows! Achieving this complex heap layout was
extremely painful and time-consuming, but the two highlights are:

- We abuse sshd's public-key parsing code to perform arbitrary sequences
  of malloc() and free() calls (at lines 1805 and 573):

------------------------------------------------------------------------
1754 cert_parse(struct sshbuf *b, struct sshkey *key, struct sshbuf *certbuf)
1755 {
....
1797         while (sshbuf_len(principals) > 0) {
....
1805                 if ((ret = sshbuf_get_cstring(principals, &principal,
....
1820                 key->cert->principals[key->cert->nprincipals++] = principal;
1821         }
------------------------------------------------------------------------
 562 cert_free(struct sshkey_cert *cert)
 563 {
 ...
 572         for (i = 0; i < cert->nprincipals; i++)
 573                 free(cert->principals[i]);
------------------------------------------------------------------------

- We were unable to find a memory leak for our small "barrier" chunks;
  instead, we use tcache chunks (which are never really freed, because
  their inuse bit is never cleared) as makeshift "barrier" chunks.

To reliably achieve this heap layout, we send five different public-key
packets to sshd (packets a/ to d/ can be sent long before SIGALRM; most
of packet e/ can also be sent long before SIGALRM, but its very last
byte must be sent at the very last moment):

a/ We malloc()ate and free() a variety of tcache chunks, to ensure that
the heap allocations that we do not control end up in these tcache
chunks and do not interfere with our careful heap layout.

b/ We malloc()ate and free() chunks of various sizes, to make our 27
pairs of large and small holes (and the corresponding "barrier" chunks).

c/ We malloc()ate and free() ~4KB chunks and 320B chunks, to:

- write the fake header (the large size field) of our potentially
  enlarged remainder chunk, into the middle of our large holes;

- write the fake footer of our potentially enlarged remainder chunk, to
  the end of our small holes (to pass the glibc's security checks);

- write our fake vtable and _codecvt pointers, into our small holes
  (which are potential FILE structures).

d/ We malloc()ate and free() one very large string (nearly 256KB), to
ensure that our large and small holes are removed from the unsorted list
of free chunks and placed into their respective malloc bins.

e/ We force sshd to perform our final sequence of malloc() calls
(malloc(~4KB), malloc(304), malloc(~4KB), malloc(304), etc), to open our
27 small race windows.

Attentive readers may have noticed that we have still not addressed
(literally and figuratively) the problem of _codecvt. In fact, _codecvt
is a pointer to a structure (_IO_codecvt) that contains a pointer to a
structure (__gconv_step) that contains the __fct function pointer that
allows us to execute arbitrary code. To reliably control __fct through
_codecvt, we simply point _codecvt to one of the glibc's malloc bins,
which conveniently contains a pointer to one of our free chunks in the
heap, which contains our own __fct function pointer to arbitrary glibc
code (all of these glibc addresses are known to us, because we assume
that the glibc is mapped at address 0xb7400000).

------------------------------------------------------------------------
Timing
------------------------------------------------------------------------

    We're running out of time
        -- The Interrupters, "As We Live"

As we implemented this third exploit, it became clear that we could not
simply re-use the timing strategy that we had used against the two older
OpenSSH versions: we were never winning this new race condition.
Eventually, we understood why:

- It takes a long time (~10ms) for sshd to parse our fifth and last
  public key (packet e/ above); in other words, our large race window is
  too large (our 27 small race windows are like needles in a haystack).

- The user_specific_delay() that was introduced recently (OpenSSH 7.8p1)
  delays sshd's response to our last public-key packet by up to ~9ms and
  therefore destroys our feedback-based timing strategy.

As a result, we developed a completely different timing strategy:

- from time to time, we send our last public-key packet with a little
  mistake that produces an error response (lines 138-142 below), right
  before the call to sshkey_from_blob() that parses our public key;

- from time to time, we send our last public-key packet with another
  little mistake that produces an error response (lines 151-155 below),
  right after the call to sshkey_from_blob() that parses our public key;

- the difference between these two response times is the time that it
  takes for sshd to parse our last public key, and this allows us to
  precisely time the transmission of our last packets (to ensure that
  sshd has the time to parse our public key in the unprivileged child,
  send it to the privileged child, and start to parse it there, before
  the delivery of SIGALRM).

------------------------------------------------------------------------
 88 userauth_pubkey(struct ssh *ssh, const char *method)
 89 {
...
138         if (pktype == KEY_UNSPEC) {
139                 /* this is perfectly legal */
140                 verbose_f("unsupported public key algorithm: %s", pkalg);
141                 goto done;
142         }
143         if ((r = sshkey_from_blob(pkblob, blen, &key)) != 0) {
144                 error_fr(r, "parse key");
145                 goto done;
146         }
...
151         if (key->type != pktype) {
152                 error_f("type mismatch for decoded key "
153                     "(received %d, expected %d)", key->type, pktype);
154                 goto done;
155         }
------------------------------------------------------------------------

With this change in strategy, it takes ~10,000 tries on average to win
the race condition; i.e., with 100 connections (MaxStartups) accepted
per 120 seconds (LoginGraceTime), it takes ~3-4 hours on average to win
the race condition, and ~6-8 hours to obtain a remote root shell
(because of ASLR).


========================================================================
Towards an amd64 exploit
========================================================================

    What's your plan for tomorrow?
        -- The Interrupters, "Take Back the Power"

We decided to target Rocky Linux 9 (a Red Hat Enterprise Linux 9
derivative), from "Rocky-9.4-x86_64-minimal.iso", for two reasons:

- its OpenSSH version (8.7p1) is vulnerable to this signal handler race
  condition and its glibc is always mapped at a multiple of 2MB (because
  of the ASLR weakness discussed in the previous "Theory" subsection),
  which makes partial pointer overwrites much more powerful;

- the syslog() function (which is async-signal-unsafe but is called by
  sshd's SIGALRM handler) of this glibc version (2.34) internally calls
  __open_memstream(), which malloc()ates a FILE structure in the heap,
  and also calls calloc(), realloc(), and free() (which gives us some
  much-needed freedom).

With a heap corruption as a primitive, two FILE structures malloc()ated
in the heap, and 21 fixed bits in the glibc's addresses, we believe that
this signal handler race condition is exploitable on amd64 (probably not
in ~6-8 hours, but hopefully in less than a week). Only time will tell.

Side note: we discovered that Ubuntu 24.04 does not re-randomize the
ASLR of its sshd children (it is randomized only once, at boot time); we
tracked this down to the patch below, which turns off sshd's rexec_flag.
This is generally a bad idea, but in the particular case of this signal
handler race condition, it prevents sshd from being exploitable: the
syslog() inside the SIGALRM handler does not call any of the malloc
functions, because it is never the very first call to syslog().

  https://git.launchpad.net/ubuntu/+source/openssh/tree/debian/patches/systemd-socket-activation.patch


========================================================================
Patches and mitigation
========================================================================

    The storm has come and gone
        -- The Interrupters, "Good Things"

On June 6, 2024, this signal handler race condition was fixed by commit
81c1099 ("Add a facility to sshd(8) to penalise particular problematic
client behaviours"), which moved the async-signal-unsafe code from
sshd's SIGALRM handler to sshd's listener process, where it can be
handled synchronously:

  https://github.com/openssh/openssh-portable/commit/81c1099d22b81ebfd20a334ce986c4f753b0db29

Because this fix is part of a large commit (81c1099), on top of an even
larger defense-in-depth commit (03e3de4, "Start the process of splitting
sshd into separate binaries"), it might prove difficult to backport. In
that case, the signal handler race condition itself can be fixed by
removing or commenting out the async-signal-unsafe code from the
sshsigdie() function; for example:

------------------------------------------------------------------------
sshsigdie(const char *file, const char *func, int line, int showfunc,
    LogLevel level, const char *suffix, const char *fmt, ...)
{
#if 0
        va_list args;

        va_start(args, fmt);
        sshlogv(file, func, line, showfunc, SYSLOG_LEVEL_FATAL,
            suffix, fmt, args);
        va_end(args);
#endif
        _exit(1);
}
------------------------------------------------------------------------

Finally, if sshd cannot be updated or recompiled, this signal handler
race condition can be fixed by simply setting LoginGraceTime to 0 in the
configuration file. This makes sshd vulnerable to a denial of service
(the exhaustion of all MaxStartups connections), but it makes it safe
from the remote code execution presented in this advisory.


========================================================================
Acknowledgments
========================================================================

We thank OpenSSH's developers for their outstanding work and close
collaboration on this release. We also thank the distros@openwall.
Finally, we dedicate this advisory to Sophia d'Antoine.


========================================================================
Timeline
========================================================================

2024-05-19: We contacted OpenSSH's developers. Successive iterations of
patches and patch reviews followed.

2024-06-20: We contacted the distros@openwall.

2024-07-01: Coordinated Release Date.