nesquena/hermes-webui

bug(shutdown): 'Stop server' button no-ops on ctl.sh-managed daemons (SIGINT inherited SIG_IGN from non-interactive bash '&')

Open

#7,078 opened on Aug 16, 2026

 (2 comments) (0 reactions) (0 assignees)Python (2,399 forks)github user discovery
bughelp wantedsprint-candidate

Repository metrics

Stars
 (17,426 stars)
PR merge metrics
 (Avg merge 14h 31m) (314 merged PRs in 30d)

Description

Symptom

The Stop server button in Settings (POST /api/shutdown) returns 200 {"status": "shutting_down"} but the WebUI process keeps running — for daemons launched via ./ctl.sh start (including the launchd-managed setup where the plist calls ctl.sh start).

Downstream consequences for the user:

  • Every subsequent ./ctl.sh start spawns a new instance that dies with [!!] FATAL: Another server is already responding on 127.0.0.1:8787. Stop the existing instance first. (_abort_if_already_serving, server.py:529) — the old process still owns the port.
  • The pid/state files (~/.hermes/webui.pid, ~/.hermes/webui.ctl.env) keep pointing at the just-died new PIDs, so ctl.sh stop also cannot reach the real process.
  • Result: the only way to actually stop the daemon is kill <pid> from a terminal.

Root cause

Three facts combine:

  1. ctl.sh start spawns the server from a non-interactive bash background job (( cd ...; exec nohup python bootstrap.py ... ) >> log 2>&1 &). POSIX/bash rule: "If job control is not in effect, asynchronous commands ignore SIGINT and SIGQUIT in addition to SIGHUP" — the child inherits SIGINT = SIG_IGN.

    Minimal proof:

    $ bash -c 'python3 -c "import signal; print(repr(signal.getsignal(signal.SIGINT)))" & wait'
    <Handlers.SIG_IGN: 1>
    
  2. CPython preserves an inherited SIG_IGN for SIGINT — it does not install its KeyboardInterrupt handler when SIGINT was ignored at interpreter startup.

  3. _handle_shutdown (api/routes.py) commits suicide with SIGINT:

    def _do_shutdown():
        import time
        time.sleep(0.3)
        os.kill(os.getpid(), signal.SIGINT)
    

    For a ctl.sh-managed daemon this signal is a silent no-op at the OS level.

    Meanwhile server.py only registers a graceful handler for SIGTERM (server.py:717 signal.signal(signal.SIGTERM, _request_shutdown)) — which is exactly why kill -TERM stops the daemon cleanly while the button's SIGINT does nothing.

Reproduction

  1. ./ctl.sh start (any non-interactive shell).
  2. curl -X POST http://127.0.0.1:8787/api/shutdown (authenticated session) → 200 {"status":"shutting_down"}.
  3. Process still alive:
    $ kill -0 <pid> && echo alive
    alive
    
  4. Signal-level equivalent:
    $ kill -INT <pid>   # no effect, process stays alive
    $ kill -TERM <pid>  # graceful shutdown (shutdown-audit + drain in the serve_forever() finally block)
    
    Note: _log_shutdown_audit() (server.py:492) has never fired in my environment's webui.log despite many stop attempts — consistent with no shutdown signal ever being processed.

Suggested fix

In server.py main(), register the existing _request_shutdown handler for SIGINT as well as SIGTERM:

try:
    signal.signal(signal.SIGTERM, _request_shutdown)
    signal.signal(signal.SIGINT, _request_shutdown)
except (ValueError, OSError):
    ...

This matches the handler's own docstring intent ("The handler is idempotent and guards against double-shutdown (e.g. repeated SIGTERM/SIGINT)") and makes the Settings stop button, external kill -INT, and foreground Ctrl-C all take the same graceful path (the finally block around serve_forever() already drains sessions either way).

Alternative (narrower): have _do_shutdown send SIGTERM instead of SIGINT. This fixes the button but leaves process-level SIGINT ignored for ctl.sh daemons.

Verification performed

  • signal.getsignal(SIGINT) in a bash-&-spawned child → SIG_IGN (mechanism).
  • Isolated instance (separate HERMES_HOME/port 8789, launched via ctl.sh): kill -INT → alive; kill -TERM → clean exit with [crash-visibility] process exit logged.
  • Production daemon (PID up 19 days, launched via launchd→ctl.sh): /api/shutdown returned 200 at 20:39:51, process still alive 12+ minutes later; all ctl.sh start attempts in between died with the port-in-use FATAL.
  • Confirmed against current upstream master (2026-08-15): server.py still registers only SIGTERM; routes.py _do_shutdown still uses SIGINT.

Related: #6396 assumes the Control Center stop works for ctl.sh-managed daemons — this bug is why the restart half of that story matters.

Contributor guide