nesquena/hermes-webui

in-chat cronjob action=run still holds global _cron_env_lock for entire run_job() — WebUI freezes (fix in #1746 missed this path)

Open

#6,753 opened on Aug 4, 2026

 (3 comments) (0 reactions) (0 assignees)Python (2,386 forks)github user discovery
bughelp wantedperformanceprioritystreamingtasks

Repository metrics

Stars
 (17,368 stars)
PR merge metrics
 (Avg merge 14h 31m) (314 merged PRs in 30d)

Description

Summary

When an agent calls the in-chat cronjob tool with action="run", the WebUI still holds the process-wide _cron_env_lock for the entire run_job() execution (minutes for a real agent run). During that window every other WebUI request — chat, opening a second window, /api/session, even a same-profile read — blocks on _cron_env_lock.acquire(). The UI appears frozen, SSE heartbeats time out, and the browser repeatedly disconnects/reconnects.

This is the same class of bug as #1574, but that fix (PR #1746 / v0.51.12) only covered the Scheduled Jobs panel manual-run path (_run_cron_tracked → subprocess). The in-chat tool path was not covered and still executes synchronously in the parent process while holding the lock.

Repro (tested 2026-08-04, exp-v0.52.153)

  1. In a WebUI chat session, ask the agent to run a cron job (e.g. cronjob action="run" on the daily news job).
  2. The job takes 2-5 minutes (web search + LLM).
  3. During the run: open a second tab/window, send any message, or reload — everything hangs; /api/chat/start returns 409; the page disconnects and auto-reconnects once the job finishes.

Evidence from request-diagnostics (thread stacks at freeze)

  • Thread-119 (_run_agent_streaming)cronjob_tools.cronjob_execute_job_now(job)run_one_jobconcurrent.futures.wait (synchronously waiting for the job)
  • 9+ HTTP threads (Thread-100/117/134/148/161/189/194/215/219) all stuck at api/profiles.py:747 _cron_env_lock.acquire()
  • POST /api/chat/start409 during the window
  • /api/session slow-request: 5.6s blocked at t2_after_state_db_load

Root cause

The in-chat path wraps the tool in a profile context that acquires the global lock and never releases it until the synchronous job completes:

streaming.py:268 _profile_scoped_cronjob_handler
  → profiles.py:667 cron_profile_context_for_home.__enter__
      → _cron_env_lock.acquire()          # held for entire run_job()
  → original_handler → _execute_job_now()  # synchronous, minutes

The panel path was fixed by moving the job body into a pinned subprocess (#1746), but the in-chat tool handler still calls original_handler synchronously inside the locked context.

Suggested fix

Apply the same subprocess boundary (or async queueing) to the in-chat cronjob tool path — e.g. make _profile_scoped_cronjob_handler trigger the job via trigger_job() (queue for next scheduler tick, like CLI hermes cron run does) instead of _execute_job_now(), or run the job body in a profile-pinned subprocess like the panel path already does.

Workaround

Use hermes cron run <job_id> from the CLI (queues via trigger_job, returns immediately) instead of the in-chat tool.

Contributor guide