haskell-nix/hnix

Allow concurrent evaluation

Open

#170 opened on Apr 12, 2018

 (1 comment) (0 reactions) (0 assignees)Nix (116 forks)github user discovery
challengingenhancementhelp wanted

Repository metrics

Stars
 (835 stars)
PR merge metrics
 (No merged PRs in 30d)

Description

Right now parsing is our slowest task, and yet parsing occurs during evaluation all the time due to import expressions. It would be nice if we could assign a pool of N workers (using the async-pool library), and perform these slow evaluations concurrently since none of them can have interfering side-effects. This should speed up the evaluation of nixpkgs considerably if there are many capabilities available.

To do this will need a few steps:

  • Make Thunk.hs thread-safe by introducing a variable that causes other threads to wait for pending evaluations to complete. Right now there is a boolean flag in an IORef; it might be sufficient to just turn this into a trinary flag in a TVar.

  • Create MonadConcurrent to abstract when concurrent evaluation is possible. It wouldn't be for the symbolic evaluator, for example, but would for an evaluator that runs in IO, such as Lazy.

  • Replace the calls to traverse that happen in Exec.hs for NList and NSet to instead use a new concurrentTraverse method defined in MonadConcurrent, which for IO would be implemented by calling Control.Concurrent.Async.Pool.mapTasks <group> and for pure evaluation would just call traverse as before.

Pinging @ryantrinkle, who requested this feature.

Contributor guide