nim-works/nimskull

Canonicalized filepaths has a large performance impact on compilation speed

Open

#546 opened on Feb 20, 2023

 (1 comment) (0 reactions) (0 assignees)Nim (39 forks)auto 404
good first issuehelp wanted

Repository metrics

Stars
 (346 stars)
PR merge metrics
 (PR metrics pending)

Description

Currently the compiler supports emiting "canonical" paths for modules, instead of say absolute, project file relative, etc, a list of possibilities can be seen in compiler/front/in_options.FilenameOption. Most pressingly, foCanonical results in massive performance problems as each attempt at canonicalzing a path results in a lot of filesystem IO.

The goal is fairly simple:

  1. get the baseline performance of compiling with --filenames=canonical and without
  2. track down the various use cases and classify them by purpose/intention
  3. with the benchmark data and use cases in hand, eliminate/reduce the filesystem access (AKA cache the data)

To measure the impact search the /tests directory for tests that specify the --filenames=canonical option, then run those with and without to observe the difference. The largest impact was observed with ./koch.py temp --stacktrace --stacktracemsgs -d:nimCompilerStacktraceHints --lib:lib -d:debug -d:usenodeids c --filenames=canonical --msgFormat=sexp tests/compilerfeatures/tstructured_parse_fail.nim, as this resulted in setMsgFrame calls which converted TLineInfo to strings triggering the canonicalization for many hot compiler procedures.

If stuck at any point, ask @saem.

Contributor guide