sphinx-doc/sphinx

Cannot recover from MemoryErrors

Closed

#7,635 opened on May 8, 2020

View on GitHubContributor guide
 (2 comments) (0 reactions) (0 assignees)Python (1,985 forks)batch import
help wantedtype:bug

Repository metrics

Stars
 (5,625 stars)
PR merge metrics
 (Avg merge 10d 11h) (11 merged PRs in 30d)

Description

We are using recommonmark with sphinx to parse 2.3k+ md files and experiencing excesive memory usage. Running 64 bit we see a peak of about 19gb of memory usage and an execution time around 30 minutes. On many machines (or running 32 bit) this is not doable. The majority of the markdown files have an rst label, and some of them have references to these labels. Are there configuration options that may be affecting this? Is there a way to review what's actually taking up this memory?

Contributor guide

Research direction
Investigate memory usage during parsing of many markdown files with recommonmark. Use memory profiler or tracemalloc to identify peak memory allocations. Compare with direct Python memory usage and explore Sphinx's handling of large document sets. Look at recommonmark's parser and Sphinx's build process in sphinx/builders/ for potential optimizations or configuration options to reduce memory footprint.
Tech stack
python
Domain
documentationperformance
Issue type
Bug
DifficultyEstimated implementation difficulty for a new contributor, from 1 for very small changes to 5 for expert-level work.
4
Estimated timeA rough time range for an experienced contributor to investigate, implement, test, and prepare a pull request.
Over 1 week
Activity statusHow available the issue appears right now: fresh, active, stale, blocked, or waiting on maintainer input.
Stale
ClarityHow clearly the issue explains the expected change, acceptance criteria, and next step.
Needs investigation
Prerequisites
PythonMemory profilingSphinx internalsCPython debugging
Newbie friendlinessA 1-100 score estimating how approachable this issue is for first-time contributors.
10