ModuleNotFoundError: No module named 'numpy.core.multiarray' When Python 3 try to load pkl file that generated by Python 2 pickle
#3,193 opened on Nov 14, 2024
Repository metrics
- Stars
- (10,351 stars)
- PR merge metrics
- (Avg merge 5d 12h) (6 merged PRs in 30d)
Description
Note: This issue is due to I was trying to use pickle(Python 3) to load a pkl file that generated by Python 2.7.x pickle. I don't think this issue needs pay much attention, as Python 2's lifecycle has been over for a while now. I have resolved this issue, steps to reproduce the issue and the solution are as follows.
-
Nuitka version, full Python version, flavor, OS, etc. as output by this exact command.
python -m nuitka --version
2.4.11
Commercial: None
Python: 3.12.7 | packaged by Anaconda, Inc. | (main, Oct 4 2024, 13:17:27) [MSC v.1929 64 bit (AMD64)]
Flavor: Anaconda Python
Executable: D:\Software\Miniconda3\envs\Test\python.exe
OS: Windows
Arch: x86_64
WindowsRelease: 11
Version C compiler: C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.41.34120\bin\Hostx64\x64\cl.exe (cl 14.3).
-
How did you install Nuitka and Python
conda virtualenv, pip install nuitka
-
The specific PyPI names and versions
python -m pip list -v
> Package Version Location Installer
> ----------- ------- -------------------------------------------------- ---------
> Nuitka 2.4.11 D:\Software\Miniconda3\envs\Test\Lib\site-packages pip
> numpy 2.1.3 D:\Software\Miniconda3\envs\Test\Lib\site-packages pip
> ordered-set 4.1.0 D:\Software\Miniconda3\envs\Test\Lib\site-packages pip
> pip 24.2 D:\Software\Miniconda3\envs\Test\Lib\site-packages conda
> setuptools 75.1.0 D:\Software\Miniconda3\envs\Test\Lib\site-packages
> wheel 0.44.0 D:\Software\Miniconda3\envs\Test\Lib\site-packages
> zstandard 0.23.0 D:\Software\Miniconda3\envs\Test\Lib\site-packages pip
- Also supply a Short, Self Contained, Correct, Example
generate_pkl_file.py
#-*- coding:utf-8 -*-
import numpy as np
import pickle
import zlib
original_data = {
'slice': [0.0, 1.0],
'data': {
"sec1": np.zeros(5),
"sec2": np.ones(5),
}
}
with open("demo_py2.pkl", 'wb') as fh:
fh.write(zlib.compress(pickle.dumps(original_data, protocol=2)))
fh.close()
The original pkl file is too large, so I wrote this script to generate small test files.
Please make sure run generate_pkl_file.py in both Python 2 (2.7.18) and Python 3 (3.12.7)environments.
This script is use to generate two test files(demo_py2.pkl and demo_py3.pkl).
Adjust the filename as needed before run this script.
test.py
#-*- coding:utf-8 -*-
# import numpy.core.multiarray
import numpy as np
import pickle
import zlib
final_data = pickle.loads(zlib.decompress(open("demo_py3.pkl", 'rb').read()), encoding='latin1')
print("Python 3:\n", final_data)
final_data = pickle.loads(zlib.decompress(open("demo_py2.pkl", 'rb').read()), encoding='latin1')
print("Python 2:\n", final_data)
Compile test.py using Nuitka . Please keep import numpy.core.multiarray line is commented out(see solution 2).
After compiling, run the generated .\test.dist\test.exe file to reproduce the issue. The output:
Python 3:
{'slice': [0.0, 1.0], 'data': {'sec1': array([0., 0., 0., 0., 0.]), 'sec2': array([1., 1., 1., 1., 1.])}}
Traceback (most recent call last):
File "path\to\test.dist\test.py", line 12, in <module>
ModuleNotFoundError: No module named 'numpy.core.multiarray'
The file generated by Python 3 is load correctly, while the file generated by Python 2 is not.
- Provide in your issue the Nuitka options used
python -m nuitka --msvc=latest --standalone .\test.py
- Note if this is a regression
numpy < 2.0 works
- Solution 1:
add
implicit-importtopath\to\myenv\Lib\site-packages\nuitka\plugins\standard stdlib3.nuitka-package.config.yml
- module-name: 'pickle' # checksum: 167cb032
implicit-imports:
- depends:
- 'numpy.core.multiarray'
anti-bloat:
- description: 'remove module ability to run as a binary'
change_function:
'_test': "'(lambda: None)'"
- Solution 2:
Simply adding the import statement
import numpy.core.multiarraybeforepickle.loadsfunction(uncomment this line in test.py) should resolve the issue. This solution meets my requirements, as in my entire project, this function is only used in specific location, so the impact of this solution is minimal.