Nuitka/Nuitka

ModuleNotFoundError: No module named 'numpy.core.multiarray' When Python 3 try to load pkl file that generated by Python 2 pickle

Open

#3,193 opened on Nov 14, 2024

 (2 comments) (0 reactions) (1 assignee)Python (565 forks)batch import
enhancementhelp wanted

Repository metrics

Stars
 (10,351 stars)
PR merge metrics
 (Avg merge 5d 12h) (6 merged PRs in 30d)

Description

Note: This issue is due to I was trying to use pickle(Python 3) to load a pkl file that generated by Python 2.7.x pickle. I don't think this issue needs pay much attention, as Python 2's lifecycle has been over for a while now. I have resolved this issue, steps to reproduce the issue and the solution are as follows.

  • Nuitka version, full Python version, flavor, OS, etc. as output by this exact command.

    python -m nuitka --version

  2.4.11
  Commercial: None
  Python: 3.12.7 | packaged by Anaconda, Inc. | (main, Oct  4 2024, 13:17:27) [MSC v.1929 64 bit (AMD64)]
  Flavor: Anaconda Python
  Executable: D:\Software\Miniconda3\envs\Test\python.exe
  OS: Windows
  Arch: x86_64
  WindowsRelease: 11
  Version C compiler: C:\Program Files (x86)\Microsoft Visual Studio\2022\BuildTools\VC\Tools\MSVC\14.41.34120\bin\Hostx64\x64\cl.exe (cl 14.3).
  • How did you install Nuitka and Python

    conda virtualenv, pip install nuitka

  • The specific PyPI names and versions

    python -m pip list -v

  > Package     Version Location                                           Installer
  > ----------- ------- -------------------------------------------------- ---------
  > Nuitka      2.4.11  D:\Software\Miniconda3\envs\Test\Lib\site-packages pip
  > numpy       2.1.3   D:\Software\Miniconda3\envs\Test\Lib\site-packages pip
  > ordered-set 4.1.0   D:\Software\Miniconda3\envs\Test\Lib\site-packages pip
  > pip         24.2    D:\Software\Miniconda3\envs\Test\Lib\site-packages conda
  > setuptools  75.1.0  D:\Software\Miniconda3\envs\Test\Lib\site-packages
  > wheel       0.44.0  D:\Software\Miniconda3\envs\Test\Lib\site-packages
  > zstandard   0.23.0  D:\Software\Miniconda3\envs\Test\Lib\site-packages pip
  • Also supply a Short, Self Contained, Correct, Example

generate_pkl_file.py

#-*- coding:utf-8 -*-

import numpy as np
import pickle
import zlib

original_data = {
    'slice': [0.0, 1.0], 
    'data': {
        "sec1": np.zeros(5), 
        "sec2": np.ones(5), 
    }
}

with open("demo_py2.pkl", 'wb') as fh:
    fh.write(zlib.compress(pickle.dumps(original_data, protocol=2)))
fh.close()

The original pkl file is too large, so I wrote this script to generate small test files. Please make sure run generate_pkl_file.py in both Python 2 (2.7.18) and Python 3 (3.12.7)environments. This script is use to generate two test files(demo_py2.pkl and demo_py3.pkl). Adjust the filename as needed before run this script.

test.py

#-*- coding:utf-8 -*-

# import numpy.core.multiarray
import numpy as np
import pickle
import zlib

final_data = pickle.loads(zlib.decompress(open("demo_py3.pkl", 'rb').read()), encoding='latin1')
print("Python 3:\n", final_data)

final_data = pickle.loads(zlib.decompress(open("demo_py2.pkl", 'rb').read()), encoding='latin1')
print("Python 2:\n", final_data)

Compile test.py using Nuitka . Please keep import numpy.core.multiarray line is commented out(see solution 2).

After compiling, run the generated .\test.dist\test.exe file to reproduce the issue. The output:

Python 3:
 {'slice': [0.0, 1.0], 'data': {'sec1': array([0., 0., 0., 0., 0.]), 'sec2': array([1., 1., 1., 1., 1.])}}
Traceback (most recent call last):
  File "path\to\test.dist\test.py", line 12, in <module>
ModuleNotFoundError: No module named 'numpy.core.multiarray'

The file generated by Python 3 is load correctly, while the file generated by Python 2 is not.

  • Provide in your issue the Nuitka options used

python -m nuitka --msvc=latest --standalone .\test.py

  • Note if this is a regression

numpy < 2.0 works

  • Solution 1: add implicit-import to path\to\myenv\Lib\site-packages\nuitka\plugins\standard stdlib3.nuitka-package.config.yml
- module-name: 'pickle' # checksum: 167cb032
  implicit-imports:  
    - depends:
      - 'numpy.core.multiarray'
  anti-bloat:        
    - description: 'remove module ability to run as a binary'
      change_function:
        '_test': "'(lambda: None)'"
  • Solution 2: Simply adding the import statement import numpy.core.multiarray before pickle.loads function(uncomment this line in test.py) should resolve the issue. This solution meets my requirements, as in my entire project, this function is only used in specific location, so the impact of this solution is minimal.

Contributor guide