我正在使用进化算法(CMAES)进行函数优化.为了更快地运行它我正在使用多处理模块.我需要优化的函数(input_A_Opt, and input_B_Opt)在下面的代码中将大矩阵作为输入.
它们是几GB的大小.当我在没有多处理的情况下运行该函数时,它运行良好.当我使用多处理时,内存似乎有问题.如果我使用小输入运行它运行良好,但当我使用完整输入运行时,我收到以下错误:
File "<ipython-input-2-bdbae5b82d3c>", line 1, in <module>
opt.myFuncOptimization()
File "/home/joe/Desktop/optimization_folder/Python/Optimization.py", line 45, in myFuncOptimization
**f_values = pool.map_async(partial_function_to_optmize, solutions).get()**
File "/usr/lib/python3.5/multiprocessing/pool.py", line 608, in get
raise self._value
File "/usr/lib/python3.5/multiprocessing/pool.py", line 385, in _handle_tasks
put(task)
File "/usr/lib/python3.5/multiprocessing/connection.py", line 206, in send
self._send_bytes(ForkingPickler.dumps(obj))
File "/usr/lib/python3.5/multiprocessing/connection.py", line 393, in _send_bytes
header = struct.pack("!i", n)
error: 'i' format requires -2147483648 <= number <= 2147483647
这里是代码的简化版本(再次,如果我使用输入10倍的输入运行它,一切正常):
import numpy as np
import cma
import multiprocessing as mp
import functools
import myFuncs
import …