为什么我的Fortran代码用f2py包含这么多内存？

Question

我试图计算大约十万点之间的所有距离.我有以下代码用Fortran编写并使用以下代码编译f2py:

C         1         2         3         4         5         6         7
C123456789012345678901234567890123456789012345678901234567890123456789012
        subroutine distances(coor,dist,n)
        double precision coor(n,3),dist(n,n)
        integer n
        double precision x1,y1,z1,x2,y2,z2,diff2

cf2py intent(in) :: coor,dist
cf2py intent(in,out):: dist
cf2py intent(hide)::n
cf2py intent(hide)::x1,y1,z1,x2,y2,z2,diff2

        do 200,i=1,n-1
            x1=coor(i,1)
            y1=coor(i,2)
            z1=coor(i,3)
            do 100,j=i+1,n
                x2=coor(j,1)
                y2=coor(j,2)
                z2=coor(j,3)
                diff2=(x1-x2)*(x1-x2)+(y1-y2)*(y1-y2)+(z1-z2)*(z1-z2)
                dist(i,j)=sqrt(diff2)
  100       continue
  200   continue

        end

我正在使用以下python代码编译fortran代码setup_collision.py:

# System imports
from distutils.core import *
from distutils      import sysconfig

# Third-party modules
import numpy
from numpy.distutils.core import Extension, setup

# Obtain the numpy include directory.  This logic works across numpy versions.
try:
    numpy_include = numpy.get_include()
except AttributeError:
    numpy_include = numpy.get_numpy_include()

# simple extension module
collision = Extension(name="collision",sources=['./collision.f'],
               include_dirs = [numpy_include],
               )

# NumyTypemapTests setup
setup(  name        = "COLLISION",
        description = "Module calculates collision energies",
        author      = "Stvn66",
        version     = "0.1",
        ext_modules = [collision]
        )

然后运行如下:

import numpy as np
import collision

coor = np.loadtxt('coordinates.txt')
n_atoms = len(coor)
dist = np.zeros((n_atoms, n_atoms), dtype=np.float16) # float16 reduces memory
n_dist = n_atoms*(n_atoms-1)/2
n_GB = n_dist * 2 / float(2**30) # 1 kB = 1024 B
n_Gb = n_dist * 2 / 1E9          # 1 kB = 1000 B
print 'calculating %d distances between %d atoms' % (n_dist, n_atoms)
print 'should use between %f and %f GB of memory' % (n_GB, n_Gb)
dist = collision.distances(coor, dist)

使用具有30,000个原子的代码,应该使用大约1 GB的内存来存储距离,而是使用10 GB.有了这个差异,使用100,000个原子执行此计算将需要100 GB而不是10 GB.我的电脑只有20 GB.

我错过了在Python和Fortran之间传递数据的相关内容吗？巨大的差异表明实施中存在重大缺陷.

Answer 1

您正在向Fortran子例程提供双精度数组.双精度中的每个元素需要8字节的内存.因为N=30,000这样做

coor(n,3) => 30,000*3*8 ~ 0.7 MB
dist(n,n) => 30,000^2*8 ~ 6.7 GB

由于Python还需要半精度浮点数,因此另外需要1-2GB.所以总体要求是9-10GB.

同样适用N=100,000,仅Fortran部分需要约75GB.

double precision您应该使用单精度reals 而不是浮点数- 如果这足以进行计算.这将导致一半的内存需求.[我没有经验,但我认为如果两个部分使用相同的精度,Python可以直接对数据进行操作......]

正如@VladimirF在他的评论中指出的那样,"通常的编译器不支持2字节实数".我检查gfortran和ifort,他们都没有.所以你需要至少使用单精度.