nar*_*ren 7 python memory-leaks sqlalchemy pyramid
如何解决这个内存泄漏？
我应该采取什么措施来清理旧的会话对象？是不是session.close()足够？
要么
这与金字塔有关吗？
Sqlalchmey setup:
----------------------------------------------------------------------------------
def get_db(request):
maker = request.registry.dbmaker
session = maker()
@profile
def cleanup(request):
_session = request.db
if request.exception is not None:
_session.rollback()
else:
_session.commit()
_session.close()
# del _session # No memory released
request.add_finished_callback(cleanup)
return session
def main(global_config, **settings):
:
:
config.registry.dbmaker = sessionmaker(bind=engine)
config.add_request_method(get_db, name='db', reify=True)
:
:
金字塔app请求处理程序就像
@view_config(route_name='list_employees', renderer='json')
def employees(request):
session = request.db
office = session.query(Office).get(1)
employees = [x.name for x in office.employees]
return employees
现在的问题是,在list_employees的每个请求中,内存都在增长.内存增加的大小几乎等于大小office.employees.
调试:
request 1 starts with memory utilization = 10MB
request 1 ends with memory utilization = 18MB
request 2 starts with memory utilization = 18MB
request 2 ends with memory utilization = 26MB
request 3 starts with memory utilization = 26MB
request 3 ends with memory utilization = 34MB
:
:
Grows eventually
employees = [x.name for x in office.employees]
This is the line where about 8-10MB memory utilized
为了调试,我在Employ和Office模型中添加了__del__方法,看起来它们正在删除.
也尝试了session.expunge(office),del office和gc.collect()
我正在调试内存消耗使用https://pypi.python.org/pypi/memory_profiler 我也在使用https://pypi.python.org/pypi/transaction是其他请求.
不使用调试金字塔工具栏.
编辑:发现此行的内存增加(employees = [x.name for x in office.employees])在6-7个请求后显示为零.但查询返回相同的行数.
编辑:添加了独立应用程序https://github.com/Narengowda/pyramid_sqlalchemy_app
编辑:它与SQLALCHEMY无关(我的坏).写了一个简单的视图函数,它没有任何sqlalchmey查询.
class Test(object):
def __init__(self):
self.x = 'sdfklhasdjkfhasklsdkjflksdfksd' *1000
self.y = 'sdfklhasdjkfhasklsdkjflksdfksd' *1000
self.z = 'sdfklhasdjkfhasklsdkjflksdfksd' *1000
self.i = 'sdfklhasdjkfhasklsdkjflksdfksd' *1000
self.v = 'sdfklhasdjkfhasklsdkjflksdfksd' *1000
self.o = 'sdfklhasdjkfhasklsdkjflksdfksd' *1000
@view_config(route_name='home', renderer='json')
def my_view(request):
return test(request)
@profile
def test(request):
count = request.GET.get('count')
l = [Test() for i in range(int(count))]
print l[0]
return {}
我能够看到这一点,下面是请求的日志
Line#Mem使用增量行内容
23 37.3 MiB 0.0 MiB @profile
24 def test(request):
25 37.3 MiB 0.0 MiB count = request.GET.get('count')
26 112.4 MiB 75.1 MiB l = [Test() for i in range(int(count))]
27 112.4 MiB 0.0 MiB print l[0]
28 112.4 MiB 0.0 MiB return {}
Line#Mem使用增量行内容
23 111.7 MiB 0.0 MiB @profile
24 def test(request):
25 111.7 MiB 0.0 MiB count = request.GET.get('count')
26 187.3 MiB 75.6 MiB l = [Test() for i in range(int(count))]
27 187.3 MiB 0.0 MiB print l[0]
28 187.3 MiB 0.0 MiB return {}
Line#Mem使用增量行内容
23 184.3 MiB 0.0 MiB @profile
24 def test(request):
25 184.3 MiB 0.0 MiB count = request.GET.get('count')
26 259.7 MiB 75.4 MiB l = [Test() for i in range(int(count))]
27 259.7 MiB 0.0 MiB print l[0]
28 259.7 MiB 0.0 MiB return {}
Line#Mem使用增量行内容
23 255.1 MiB 0.0 MiB @profile
24 def test(request):
25 255.1 MiB 0.0 MiB count = request.GET.get('count')
26 330.4 MiB 75.3 MiB l = [Test() for i in range(int(count))]
27 330.4 MiB 0.0 MiB print l[0]
28 330.4 MiB 0.0 MiB return {}
Line#Mem使用增量行内容
23 328.2 MiB 0.0 MiB @profile
24 def test(request):
25 328.2 MiB 0.0 MiB count = request.GET.get('count')
26 330.5 MiB 2.3 MiB l = [Test() for i in range(int(count))]
27 330.5 MiB 0.0 MiB print l[0]
28 330.5 MiB 0.0 MiB return {}
Line#Mem使用增量行内容
23 330.5 MiB 0.0 MiB @profile
24 def test(request):
25 330.5 MiB 0.0 MiB count = request.GET.get('count')
26 330.5 MiB 0.0 MiB l = [Test() for i in range(int(count))]
27 330.5 MiB 0.0 MiB print l[0]
28 330.5 MiB 0.0 MiB return {}
我已经尝试过多次使用不同的计数查询参数,看到内存利用率的增加在确切的5个请求(魔术)之后停止.
此外,我尝试打印所有对象并比较那些我观察到的地址,看看请求4和5的日志.看起来像GC发生,因此内存从330.4 Mi减少到328.2 MiB但你不会看到75.3 MiB内存利用率创建新对象(第26行),但你可以看到仅增加2.3 MiB.后来我验证了最后两个请求中创建的对象的地址,发现最后两个请求的对象的80%地址是相同的
<pyramid_sqa.views.Test object at 0x3a042d0>
<pyramid_sqa.views.Test object at 0x3a04310>
<pyramid_sqa.views.Test object at 0x3a04350>
<pyramid_sqa.views.Test object at 0x3a04390>
<pyramid_sqa.views.Test object at 0x3a043d0>
<pyramid_sqa.views.Test object at 0x3a04410>
<pyramid_sqa.views.Test object at 0x3a04450>
<pyramid_sqa.views.Test object at 0x3a04490>
<pyramid_sqa.views.Test object at 0x3a044d0>
<pyramid_sqa.views.Test object at 0x3a04510>
<pyramid_sqa.views.Test object at 0x3a04390>
<pyramid_sqa.views.Test object at 0x3a043d0>
<pyramid_sqa.views.Test object at 0x3a04410>
<pyramid_sqa.views.Test object at 0x3a04450>
<pyramid_sqa.views.Test object at 0x3a04490>
<pyramid_sqa.views.Test object at 0x3a044d0>
<pyramid_sqa.views.Test object at 0x3a04290>
<pyramid_sqa.views.Test object at 0x3a04550>
<pyramid_sqa.views.Test object at 0x3a04590>
<pyramid_sqa.views.Test object at 0x3a045d0>
所以创建了新对象,python正在重用内存(重用对象!!？)
如果我的服务器内存像这样拍摄内存可以吗？
Python 对 Python 对象进行自己的内存管理，即使 CPython GC 释放 Python 对象，它仍然不会将内存释放给操作系统（如 malloc()/free() 可能会这样做）。当 GC 释放 Python 对象时，内存可用于新的 Python 对象。这是当请求 6 的内存消耗没有增加时您看到的效果。在请求 5 之后，GC 释放了已删除的对象，并且请求 6 中的新对象可以使用释放的内存。
所以你没有内存泄漏，你只是发现了 CPython 内存管理的工作原理。内存消耗不会无限制地增长。