Mar*_*icu 7 python storage psutil
This is NOT a duplicate of this. I'm not interested in finding out my memory consumption or the matter, as I'm already doing that below. The question is WHY the memory consumption is like this.
Also, even if I did need a way to profile my memory do note that guppy (the suggested Python memory profiler in the aforementioned link does not support Python 3 and the alternative guppy3 does not give accurate results whatsoever yielding in results such as (see actual sizes below):
Partition of a set of 45968 objects. Total size = 5579934 bytes.
Index Count % Size % Cumulative % Kind (class / dict of class)
0 13378 29 1225991 22 1225991 22 str
1 11483 25 843360 15 2069351 37 tuple
2 2974 6 429896 8 2499247 45 types.CodeType
Right, so I have this simple script which I'm using to do some RAM consumption tests, by reading a file in 2 different ways:
reading a file one line at a time, processing, and discarding it (via generators), which is efficient and recommended for basically any file size (especially large files), which works as expected.
reading a whole file into memory (I know this is advised against, however this was just for educational purposes).
import os
import psutil
import time
with open('errors.log') as file_handle:
statistics = os.stat('errors.log') # See below for contents of this file
file_size = statistics.st_size / 1024 ** 2
process = psutil.Process(os.getpid())
ram_usage_before = process.memory_info().rss / 1024 ** 2
print(f'File size: {file_size} MB')
print(F'RAM usage before opening the file: {ram_usage_before} MB')
file_handle.read() # loading whole file in memory
ram_usage_after = process.memory_info().rss / 1024 ** 2
print(F'Expected RAM usage after loading the file: {file_size + ram_usage_before} MB')
print(F'Actual RAM usage after loading the file: {ram_usage_after} MB')
# time.sleep(30)
File size: 111.75 MB
RAM usage before opening the file: 8.67578125 MB
Expected RAM usage after loading the file: 120.42578125 MB
Actual RAM usage after loading the file: 343.2109375 MB
I also added a 30 second sleep to check with awk at the os level, where I've used the following command:
ps aux | awk '{print $6/1024 " MB\t\t" $11}' | sort -n
which yields:
...
343.176 MB python # my script
619.883 MB /Applications/PyCharm.app/Contents/MacOS/pycharm
2277.09 MB com.docker.hyperkit
The file contains about 800K copies of the following line:
[2019-09-22 16:50:17,236] ERROR in views, line 62: 404 Not Found: The
following URL: http://localhost:5000/favicon.ico was not found on the
server.
Is it because of block sizes or dynamic allocation, whereby the contents would be loaded in blocks and a lot of that memory would actually be unused ?
当您在 Python 中打开文件时，默认情况下您是在Text-mode 中打开它。这意味着二进制数据是根据操作系统默认值或明确给定的编解码器解码的。
与所有数据一样，文本数据在计算机中由字节表示。大多数英文字母表都可以用单个字节表示，例如字母“A”通常被翻译成数字 65，或者二进制：01000001。这种编码（ASCII）是许多情况下够用了，但是当你想在像罗马尼亚语言来编写文字，这已经是不够的，因为人物?，?等不ASCII的一部分。
有一段时间，人们对每种语言（组）使用不同的编码，例如基于拉丁字母表的语言使用 Latin-x 编码组 (ISO-8859-x)，以及其他（尤其是CJK）语言的其他编码。
如果要表示某些亚洲语言或几种不同的语言，则需要将一个字符编码为多个字节的编码。这可以是固定数字（例如在 UTF-32 和 UTF-16 中）或可变数字，如当今最常见的“流行”编码 UTF-8。
回到 Python：Python 字符串接口承诺了许多属性，其中包括 O(1) 复杂度的随机访问，这意味着即使从很长的字符串中也可以非常快速地获取第 1245 个字符。这与紧凑的 UTF-8 编码相冲突：因为一个“字符”（实际上：一个 unicode 代码点）有时是一个字节，有时是几个字节长，Python 不能仅仅跳转到内存地址start_of_string + length_of_one_character * offset，因为length_of_one_characterUTF-8 中的不同。因此 Python 需要使用固定字节长度的编码。
出于优化原因，它并不总是使用 UCS-4 (~UTF-32)，因为当文本仅为 ASCII时，这会浪费大量空间。相反，Python 会动态选择 Latin-1、UCS-2 或 UCS-4 来在内部存储字符串。
用一个例子把所有东西放在一起：
假设您想将字符串“solu?ie”存储在内存中，来自一个编码为 UTF-8 的文件。由于字母?需要两个字节来表示，Python 选择了 UCS-2：
characters | s | o | l | u | ? | i | e
utf-8 |0x73 |0x6f |0x6c |0x75 |0xc5 0xa3|0x69 |0x65
ucs-2 |0x00 0x73|0x00 0x6f|0x00 0x6c|0x00 0x75|0x01 0x63|0x00 0x69|0x00 0x65
如您所见，UTF-8（磁盘上的文件）需要 8 个字节，而 UCS-2 需要 14 个字节。
再加上 Python 字符串和 Python 解释器本身的开销，您的计算又变得有意义了。
当您以二进制模式 ( open(..., 'rb'))打开文件时，您不会解码字节，而是按原样处理它们。如果文件中有文本（因为为了处理数据，您迟早会希望将其转换为字符串，然后您必须在其中进行解码），这是有问题的，但是如果它确实是二进制数据，例如作为图像，它很好（并且更好）。
这个答案包含简化。谨慎使用。