记录 Linux 进程的 RAM 内存高水位线

Question

我想在进程完成时记录 Linux 进程（内核 3.2.0-36）的 RAM 内存高水位标记。我在 Perl 脚本中调用该进程。就像是：

my $cmd = "logmemory -o mem.log mycmd options 1>cmd.out 2>cmd.err";
unless(system("$cmd") == 0) { die $!; }

有任何想法吗？

Answer 1

看看/proc/[pid]/status，特别是这个参数。

• VmHWM：峰值驻留集大小（“高水位线”）。

或者，您可以使用/usr/bin/time -v命令。这是它的一个例子：

Command exited with non-zero status 1
    Command being timed: "xz -9ek access_log.3 access_log.xz"
    User time (seconds): 6.96
    System time (seconds): 0.34
    Percent of CPU this job got: 99%
    Elapsed (wall clock) time (h:mm:ss or m:ss): 0:07.34
    Average shared text size (kbytes): 0
    Average unshared data size (kbytes): 0
    Average stack size (kbytes): 0
    Average total size (kbytes): 0
  **Maximum resident set size (kbytes): 383456**
    Average resident set size (kbytes): 0
    Major (requiring I/O) page faults: 0
    Minor (reclaiming a frame) page faults: 24000
    Voluntary context switches: 3
    Involuntary context switches: 225
    Swaps: 0
    File system inputs: 0
    File system outputs: 0
    Socket messages sent: 0
    Socket messages received: 0
    Signals delivered: 0
    Page size (bytes): 4096
    Exit status: 1

Answer 2

内核已经为您收集了进程的 RAM 高水位标记信息（来自man proc）：

/proc/[pid]/status
Provides much of the information in /proc/[pid]/stat and /proc/[pid]/statm in a format that's easier for humans to parse.
(...)
* VmHWM: Peak resident set size ("high water mark").
(...)

棘手的部分是该值应在进程终止前立即读取。

我尝试了不同的方法（答案末尾有更多内容），对我有用的方法是在 C 中实现：

• logmemory调用fork()以创建子进程。

• 子进程调用，ptrace()以便在子进程logmemory每次执行系统调用时通知父进程（即）。

• 子进程用于execvp()运行mycmd.

• logmemory耐心等待通知。在这种情况下，它会检查是否已mycmd调用exit_group。如果是这种情况，它会读取/proc/<pid>/status、将值复制到子级mem.log并与子级分离。否则，logmemory允许mycmd继续并等待下一个通知。

缺点是ptrace()监控程序变慢，我在下面展示了一些比较。

此版本logmemory不仅记录，VmHWM而且：

• VmPeak （峰值虚拟内存大小，包括所有代码、数据和共享库以及已换出的页面和已映射但未使用的页面）

• 时间戳

• 命令名称和参数

这是代码，肯定可以改进 - 我不精通 C。不过它按预期工作（在 32 位 Ubuntu 12.04 和 64 位 SuSE Linux Enterprise Server 10 SP4 上测试）：

// logmemory.c
#include <stdio.h>
#include <sys/ptrace.h>
#include <unistd.h>
#include <syscall.h>
#include <sys/reg.h>
#include <stdlib.h>
#include <string.h>
#include <time.h>

#define STRINGLENGTH 2048

int main(int argc, char **argv)
{   
    pid_t child_pid;
    long syscall;
    int status, index;
    FILE *statusfile, *logfile;
    char opt, statusfile_path[STRINGLENGTH], line[STRINGLENGTH], command[STRINGLENGTH], logfile_path[STRINGLENGTH] = "";
    time_t now;
    extern char *optarg;
    extern int optind;

    // Error checking
    if (argc == 1) {
        printf("Error: program to execute is missing. Exiting...\n");
        return 0;
    }
    // Get options
    while ((opt = getopt (argc, argv, "+o:")) != -1)
        switch (opt) {
            case 'o':
                strncpy(logfile_path, optarg, 2048);
                break;
            case ':':
                fprintf (stderr, "Aborting: argument for option -o is missing\n");
                return 1;
            case '?':
                fprintf (stderr, "Aborting: only valid option is -o\n");
                return 1;
    }
    // More error checking
    if (!strcmp(logfile_path, "")) {
        fprintf(stderr, "Error: log filename can't be empty\n");
        return 1;
    }
    child_pid = fork();
    // The child process executes this:
    if (child_pid == 0) {
        // Trace child process:
        ptrace(PTRACE_TRACEME, 0, NULL, NULL);
        // Execute command using $PATH
        execvp(argv[optind], (char * const *)(argv+optind));

    // The parent process executes this:
    } else {
        // Loop until child process terminates
        do {
            // Set ptrace to stop when syscall is executed
            ptrace(PTRACE_SYSCALL, child_pid, NULL, NULL);
            wait(&status);
            // Get syscall number
            syscall = ptrace(PTRACE_PEEKUSER, child_pid,
#ifdef __i386__
                          4 * ORIG_EAX,
#else
                          8 * ORIG_RAX,
#endif
                          NULL);
        } while (syscall != SYS_exit_group);

        // Construct path to status file and check whether status and log file can be opened
        snprintf(statusfile_path, STRINGLENGTH, "/proc/%d/status", child_pid);
        if ( !(logfile = fopen(logfile_path, "a+")) || !(statusfile = fopen(statusfile_path, "r")) ) {
            ptrace(PTRACE_DETACH, child_pid, NULL, NULL);
            return 1;
        }

        // Copy timestamp and command to logfile
        now = time(NULL);
        fprintf(logfile, "Date: %sCmd: ", asctime(localtime(&now)));
        for (index = optind; index < argc; index++)
           fprintf(logfile, " %s", argv[index]);
        fprintf(logfile, "\n");

        // Read status file line by line and copy lines containing VmPeak and VmHWM to logfile
        while (fgets(line, STRINGLENGTH, statusfile)) {
            if (strstr(line,"VmPeak") || strstr(line,"VmHWM"))
                fprintf(logfile, "%s", line);
        }
        fprintf(logfile, "\n");

        // Close files
        fclose(statusfile);
        fclose(logfile);

        // Detach from child process
        ptrace(PTRACE_DETACH, child_pid, NULL, NULL);
    }
    return 0;
}

将其另存为logmemory.c并像这样编译：

$ gcc logmemory.c -o logmemory

像这样运行它：

$ ./logmemory 
Error: program to execute is missing. Exiting...
$ ./logmemory -o mem.log ls -l
(...)
$ ./logmemory -o mem.log free
             total       used       free     shared    buffers     cached
Mem:       1025144     760660     264484          0       6644     143980
-/+ buffers/cache:     610036     415108
Swap:      1046524     544228     502296
$ ./logmemory -o mem.log find /tmp -name \*txt
(...)
$ cat mem.log
Date: Mon Feb 11 21:17:55 2013
Cmd:  ls -l
VmPeak:     5004 kB
VmHWM:      1284 kB

Date: Mon Feb 11 21:18:01 2013
Cmd:  free
VmPeak:     2288 kB
VmHWM:       448 kB

Date: Mon Feb 11 21:18:26 2013
Cmd:  find /tmp -name *txt
VmPeak:     4700 kB
VmHWM:       908 kB

我写了这个 C 程序来测试logmemory的准确性：

// bigmalloc.c
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#define ITERATIONS 200
int main(int argc, char **argv)
{
    int i=0;
    for (i=0; i<ITERATIONS; i++) {
        void *m = malloc(1024*1024);
        memset(m,0,1024*1024);
    }
    return 0;
}

像往常一样编译并在里面运行logmemory：

$ gcc bigmalloc.c -o bigmalloc
$ ./logmemory -o mem.log ./bigmalloc
$ tail mem.log

Date: Mon Feb 11 21:26:01 2013
Cmd:  ./bigmalloc
VmPeak:   207604 kB
VmHWM:    205932 kB

正确报告使用了 200 MB。

附带说明：（time至少在 Ubuntu 12.04 上）令人惊讶地输出一个与内核报告的值大不相同的值：

$ /usr/bin/time --format %M ./bigmalloc
823872

哪里M（来自man time）：

M   Maximum resident set size of the process during its lifetime, in Kilobytes.

如上所述，这是有代价的，因为logmemory会减慢受监控程序的执行速度，例如：

$ time ./logmemory -o mem.log ./bigmalloc
real    0m0.288s
user    0m0.000s
sys     0m0.004s
$ time ./bigmalloc
real    0m0.104s
user    0m0.008s
sys     0m0.092s

$ time find /var -name \*log
(...)
real    0m0.036s
user    0m0.000s
sys     0m0.032s
$ time ./logmemory -o mem.log find /var -name \*log
(...)
real    0m0.124s
user    0m0.000s
sys     0m0.052s

我（未成功）尝试的其他方法是：

• 创建后台进程以/proc/<pid>/status在mycmd运行时读取的 shell 脚本。

• 分叉和执行mycmd但暂停直到孩子成为僵尸的AC 程序，因此避免ptrace和它产生的开销。好主意，我想，遗憾的是VmHWM并VmPeak不再可以从/proc/<pid>/status一个僵尸。