Valgrind 1017 calloc/memcopy struct dirent OK,1018th - 无效读取

Question

好,我不知道我已经打什么限制或是否这是一个问题与valgrind,libc或me,但我需要知道这是可重复的,如果是这样,在这个问题所在.我把问题归结为我的两个AMD机箱上的MCVE产品.基本上,我动态分配struct dirent *指针,然后struct dirent为每个成功分配一个 readdir.valgrind 没有投诉1017,但随后数字1018,我得到一个invalid read错误(没有涉及重新分配),例如

==9881== Invalid read of size 8
==9881==    at 0x4C2F316: memcpy@@GLIBC_2.14 (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==9881==    by 0x40098E: main (readdir_mcve.c:35)
==9881==  Address 0x51df070 is 0 bytes after a block of size 32,816 alloc'd
==9881==    at 0x4C2ABD0: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==9881==    by 0x4EE93E3: __alloc_dir (in /usr/lib/libc-2.23.so)
==9881==    by 0x4EE94D2: opendir_tail (in /usr/lib/libc-2.23.so)
==9881==    by 0x400802: main (readdir_mcve.c:9)

(block of size 32,816看起来好奇,但我没有找到任何帮助打破它)

代码将目录名作为第一个参数打开,然后将要读取的文件的限制作为第二个参数(默认为1000):

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/types.h>
#include <dirent.h>

int main (int argc, char **argv) {

    DIR *dp = opendir (argc > 1 ? argv[1] : "."); /* open directory (. default) */
    struct dirent *de = NULL, **dlist = NULL;     /* ptr and ptr2ptr to dirent  */
    size_t nptrs = argc > 2 ? (size_t)strtoul (argv[2], NULL, 10) : 1000,
           i = 0, idx = 0;                        /* index, allocation counter  */

    if (!dp) {
        fprintf (stderr, "error: opendir failed.\n");
        return 1;
    }

    /* allocate nptrs dirent pointers  */
    if (!(dlist = calloc (nptrs, sizeof *dlist))) {
        fprintf (stderr, "error: virtual memory exhausted - dlist\n");
        return 1;
    }

    while ((de = readdir (dp))) {

        /* skip dot files */
        if (!strcmp (de->d_name, ".") || !strcmp (de->d_name, ".."))
            continue;

        if (!(dlist[idx] = calloc (1, sizeof **dlist))) { /* alloc dirent */
            fprintf (stderr, "error: dlist memory allocation failed\n");
            return 1;
        }
        memcpy (dlist[idx++], de, sizeof *de);   /* copy de to dlist[idx] */

        if (idx == nptrs)   /* post-check/realloc, insures sentinel NULL */
            break;
    }
    closedir (dp);

    for (i = 0; i < idx; i++) {
        printf (" file[%3zu] : %s\n", i, dlist[i]->d_name);
        free (dlist[i]);
    }
    free (dlist);

    return 0;
}

您可以创建一个简单的测试目录:

$ mkdir readdir_tst
$ for i in {1..1024}; do
  printf -v fname "file%04d" "$i"
  touch "readdir_tst/$fname"
  done

然后一切都很好读取1014文件名(1017分配):

$ valgrind ./bin/readdir_mcve readdir_tst 1014 > /dev/null
==9880== Memcheck, a memory error detector
==9880== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
==9880== Using Valgrind-3.11.0 and LibVEX; rerun with -h for copyright info
==9880== Command: ./bin/readdir_mcve readdir_tst 1014
==9880==
==9880==
==9880== HEAP SUMMARY:
==9880==     in use at exit: 0 bytes in 0 blocks
==9880==   total heap usage: 1,017 allocs, 1,017 frees, 328,944 bytes allocated
==9880==
==9880== All heap blocks were freed -- no leaks are possible
==9880==
==9880== For counts of detected and suppressed errors, rerun with: -v
==9880== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

但在档案1015(分配1018)我遇到了一个__alloc_dir问题,valgrind抛出一个Invalid read of size 8 ... is 0 bytes after a block of size 32,816 alloc'd:

$ valgrind ./bin/readdir_mcve readdir_tst 1015 > /dev/null
==9881== Memcheck, a memory error detector
==9881== Copyright (C) 2002-2015, and GNU GPL'd, by Julian Seward et al.
==9881== Using Valgrind-3.11.0 and LibVEX; rerun with -h for copyright info
==9881== Command: ./bin/readdir_mcve readdir_tst 1015
==9881==
==9881== Invalid read of size 8
==9881==    at 0x4C2F316: memcpy@@GLIBC_2.14 (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==9881==    by 0x40098E: main (readdir_mcve.c:35)
==9881==  Address 0x51df070 is 0 bytes after a block of size 32,816 alloc'd
==9881==    at 0x4C2ABD0: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==9881==    by 0x4EE93E3: __alloc_dir (in /usr/lib/libc-2.23.so)
==9881==    by 0x4EE94D2: opendir_tail (in /usr/lib/libc-2.23.so)
==9881==    by 0x400802: main (readdir_mcve.c:9)
==9881==
==9881==
==9881== HEAP SUMMARY:
==9881==     in use at exit: 0 bytes in 0 blocks
==9881==   total heap usage: 1,018 allocs, 1,018 frees, 329,232 bytes allocated
==9881==
==9881== All heap blocks were freed -- no leaks are possible
==9881==
==9881== For counts of detected and suppressed errors, rerun with: -v
==9881== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)

代码继续读取并打印所有目录条目就好了,但这是valgrind我感到困惑的错误.我让它重新分配,而不是呼吁break达到分配限制,它处理4000+文件/usr/bin没有任何问题,除了valgrind错误.(被剥离与MCVE无关).我在SO上发现的最接近的是Valgrind malloc泄漏,但这不适用于此.任何人都可以重现这个,如果是这样,是这个valgrind,libc还是me？

注意:我得到的结果相同libc-2.18.so.

---

GNU libc dirent.h提供了更多信息

在通过答案指向正确的方向并继续搜索之后,似乎libc 可以通过多种方式确定其长度d_name.它取决于编译器可用的各种定义.在dirent.h中解释说:

46 /* This file defines `struct dirent'.
47 
48    It defines the macro `_DIRENT_HAVE_D_NAMLEN' iff there is a `d_namlen'
49    member that gives the length of `d_name'.
...
59 */
...
67 /* These macros extract size information from a `struct dirent *'.
68    They may evaluate their argument multiple times, so it must not
69    have side effects.  Each of these may involve a relatively costly
70    call to `strlen' on some systems, so these values should be cached.
71 
72    _D_EXACT_NAMLEN (DP) returns the length of DP->d_name, not including
73    its terminating null character.
74 
75    _D_ALLOC_NAMLEN (DP) returns a size at least (_D_EXACT_NAMLEN (DP) + 1);
76    that is, the allocation size needed to hold the DP->d_name string.
77    Use this macro when you don't need the exact length, just an upper bound.
78    This macro is less likely to require calling `strlen' than _D_EXACT_NAMLEN.
79    */
80 
81 #ifdef _DIRENT_HAVE_D_NAMLEN
82 # define _D_EXACT_NAMLEN(d) ((d)->d_namlen)
83 # define _D_ALLOC_NAMLEN(d) (_D_EXACT_NAMLEN (d) + 1)
84 #else
85 # define _D_EXACT_NAMLEN(d) (strlen ((d)->d_name))
86 # ifdef _DIRENT_HAVE_D_RECLEN
87 #  define _D_ALLOC_NAMLEN(d) (((char *) (d) + (d)->d_reclen) - &(d)->d_name[0])
88 # else
89 #  define _D_ALLOC_NAMLEN(d) (sizeof (d)->d_name > 1 ? sizeof (d)->d_name : \
90                               _D_EXACT_NAMLEN (d) + 1)
91 # endif
92 #endif
...

虽然有许多不同的编译路径可以根据各种定义来实现,但如果__USE_XOPEN2K8设置了,那么最易读的路径就会出现:

221 #ifdef __USE_XOPEN2K8
222 ...
230 # ifdef __USE_MISC
231 #  ifndef MAXNAMLEN
232 /* Get the definitions of the POSIX.1 limits.  */
233 #  include <bits/posix1_lim.h>
234 
235 /* `MAXNAMLEN' is the BSD name for what POSIX calls `NAME_MAX'.  */
236 #   ifdef NAME_MAX
237 #    define MAXNAMLEN   NAME_MAX
238 #   else
239 #    define MAXNAMLEN   255
240 #   endif
241 #  endif
242 # endif

因此,在这种情况下,d_name或者是NAME_MAX或255取决于NAME_MAX定义(并且因此设置为256通过_D_ALLOC_NAMLEN (DP)宏).感谢放松,让我指向正确的方向.我不知道,如果我们永远不会明白为什么确切的答案1017 struct dirent分配出现没有问题,为什么valgrind开始抱怨上的数字1018,但至少我们现在明白了其中的问题的来源是,为什么复制struct dirent与memcpy可能造成的问题.

Answer 1

你不能这样复制strucft dirent,似乎手册页和代码不同步.

这是当前的声明:

struct dirent
  {
#ifndef __USE_FILE_OFFSET64
    __ino_t d_ino;      /* File serial number.  */
#else
    __ino64_t d_ino;
#endif
    unsigned short int d_reclen; /* Length of the whole `struct dirent'.  */
    unsigned char d_type;   /* File type, possibly unknown.  */
    unsigned char d_namlen; /* Length of the file name.  */

    /* Only this member is in the POSIX standard.  */
    char d_name[1];     /* File name (actually longer).  */
  };

很明显,因为d_name声明[1],你将无法使用适当的大小sizeof.你需要做更聪明的存储,即strdup()名称或东西(如果你只对名字感兴趣).

我不是百分之百确定为什么这会导致破损,但我敢打赌你会看到某种UB(注意你在阅读时死亡,如果你到达printf()了复制的字符串会触发UB).