在64位系统上以低地址分配内存的最可靠/可移植方式是什么？

Question

我需要分配大块内存(由我的自定义分配器使用),它们位于虚拟地址空间的前32GB内.

我想如果我需要,比方说,1MB块,我可以从低地址开始使用mmap和MAP_FIXED_NOREPLACE(或VirtualAlloc)以1MB的增量进行迭代,直到调用成功.从最后一个成功的块继续下一个.

这听起来很笨拙,但至少它会对OS地址空间布局的变化和ASLR算法的变化有所增强.根据我对当前操作系统布局的理解,前32GB应该有足够的内存,但是我可能会遗漏一些东西？

Windows,Linux,OS X,iOS或Android中有什么东西会破坏这个方案吗？有没有更好的办法？

如果您想知道,这是为了实现编程语言的VM,在64位系统上将所有指针拟合为32位值可以提供巨大的内存使用优势甚至速度增益.由于所有对象至少为8字节对齐,因此较低的3位可以移出,将指针范围从4GB扩展到32GB.

Answer 1

对于Windows中限制分配的内存范围,我们可以使用NtAllocateVirtualMemory函数 - 这个api可用于用户和内核模式.在用户模式下,它由ntdll.dll导出(使用来自wdk的ntdll.lib或ntdllp.lib).在此api存在参数中 - ZeroBits - 在剖面视图的基址中必须为零的高阶地址位的数量.但在msdn链接中关于ZeroBits的下一个单词是不正确的.正确的是:

ZeroBits

提供截面视图基址中必须为零的高位地址位数.此参数的值必须小于或等于最大零位数,并且仅在内存管理确定分配视图的位置时使用(即,当BaseAddress为null时).

如果ZeroBits为零,则不应用零位约束.

如果ZeroBits大于0且小于32,则它是位31的前导零位数.位63:32也需要为零.这保持了与32位系统的兼容性.如果ZeroBits大于32,则将其视为掩码,然后在掩码中计算前导零的数量.然后,这成为零位参数.

所以我们真的可以使用ZeroBits作为掩码 - 这是最耗电的.但是可以使用并且作为31位的零位计数(在这种情况下,63-32位将始终等于0).因为分配粒度(当前为64kb - 0x10000) - 低16位始终为0.因此,位数模式下ZeroBits的有效值- 从1到15(= 31-16).为了更好地理解此参数的工作原理 - 查看示例代码.为了更好的演示效果,我将使用

MEM_TOP_DOWN

应基于ZeroBits在可能的最高虚拟地址创建指定区域.

PVOID BaseAddress;
ULONG_PTR ZeroBits;
SIZE_T RegionSize = 1;
NTSTATUS status;

for (ZeroBits = 0xFFFFFFFFFFFFFFFF;;)
{
    if (0 <= (status = NtAllocateVirtualMemory(NtCurrentProcess(), &(BaseAddress = 0), 
        ZeroBits, &RegionSize, MEM_RESERVE|MEM_TOP_DOWN, PAGE_NOACCESS)))
    {
        DbgPrint("%p:%p\n", ZeroBits, BaseAddress);
        NtFreeVirtualMemory(NtCurrentProcess(), &BaseAddress, &RegionSize, MEM_RELEASE);

        ZeroBits >>= 1;
    }
    else
    {
        DbgPrint("%x\n", status);
        break;
    }
}

for(ZeroBits = 0;;) 
{
    if (0 <= (status = NtAllocateVirtualMemory(NtCurrentProcess(), &(BaseAddress = 0), 
        ZeroBits, &RegionSize, MEM_RESERVE|MEM_TOP_DOWN, PAGE_NOACCESS)))
    {
        DbgPrint("%x:%p\n", ZeroBits++, BaseAddress);
        NtFreeVirtualMemory(NtCurrentProcess(), &BaseAddress, &RegionSize, MEM_RELEASE);
    }
    else
    {
        DbgPrint("%x\n", status);
        break;
    }
}

并输出:

FFFFFFFFFFFFFFFF:00007FF735B40000
7FFFFFFFFFFFFFFF:00007FF735B40000
3FFFFFFFFFFFFFFF:00007FF735B40000
1FFFFFFFFFFFFFFF:00007FF735B40000
0FFFFFFFFFFFFFFF:00007FF735B40000
07FFFFFFFFFFFFFF:00007FF735B40000
03FFFFFFFFFFFFFF:00007FF735B40000
01FFFFFFFFFFFFFF:00007FF735B40000
00FFFFFFFFFFFFFF:00007FF735B40000
007FFFFFFFFFFFFF:00007FF735B40000
003FFFFFFFFFFFFF:00007FF735B40000
001FFFFFFFFFFFFF:00007FF735B40000
000FFFFFFFFFFFFF:00007FF735B40000
0007FFFFFFFFFFFF:00007FF735B40000
0003FFFFFFFFFFFF:00007FF735B40000
0001FFFFFFFFFFFF:00007FF735B40000
0000FFFFFFFFFFFF:00007FF735B40000
00007FFFFFFFFFFF:00007FF735B40000
00003FFFFFFFFFFF:00003FFFFFFF0000
00001FFFFFFFFFFF:00001FFFFFFF0000
00000FFFFFFFFFFF:00000FFFFFFF0000
000007FFFFFFFFFF:000007FFFFFF0000
000003FFFFFFFFFF:000003FFFFFF0000
000001FFFFFFFFFF:000001FFFFFF0000
000000FFFFFFFFFF:000000FFFFFF0000
0000007FFFFFFFFF:0000007FFFFF0000
0000003FFFFFFFFF:0000003FFFFF0000
0000001FFFFFFFFF:0000001FFFFF0000
0000000FFFFFFFFF:0000000FFFFF0000
00000007FFFFFFFF:00000007FFFF0000
00000003FFFFFFFF:00000003FFFF0000
00000001FFFFFFFF:00000001FFFF0000
00000000FFFFFFFF:00000000FFFF0000
000000007FFFFFFF:000000007FFF0000
000000003FFFFFFF:000000003FFF0000
000000001FFFFFFF:000000001FFF0000
000000000FFFFFFF:000000000FFF0000
0000000007FFFFFF:0000000007FF0000
0000000003FFFFFF:0000000003FF0000
0000000001FFFFFF:0000000001FF0000
0000000000FFFFFF:0000000000FF0000
00000000007FFFFF:00000000007F0000
00000000003FFFFF:00000000003F0000
00000000001FFFFF:00000000001F0000
00000000000FFFFF:00000000000F0000
000000000007FFFF:0000000000070000
000000000003FFFF:0000000000030000
000000000001FFFF:0000000000010000
c0000017
0:00007FF735B40000
1:000000007FFF0000
2:000000003FFF0000
3:000000001FFF0000
4:000000000FFF0000
5:0000000007FF0000
6:0000000003FF0000
7:0000000001FF0000
8:0000000000FF0000
9:00000000007F0000
a:00000000003F0000
b:00000000001F0000
c:00000000000F0000
d:0000000000070000
e:0000000000030000
f:0000000000010000
c0000017

所以,如果我们想要限制内存分配32Gb(0x800000000)- 我们可以使用ZeroBits = 0x800000000 - 1:

NtAllocateVirtualMemory(NtCurrentProcess(), &(BaseAddress = 0), 
            0x800000000 - 1, &RegionSize, MEM_RESERVE|MEM_TOP_DOWN, PAGE_NOACCESS)

因此,内存将在范围内分配[0, 7FFFFFFFF](实际上[0, 7FFFF0000]因为分配粒度低16位的地址始终为0)

你可以RtlCreateHeap在分配的区域范围内创建堆并从这个堆中分配内存(注意 - 这也是用户模式api - 使用ntdll [p] .lib进行链接器输入)

PVOID BaseAddress = 0;
SIZE_T RegionSize = 0x10000000;// reserve 256Mb
if (0 <= NtAllocateVirtualMemory(NtCurrentProcess(), &BaseAddress, 
    0x800000000 - 1, &RegionSize, MEM_RESERVE, PAGE_READWRITE))
{
    if (PVOID hHeap = RtlCreateHeap(0, BaseAddress, RegionSize, 0, 0, 0))
    {
        HeapAlloc(hHeap, 0, <somesize>);
        RtlDestroyHeap(hHeap);
    }

    VirtualFree(BaseAddress, 0, MEM_RELEASE);
}