YARN DRF的解释

Question

我正在阅读第4版的"Hadoop The Definitive Guide",并且对YARN'S DRF(第4章,Dominant Resource Fairness)中发现了这个解释

想象一下,一个总共有100个CPU和10 TB内存的集群.应用程序A请求(2个CPU,300 GB)的容器,而应用程序B请求容器(6个CPU,100 GB).A的请求是群集的(2%,3%),因此内存占主导地位,因为其比例(3%)大于CPU(2%).B的要求是(6%,1%),因此CPU占主导地位.由于B的容器请求是主导资源的两倍(6%对3%),因此在公平共享下将分配一半的容器.

我无法理解的意思it will be allocated half as many containers under fair sharing.我猜it这里是Application B,并Application B分配了应用程序A的容器数量的一半.这样对吗？为什么Application B即使需要更多资源,也会分配较小的容器？

对某些解释文件的任何建议和指示都会受到如此多的赞赏.先感谢您.

Answer 1

主导资源计算器基于主导资源公平(DRF)的概念.

要了解DRF,您可以参考以下文章:https://people.eecs.berkeley.edu/~alig/papers/drf.pdf

在本文中,请参阅第4.1节,其中给出了一个示例.

DRF尝试均衡主导份额(A的内存要求= B的CPU要求).

说明

Total Resouces Available:100个CPU,10000 GB内存

Requirements of Application A:2个CPU,300 GB内存

Requirements of Application B:6个CPU,100 GB内存

A's dominant resource is Memory (2%的CPU与3%的内存)

B's dominant resource is CPU (6%的CPU与1%的内存)

假设"A"被分配x容器而"B"被分配y容器.

1. A的资源要求

   2x CPUs + 300x GB Memory (2 CPUs and 300 GB Memory for each container)
   

2. B的资源要求:

   6y CPUs + 100y GB Memory (6 CPUs and 100 GB Memory for each container)
   

3. 总要求是:

   2x + 6y <= 100 CPUs
   
   300x + 100y <= 10000 GB Memory
   

4. DRF将尝试平衡A和B的主要需求.

   A's dominant need: 300x / 10000 GB (300x out of 10000 GB of total memory)
   
   B's dominant need: 6y / 100 CPUs (6y out of 100 CPUs)
   
   DRF will try to equalise: (300x / 10000) = (6y / 100)
   
   Solving the above equation gives: x = 2y
   

如果x = 2y在步骤3中替换并求解方程式,则得到x = 20且y = 10.

它的意思是:

• Application A is allocated 20 containers: (40 CPUs, 6000 GB of Memory)
• Application B is allocated 10 containers: (60 CPUs, 1000 GB of memoty)

你可以看到:

Total allocated CPU is: 40 + 60 <= 100个CPU可用

Total allocated Memory is: 6000 + 1000 <= 10000 GB的内存可用

所以,上面的解决方案解释了句子的含义:

Since B’s container requests are twice as big in the dominant resource (6% 
versus 3%), it will be allocated half as many containers under fair sharing.