为什么多线程代码在更快的机器上运行得更慢？

Question

考虑以下c ++代码:

#include "threadpool.hpp"
#include <chrono>
#include <list>
#include <iostream>
#include <cmath>

int loop_size;

void process(int num) {
    double x = 0;
    double sum = 0;
    for(int i = 0; i < loop_size; ++i) {
        x += 0.0001;
        sum += sin(x) / cos(x) + cos(x) * cos(x);
    }
}

int main(int argc, char* argv[]) {
    if(argc < 3) {
        std::cerr << argv[0] << " [thread_pool_size] [threads] [sleep_time]" << std::endl;
        exit(0);
    }
    thread_pool* pool = nullptr;
    int th_count = std::atoi(argv[1]);
    if(th_count != 0) {
        pool = new thread_pool(th_count);
    }
    loop_size = std::stoi(argv[3]);
    int max = std::stoi(argv[2]);
    auto then = std::chrono::steady_clock::now();
    std::list<std::thread> ths;
    if(th_count == 0) {
        for(int i = 0; i < max; ++i) {
            ths.emplace_back(&process, i);
        }
        for(std::thread& t : ths) {
            t.join();
        }
    } else {
        for(int i = 0; i < max; ++i) {
            pool->enqueue(std::bind(&process, i));
        }
        delete pool;
    }
    int diff = std::chrono::duration_cast<std::chrono::milliseconds>(std::chrono::steady_clock::now() - then).count();
    std::cerr << "Time: " << diff << '\n';
    return 0;
}

并且"threadpool.hpp"是这个github repo的修改版本, 它可以在这里找到

我在我的机器(Corei7-6700)和88核服务器(2x Xeon E5-2696 v4)上编译了上面的代码.结果我无法解释.

这是我运行代码的方式:

tp <threadpool size> <number of threads> <iterations>

同样的代码在更快的机器上运行得更慢!我的本地计算机上有8个核心,远程服务器上有88个核心,结果如下:(最后两列表示每台计算机的平均完成时间,以毫秒为单位)

+============+=========+============+=============+====================+
| Threadpool | Threads | Iterations | Corei7-6700 | 2x Xeon E5-2696 v4 |
+============+=========+============+=============+====================+
|        100 |  100000 |       1000 |        1300 |               6000 |
+------------+---------+------------+-------------+--------------------+
|       1000 |  100000 |       1000 |        1400 |               5000 |
+------------+---------+------------+-------------+--------------------+
|      10000 |  100000 |       1000 |        1470 |               3400 |
+------------+---------+------------+-------------+--------------------+

似乎拥有更多内核会使代码运行得更慢.因此,我将服务器(任务集)上的CPU关联性降低到8个核心并再次运行代码:

taskset 0-7 tp <threadpool size> <number of threads> <iterations>

这是新数据:

+============+=========+============+=============+====================+
| Threadpool | Threads | Iterations | Corei7-6700 | 2x Xeon E5-2696 v4 |
+============+=========+============+=============+====================+
|        100 |  100000 |       1000 |        1300 |                900 |
+------------+---------+------------+-------------+--------------------+
|       1000 |  100000 |       1000 |        1400 |               1000 |
+------------+---------+------------+-------------+--------------------+
|      10000 |  100000 |       1000 |        1470 |               1070 |
+------------+---------+------------+-------------+--------------------+

我在32核Xeon和22核的Xeon机器上测试过相同的代码,模式类似:内核更少,使多线程代码运行得更快.但为什么？

重要说明:这是为了解决我原来的问题:

为什么拥有更多更快的内核会使我的多线程软件更慢？

笔记:

1. 所有机器上的操作系统和编译器都是相同的:debian 9.0 amd64运行内核4.0.9-3,6.3.0 20170516
2. 没有额外的闪存,默认优化: g++ ./threadpool.cpp -o ./tp -lpthread

Answer 1

通常,对于像这样的CPU绑定代码,您不应期望在池中运行更多线程比使用核心执行它们有任何好处.

例如,将池与1, 2, ... N/2 ... N ... N*2N核心套接字的线程进行比较可能会很有趣.具有10*N个线程的池实际上只是测试调度程序在负载下的行为方式.

然后,通常,您还需要了解每个任务的开销:将工作分成的任务越多,创建,销毁和同步对这些任务的访问所花费的时间就越多.改变固定工作量的子任务大小是一个很好的方法来看到这一点.

最后,它有助于了解您正在使用的物理架构.NUMA服务器平台可以使用两个套接字完成两倍的工作,因为相同的单个CPU可以单独执行 - 如果每个套接字只访问其自己的直接连接的内存.一旦您通过QPI传输数据,性能就会下降.在整个QPI中像你的互斥锁一样反弹一致的高速缓存行可以减慢整个事情的速度.

同样,如果您有N个内核并且想要在池中运行N个线程 - 您知道它们是物理内核还是超线程逻辑内核？如果他们是HT,你知道你的线程是否能够全速运行,或者他们是否会争夺有限的共享资源？

Answer 2

您正在将大量工作者排入线程池,这需要很少的时间来执行.因此,您通过线程池(而不是实际工作)的实现而受到瓶颈,特别是其互斥体处理争用的方式.我试图替换thread_pool与folly::CPUThreadPoolExecutor,哪一种帮助:

thread_pool version:
2180 ms | thread_pool_size=100   num_workers=100000 loop_size=1000 affinity=0-23
2270 ms | thread_pool_size=1000  num_workers=100000 loop_size=1000 affinity=0-23
2400 ms | thread_pool_size=10000 num_workers=100000 loop_size=1000 affinity=0-23
 530 ms | thread_pool_size=100   num_workers=100000 loop_size=1000 affinity=0-7
1930 ms | thread_pool_size=1000  num_workers=100000 loop_size=1000 affinity=0-7
2300 ms | thread_pool_size=10000 num_workers=100000 loop_size=1000 affinity=0-7
folly::CPUThreadPoolExecutor version:
 830 ms | thread_pool_size=100   num_workers=100000 loop_size=1000 affinity=0-23
 780 ms | thread_pool_size=1000  num_workers=100000 loop_size=1000 affinity=0-23
 800 ms | thread_pool_size=10000 num_workers=100000 loop_size=1000 affinity=0-23
 880 ms | thread_pool_size=100   num_workers=100000 loop_size=1000 affinity=0-7
1130 ms | thread_pool_size=1000  num_workers=100000 loop_size=1000 affinity=0-7
1120 ms | thread_pool_size=10000 num_workers=100000 loop_size=1000 affinity=0-7

我建议你(1)在每个线程中做更多工作; (2)使用与CPU一样多的线程; (3)使用更好的线程池.让我们设置thread_pool_sizeCPU的数量,然后乘以loop_size10:

thread_pool version:
1880 ms | thread_pool_size=24 num_workers=100000 loop_size=10000 affinity=0-23
4100 ms | thread_pool_size=8  num_workers=100000 loop_size=10000 affinity=0-7
folly::CPUThreadPoolExecutor version:
1520 ms | thread_pool_size=24 num_workers=100000 loop_size=10000 affinity=0-23
2310 ms | thread_pool_size=8  num_workers=100000 loop_size=10000 affinity=0-7

请注意,通过将每个线程的工作量增加10倍,我们实际上使thread_pool版本更快,并且folly::CPUThreadPoolExecutor版本只花费了2倍的时间.让我们乘以loop_size10倍:

thread_pool version:
28695 ms | thread_pool_size=24 num_workers=100000 loop_size=100000 affinity=0-23
81600 ms | thread_pool_size=8  num_workers=100000 loop_size=100000 affinity=0-7
folly::CPUThreadPoolExecutor version:
 6830 ms | thread_pool_size=24 num_workers=100000 loop_size=100000 affinity=0-23
14400 ms | thread_pool_size=8  num_workers=100000 loop_size=100000 affinity=0-7

因为folly::CPUThreadPoolExecutor结果不言而喻:在每个线程中做更多的工作会让你更接近并行性的真正线性收益.而thread_pool只是似乎没有达到任务; 它无法正确处理这种互斥争用的规模.

这是我用来测试的代码(用gcc 5.5编译,完全优化):

#include <chrono>
#include <cmath>
#include <iostream>
#include <memory>
#include <vector>

#define USE_FOLLY 1

#if USE_FOLLY
#include <folly/executors/CPUThreadPoolExecutor.h>
#include <folly/futures/Future.h>
#else
#include "threadpool.hpp"
#endif

int loop_size;
thread_local double dummy = 0.0;

void process(int num) {
  double x = 0;
  double sum = 0;
  for (int i = 0; i < loop_size; ++i) {
    x += 0.0001;
    sum += sin(x) / cos(x) + cos(x) * cos(x);
  }
  dummy += sum; // prevent optimization
}

int main(int argc, char* argv[]) {
  if (argc < 3) {
    std::cerr << argv[0] << " [thread_pool_size] [threads] [sleep_time]"
              << std::endl;
    exit(0);
  }
  int th_count = std::atoi(argv[1]);
#if USE_FOLLY
  auto executor = std::make_unique<folly::CPUThreadPoolExecutor>(th_count);
#else
  auto pool = std::make_unique<thread_pool>(th_count);
#endif
  loop_size = std::stoi(argv[3]);
  int max = std::stoi(argv[2]);

  auto then = std::chrono::steady_clock::now();
#if USE_FOLLY
  std::vector<folly::Future<folly::Unit>> futs;
  for (int i = 0; i < max; ++i) {
    futs.emplace_back(folly::via(executor.get()).then([i]() { process(i); }));
  }
  folly::collectAll(futs).get();
#else
  for (int i = 0; i < max; ++i) {
    pool->enqueue([i]() { process(i); });
  }
  pool = nullptr;
#endif

  int diff = std::chrono::duration_cast<std::chrono::milliseconds>(
                 std::chrono::steady_clock::now() - then)
                 .count();
  std::cerr << "Time: " << diff << '\n';
  return 0;
}