我应该使用ThreadPools或任务并行库进行IO绑定操作

Question

在我的一个有点聚合的项目中,我从网上解析了feed,podcast等.

如果我使用顺序方法,考虑到大量资源,处理所有资源需要相当长的时间(因为网络问题和类似的东西);

foreach(feed in feeds)
{
   read_from_web(feed)
   parse(feed)
}

所以我想实现并发,并且无法决定我是否应该使用ThreadPools来处理工作线程,或者只是依靠TPL来对它进行排序.

ThreadPools肯定会用工作线程处理我的工作,我会得到我所期望的(在多核CPU环境中,其他核心也将被利用).

但我仍然想考虑TPL,因为它推荐的方法,但我有点担心它.首先,我知道TPL使用ThreadPools,但增加了额外的决策层.我主要关心的是单核环境存在的情况.如果我没错,TPL从一开始就用一个数量的工作线程开始,这些工作线程数等于可用的CPU核心数.我担心TPL会产生与我的IO绑定案例的顺序方法类似的结果.

因此,对于IO绑定操作(在我的情况下从Web读取资源),最好是使用ThreadPools来控制事物,还是更好地依赖于TPL？TPL也可以用于IO绑定场景吗？

更新:我主要担心的是 - 在单核CPU环境中,TPL只是表现得像顺序方式,还是会提供并发性？我已经阅读了使用Microsoft .NET的并行编程,所以这本书却找不到确切的答案.

注意:这是我之前的问题的重新措辞[ 是否可以将线程并发和并行一起使用？这是错误的措辞.

Answer 1

所以我决定为此编写测试并在实际数据上看到它.

测试传奇

• Itr:迭代
• Seq:顺序方法.
• PrlEx:Parallel Extensions - Parallel.ForEach
• TPL:任务并行库
• TPool:ThreadPool

检测结果

单核CPU [Win7-32​​] - 在VMWare下运行 -

Test Environment: 1 physical cpus, 1 cores, 1 logical cpus.
Will be parsing a total of 10 feeds.
________________________________________________________________________________

Itr.    Seq.    PrlEx   TPL     TPool
________________________________________________________________________________

#1      10.82s  04.05s  02.69s  02.60s
#2      07.48s  03.18s  03.17s  02.91s
#3      07.66s  03.21s  01.90s  01.68s
#4      07.43s  01.65s  01.70s  01.76s
#5      07.81s  02.20s  01.75s  01.71s
#6      07.67s  03.25s  01.97s  01.63s
#7      08.14s  01.77s  01.72s  02.66s
#8      08.04s  03.01s  02.03s  01.75s
#9      08.80s  01.71s  01.67s  01.75s
#10     10.19s  02.23s  01.62s  01.74s
________________________________________________________________________________

Avg.    08.40s  02.63s  02.02s  02.02s
________________________________________________________________________________

单核CPU [WinXP] - 在VMWare下运行 -

Test Environment: 1 physical cpus, NotSupported cores, NotSupported logical cpus.
Will be parsing a total of 10 feeds.
________________________________________________________________________________

Itr.    Seq.    PrlEx   TPL     TPool
________________________________________________________________________________

#1      10.79s  04.05s  02.75s  02.13s
#2      07.53s  02.84s  02.08s  02.07s
#3      07.79s  03.74s  02.04s  02.07s
#4      08.28s  02.88s  02.73s  03.43s
#5      07.55s  02.59s  03.99s  03.19s
#6      07.50s  02.90s  02.83s  02.29s
#7      07.80s  04.32s  02.78s  02.67s
#8      07.65s  03.10s  02.07s  02.53s
#9      10.70s  02.61s  02.04s  02.10s
#10     08.98s  02.88s  02.09s  02.16s
________________________________________________________________________________

Avg.    08.46s  03.19s  02.54s  02.46s
________________________________________________________________________________

双核CPU [Win7-64]

Test Environment: 1 physical cpus, 2 cores, 2 logical cpus.
Will be parsing a total of 10 feeds.
________________________________________________________________________________

Itr.    Seq.    PrlEx   TPL     TPool
________________________________________________________________________________

#1      07.09s  02.28s  02.64s  01.79s
#2      06.04s  02.53s  01.96s  01.94s
#3      05.84s  02.18s  02.08s  02.34s
#4      06.00s  01.43s  01.69s  01.43s
#5      05.74s  01.61s  01.36s  01.49s
#6      05.92s  01.59s  01.73s  01.50s
#7      06.09s  01.44s  02.14s  02.37s
#8      06.37s  01.34s  01.46s  01.36s
#9      06.57s  01.30s  01.58s  01.67s
#10     06.06s  01.95s  02.88s  01.62s
________________________________________________________________________________

Avg.    06.17s  01.76s  01.95s  01.75s
________________________________________________________________________________

四核CPU [Win7-64] - 支持HyprerThreading -

Test Environment: 1 physical cpus, 4 cores, 8 logical cpus.
Will be parsing a total of 10 feeds.
________________________________________________________________________________

Itr.    Seq.    PrlEx   TPL     TPool
________________________________________________________________________________

#1      10.56s  02.03s  01.71s  01.69s
#2      07.42s  01.63s  01.71s  01.69s
#3      11.66s  01.69s  01.73s  01.61s
#4      07.52s  01.77s  01.63s  01.65s
#5      07.69s  02.32s  01.67s  01.62s
#6      07.31s  01.64s  01.53s  02.17s
#7      07.44s  02.56s  02.35s  02.31s
#8      08.36s  01.93s  01.73s  01.66s
#9      07.92s  02.15s  01.72s  01.65s
#10     07.60s  02.14s  01.68s  01.68s
________________________________________________________________________________

Avg.    08.35s  01.99s  01.75s  01.77s
________________________________________________________________________________

概要

• 无论您是在单核环境还是多核环境上运行,Parallel Extensions,TPL和ThreadPool的行为都相同,并提供近似结果.
• 仍然TPL具有诸如简单的异常处理,取消支持和轻松返回任务结果的能力等优点.虽然Parallel Extensions也是另一种可行的选择.

自己运行测试

您可以在此处下载源代码并自行运行.如果你可以发布结果,我也会添加它们.

更新:修复了源链接.

Answer 2

如果您正在尝试最大化IO绑定任务的吞吐量,则绝对必须将传统的异步处理模型(APM)API与基于TPL的工作相结合.在异步IO回调挂起时,APM API是解除CPU线程的唯一方法.第三方物流提供的TaskFactory::FromAsync辅助方法,以帮助相结合APM和TPL代码.

查看MSDN上名为TPL和Traditional .NET异步编程的.NET SDK的这一部分,了解有关如何组合这两种编程模型以实现异步天堂的更多信息.