在Perl中,如何使用多个线程来处理大型数组的部分？

Question

我需要Perl中的多线程帮助.

基本逻辑是启动20个线程.我有一个数组@dataarray,我希望将20个数据块传递给每个线程.比如,其中@dataarray有200行数据,所以前10行将进入线程1,接下来10应该发送到线程2,因此它们不会覆盖彼此的数据,最终在处理线程后应该将返回结果更新为@outputarrayat与源相同的索引位置@datarray.

例如:第19行(索引位置18)@dataarray被发送到第2号线程,因此在处理之后,线程2应该更新$outputarray[18] = $processed_string.

只需要弄清楚如何从数组的位置发送到特定线程的位置.

#!/usr/bin/perl

use strict;
use threads;
my $num_of_threads = 20;
my @threads = initThreads();
my @dataarray;

foreach(@threads)
    {
    $_ = threads->create(\&doOperation);
    }

foreach(@threads)
     {
     $_->join();
     }

sub initThreads
     {
       my @initThreads;
         for(my $i = 1;$i<=$num_of_threads;$i++)
         {
         push(@initThreads,$i);
     }
     return @initThreads;
     }

sub doOperation
    {
    # Get the thread id. Allows each thread to be identified.
    my $id = threads->tid();
    # Process something--- on array chunk
    print "Thread $id done!\n";
    # Exit the thread
    threads->exit();
    }

Answer 1

我不认为我之前说过必须使用threads :: shared是正确的.我不得不去检查文档,我不确定.我不知道什么时候涉及到Perl的线程.

更新:事实证明,我的不完全理解再次被揭示出来.至少,您需要threads::shared能够@output从每个线程中放入结果.

#!/usr/bin/env perl

use strict; use warnings;
use threads;
use threads::shared;

use List::Util qw( sum );
use YAML;

use constant NUM_THREADS => 20;

my @output :shared;
my @data = ( ([1 .. 10]) x 200);

# note that you'll need different logic to handle left over
# chunks if @data is not evenly divisible by NUM_THREADS
my $chunk_size = @data / NUM_THREADS;

my @threads;

for my $chunk ( 1 .. NUM_THREADS ) {
    my $start = ($chunk - 1) * $chunk_size;
    push @threads, threads->create(
        \&doOperation,
        \@data,
        $start,
        ($start + $chunk_size - 1),
        \@output,
    );
}

$_->join for @threads;

print Dump \@output;

sub doOperation{
    my ($data, $start, $end, $output) = @_;

    my $id = threads->tid;

    print "Thread [$id] starting\n";

    for my $i ($start .. $end) {
        print "Thread [$id] processing row $i\n";
        $output->[$i] = sum @{ $data->[$i] };
        sleep 1 if 0.2 > rand;
    }

    print "Thread $id done!\n";

    return;
}

输出:

- 55
- 55
- 55
…
- 55
- 55
- 55
- 55