Java多线程解析器

Question

我正在编写一个多线程解析器.解析器类如下.

public class Parser extends HTMLEditorKit.ParserCallback implements Runnable {

    private static List<Station> itemList = Collections.synchronizedList(new ArrayList<Item>());
    private boolean h2Tag = false;
    private int count;
    private static int threadCount = 0;

    public static List<Item> parse() {
        for (int i = 1; i <= 1000; i++) { //1000 of the same type of pages that need to parse

            while (threadCount == 20) { //limit the number of simultaneous threads
                try {
                    Thread.sleep(50);
                } catch (InterruptedException ex) {
                    ex.printStackTrace();
                }
            }

            Thread thread = new Thread(new Parser());
            thread.setName(Integer.toString(i));
            threadCount++; //increase the number of working threads
            thread.start();            
        }

        return itemList;
    }

    public void run() {
        //Here is a piece of code responsible for creating links based on
        //the thread name and passed as a parameter remained i,
        //connection, start parsing, etc.        
        //In general, nothing special. Therefore, I won't paste it here.

        threadCount--; //reduce the number of running threads when current stops
    }

    private static void addItem(Item item) {
        itenList.add(item);
    }

    //This method retrieves the necessary information after the H2 tag is detected
    @Override
    public void handleText(char[] data, int pos) {
        if (h2Tag) {
            String itemName = new String(data).trim();

        //Item - the item on which we receive information from a Web page
        Item item = new Item();
        item.setName(itemName);
        item.setId(count);
        addItem(item);

        //Display information about an item in the console
        System.out.println(count + " = " + itemName); 
        }
    }

    @Override
    public void handleStartTag(HTML.Tag t, MutableAttributeSet a, int pos) {
        if (HTML.Tag.H2 == t) {
            h2Tag = true;
        }
    }

    @Override
    public void handleEndTag(HTML.Tag t, int pos) {
        if (HTML.Tag.H2 == t) {
            h2Tag = false;
        }
    }
}

从另一个类解析器运行如下:

List<Item> list = Parser.parse();

一切都很好,但有一个问题.在最终列表中解析结束时"List itemList"包含980个元素而不是1000.但是在控制台中有1000个元素(项目).也就是说,某些线程由于某种原因没有在handleText方法中调用addItem方法.

我已经尝试将itemList的类型更改为ArrayList,CopyOnWriteArrayList,Vector.使方法addItem同步,更改其对synchronized块的调用.所有这些只会稍微改变元素的数量,但无法获得最后的千元.

我还尝试解析少量页面(十个).结果列表为空,但在控制台中全部为10.

如果我删除多线程,那么一切正常,但当然,慢慢来.这不好.

如果减少并发线程的数量,列表中的项目数量接近期望的1000,如果增加 - 距离1000有一点距离.也就是说,我认为,记录到列表的能力很难.但那为什么同步不起作用？

有什么问题？

Answer 1

在您的parse()调用返回后,您的所有1000个线程都已启动,但无法保证它们已完成.事实上,他们并不是你看到的问题.我强烈建议不要自己写这个,而是使用SDK为这类工作提供的工具.

文档线程池和它们ThreadPoolExecutor是一个很好的起点.再说一遍,如果你不完全确定自己也有,那么不要自己实现,因为编写这样的多线程代码是很痛苦的.

您的代码应如下所示:

ExecutorService executor = Executors.newFixedThreadPool(20);
List<Future<?>> futures = new ArrayList<Future<?>>(1000);
for (int i = 0; i < 1000; i++) { 
   futures.add(executor.submit(new Runnable() {...}));
}
for (Future<?> f : futures) {
   f.get();
}