在Nutch 2.x中提取HTML元标记并使用Solr 4索引它

Question

我正在使用Nutch 2.0来抓取一些网站,但我没有看到HTML元标记,如标题,描述被提取并存储在MySQL数据库中.任何想法我怎么能让它工作？

谢谢Arash

Answer 1

确保在您的定义中包含parse-metatags和index-metadata插件plugin.includesnutch-site.xml

然后添加metatags.names index.parse.md和index.content.md属性并将它们指向适当的标记.看看我的:

<property>
        <name>plugin.includes</name>
        <value>protocol-http|protocol-httpclient|urlfilter-regex|parse-(html|tika|metatags)|index-(basic|anchor|metadata)|scoring-opic|urlnormalizer-(pass|regex|basic)</value>
</property>
<property>
        <name>metatags.names</name>
        <value>*</value>
</property>
<property>
        <name>index.parse.md</name>
        <value>metatag.description,metatag.author,metatag.twitter:image</value>
</property>
<property>
        <name>index.content.md</name>
        <value>author,description,twitter:image</value>
</property>

测试您的配置.我针对readwrite.com上的一篇文章运行了这个测试:

bin/nutch indexchecker http://readwrite.com/2013/03/20/whats-behind-china-attacks-on-apple-and-android

输出将告诉您是否正在解析正确的值.就我而言,我想要author,description并且twitter:image:

fetching: http://readwrite.com/2013/03/20/whats-behind-china-attacks-on-apple-and-android
parsing: http://readwrite.com/2013/03/20/whats-behind-china-attacks-on-apple-and-android
contentType: text/html
content :   What's Really Behind China's Attacks On Apple And Android? – ReadWrite Sections Sections Social Mobi
title : What's Really Behind China's Attacks On Apple And Android? – ReadWrite
host :  readwrite.com
metatag.author :    Brian S Hall
tstamp :    Wed Mar 20 13:33:38 EDT 2013
metatag.twitter:image : http://readwrite.com/files/styles/150_150sc/public/fields/China%20graphic%20brian%20final.jpg
metatag.description :   Repeated outbursts suggest China could be growing concerned over America's dominance in the smartpho
url :   http://readwrite.com/2013/03/20/whats-behind-china-attacks-on-apple-and-android

缺点是parse-metatags只能按名称匹配标签而不是属性.例如,可以错过<meta name="foo" content="bar">Open Graph标签<meta property="og:image" content="http://readwrite.com/sample.jpg" />.