小编Jul*_*che的帖子

Exception in thread "main" java.lang.IllegalArgumentException: Fetcher: No agents listed in 'http.agent.name' property.
        at org.apache.nutch.fetcher.Fetcher.checkConfiguration(Fetcher.java:1166)
        at org.apache.nutch.fetcher.Fetcher.fetch(Fetcher.java:1068)
        at org.apache.nutch.crawl.Crawl.run(Crawl.java:135)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
        at org.apache.nutch.crawl.Crawl.main(Crawl.java:54)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:616)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:156)

每次我跑./nutch crawl urls -dir crawl -depth 3 -topN 5.nutch决定抛出这个错误.我有我的nutch-site.xml和nutch-default.xml设置.

 <property>
  <name>http.agent.name</name>
  <value>blah</value>
  </property>

进行了描述,使其更容易阅读.但我没有看到可以指定代理名称的其他位置.如果有人有任何建议我会很感激.

我有一个.Net爬虫在用户发出请求时运行(因此,它需要很快).它实时抓取400多个链接.(这是业务问题.)

问题:我需要检测链接是否是xml(想想rss或atom feed)或html.如果链接是xml然后我继续处理,但如果链接是html我可以跳过它.通常,我有2 xml(s)和398+ html(s).目前,我有多个线程,但处理仍然很慢,通常75秒运行,10个线程用于400多个链接,或280秒运行1个线程.(我想添加更多线程,但请参见下文..)

我面临的挑战是我如下阅读流:

var request = WebRequest.Create(requestUriString: uri.AbsoluteUri);
// ....
var response = await request.GetResponseAsync();
//....
using (var reader = new StreamReader(stream: response.GetResponseStream(), encoding: encoding)) {
                        char[] buffer = new char[1024];
                        await reader.ReadAsync(buffer: buffer, index: 0, count: 1024);
                        responseText = new string(value: buffer);
}
// parse first byts of reasponseText to check if xml 

问题是我的优化只获得1024是没用的,因为GetResponseAsync正在下载整个流,无论如何,正如我所看到的.(我的另一个选择是查找标题ContentType,但这是非常相似的AFAIK,因为我还是得到了内容 - 如果你不建议使用OPTIONS,我到目前为止没用 - 另外xml可能是内容类型错误标记(？),我将错过一些内容.)

如果有任何我失踪的优化请帮助,因为我的想法已经用完了.

(我的确考虑通过在多个服务器上分散负载来优化这种设计,这样我就可以平衡网络与并行性,但这与目前的体系结构有一些变化,我现在无法负担这一点.)

我收到以下异常

org.apache.flink.util.FlinkException: The assigned slot container_1546939492951_0001_01_003659_0 was removed.
at org.apache.flink.runtime.resourcemanager.slotmanager.SlotManager.removeSlot(SlotManager.java:789)
at org.apache.flink.runtime.resourcemanager.slotmanager.SlotManager.removeSlots(SlotManager.java:759)
at org.apache.flink.runtime.resourcemanager.slotmanager.SlotManager.internalUnregisterTaskManager(SlotManager.java:951)
at org.apache.flink.runtime.resourcemanager.slotmanager.SlotManager.unregisterTaskManager(SlotManager.java:372)
at org.apache.flink.runtime.resourcemanager.ResourceManager.closeTaskManagerConnection(ResourceManager.java:823)
at org.apache.flink.yarn.YarnResourceManager.lambda$onContainersCompleted$0(YarnResourceManager.java:346)
at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:332)
at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:158)
at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:70)
at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.onReceive(AkkaRpcActor.java:142)
at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.onReceive(FencedAkkaRpcActor.java:40)
at akka.actor.UntypedActor$$anonfun$receive$1.applyOrElse(UntypedActor.scala:165)
at akka.actor.Actor$class.aroundReceive(Actor.scala:502)
at akka.actor.UntypedActor.aroundReceive(UntypedActor.scala:95)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:526)
at akka.actor.ActorCell.invoke(ActorCell.scala:495)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:257)
at akka.dispatch.Mailbox.run(Mailbox.scala:224)
at akka.dispatch.Mailbox.exec(Mailbox.scala:234)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

在运行涉及连接两个非常大的数据集的批处理过程时。

这是我在概览中可以看到的内容。失败发生在没有得到任何输入的任务管理器上。奇怪的是，尽管前面有重新平衡，前一组（分区 -> 平面地图 -> 地图）并没有向该任务管理器发送任何内容。

我正在 EMR 上运行它。我看到有一个slot.idle.timeout，这会产生影响，如果有，我该如何为该工作指定它？可以在命令行上完成吗？