相关疑难解决方法(0)

我在EC2中创建了一个ubuntu单节点hadoop集群.

测试简单的文件上传到hdfs可以从EC2机器上运行,但不能在EC2之外的机器上运行.

我可以通过远程机器的Web界面浏览文件系统,它显示一个在服务中报告的datanode.已打开安全性中从0到60000(!)的所有tcp端口,所以我不认为是这样.

我收到了错误

java.io.IOException: File /user/ubuntu/pies could only be replicated to 0 nodes, instead of 1
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1448)
at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:690)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:342)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1350)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1346)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:742)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1344)

at org.apache.hadoop.ipc.Client.call(Client.java:905)
at org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:198)
at $Proxy0.addBlock(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
at $Proxy0.addBlock(Unknown Source)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:928)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:811)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:427)

namenode log只会给出相同的错误.其他人似乎没有任何有趣的东西

有任何想法吗？

干杯

环境:ubuntu 14.04,hadoop 2.6

当我键入start-all.sh和jps,DataNode不会列出终端上

>jps
9529 ResourceManager
9652 NodeManager
9060 NameNode
10108 Jps
9384 SecondaryNameNode

根据这个答案:Datanode进程没有在Hadoop中运行

我尝试了最好的解决方案

• bin/stop-all.sh (or stop-dfs.sh and stop-yarn.sh in the 2.x serie)
• rm -Rf /app/tmp/hadoop-your-username/*
• bin/hadoop namenode -format (or hdfs in the 2.x series)

但是,现在我明白了:

>jps
20369 ResourceManager
26032 Jps
20204 SecondaryNameNode
20710 NodeManager

如你所见,即使NameNode是失踪,请帮助我.

DataNode logs:https://gist.github.com/fifiteen82726/b561bbd9cdcb9bf36032

NmaeNode logs:https://gist.github.com/fifiteen82726/02dcf095b5a23c1570b0

mapred-site.xml :

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under …

我正在尝试在我的 Windows 10 机器上运行 Hadoop 3.1.1。我修改了所有文件：

• hdfs-site.xml
• mapred-site.xml
• 核心站点.xml
• 纱线站点.xml

然后，我执行了以下命令：

C:\hadoop-3.1.1\bin> hdfs namenode -format

格式运行正确，因此我指示C:\hadoop-3.1.1\sbin执行以下命令：

C:\hadoop-3.1.1\sbin> start-dfs.cmd

命令提示符会打开 2 个新窗口：一个用于 datanode，另一个用于 namenode。

namenode 窗口继续运行：

2018-09-02 21:37:06,232 INFO ipc.Server: IPC Server Responder: starting
2018-09-02 21:37:06,232 INFO ipc.Server: IPC Server listener on 9000: starting
2018-09-02 21:37:06,247 INFO namenode.NameNode: NameNode RPC up at: localhost/127.0.0.1:9000
2018-09-02 21:37:06,247 INFO namenode.FSNamesystem: Starting services required for active state
2018-09-02 21:37:06,247 INFO namenode.FSDirectory: Initializing quota with 4 thread(s)
2018-09-02 21:37:06,247 INFO namenode.FSDirectory: Quota initialization completed …