hdfs显示本地文件列表

Question

我在 OS X 中安装了 Hadoop，一切顺利。我的经验是最近的，并努力了解有关 Hadoop 应用程序开发的更多信息。

昨天，当我需要在 Hadoop 中查找目录和/或文件列表时，我只需键入

$ hadoop fs -ls 

而且，它会向我显示集群中的所有内容。

今天，它显示了文件系统中的所有本地内容。我必须提供 hdfs 的确切地址才能获取内容列表，

$ hadoop fs -ls hdfs://localhost:8020/user/myName

我的core-site.xml文件和以前一样，

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<!-- Put site-specific property overrides in this file. -->
 <configuration>
    <property>
        <name>hadoop.tmp.dir</name>
        <value>/usr/local/Cellar/hadoop/hdfs/tmp</value>
        <description>A base for other temporary directories.</description>
    </property>
    <property>
        <name>fs.default.name</name>
        <value>hdfs://localhost:8020</value>
    </property>
</configuration>

在启动 hadoop 守护进程之前，我停止了集群并再次使用以下命令重新格式化了分布式文件系统，以便我们可以在执行 map-reduce 作业时将数据源放入 hdfs 文件系统中

$ hdfs namenode -format

我收到管理报告告知 FileSystem file:/// 不是 HDFS 文件系统，

$ hadoop dfsadmin -report
WARNING: Use of this script to execute dfsadmin is deprecated.
WARNING: Attempting to execute replacement "hdfs dfsadmin" instead.

2018-10-18 18:01:27,316 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
report: FileSystem file:/// is not an HDFS file system
Usage: hdfs dfsadmin [-report] [-live] [-dead] [-decommissioning] [-enteringmaintenance] [-inmaintenance]

在该core-site.xml文件中，我还将配置更新为以下内容，

<property>
    <!-- <name>fs.default.name</name> -->
    <!-- <value>hdfs://localhost:8020</value> -->
    <name>fs.defaultFS</name>
    <value>hdfs://localhost.localdomain:8020/</value>
</property>

那时我已经改过自新，但这并没有改变什么。正如另一个答案提到的，haddop home 已在~/.bashrc文件中提供，

export HADOOP_HOME=/Users/chaklader/hadoop
export HADOOP_INSTALL=$HADOOP_HOME
export HADOOP_MAPRED_HOME=$HADOOP_HOME
export HADOOP_COMMON_HOME=$HADOOP_HOME
export HADOOP_HDFS_HOME=$HADOOP_HOME
export YARN_HOME=$HADOOP_HOME
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_HOME/lib/native
export PATH=$PATH:$HADOOP_HOME/sbin:$HADOOP_HOME/bin

如何切换到HDFS文件系统？任何形式的建议将不胜感激。

Answer 1

您需要确保已添加一个名为 的环境变量，HADOOP_CONF_DIR将其设置为包含来自 Hadoop 的 XML 文件的目录。

您可以在.bashrc主文件夹中执行此操作

否则，您将获得默认文件系统 ，file://它仍然有效并且仍然可以正常运行 MapReduce 作业

FWIW，这是我的核心网站

$ cat /usr/local/Cellar/hadoop/3.1.1/libexec/etc/hadoop/core-site.xml

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
    <property>
        <name>hadoop.tmp.dir</name>
        <value>file:///tmp/hadoop/hdfs/tmp</value>
        <description>A base for other temporary directories.</description>
    </property>
  <property>
    <name>fs.default.name</name>
    <value>hdfs://localhost:9000</value>
  </property>
</configuration>

和hdfs站点

$ cat /usr/local/Cellar/hadoop/3.1.1/libexec/etc/hadoop/hdfs-site.xml

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<!-- Put site-specific property overrides in this file. -->

<configuration>
    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>
  <property>
      <name>dfs.namenode.name.dir</name>
      <value>file:///tmp/hadoop/hdfs/names</value>
  </property>
  <property>
    <name>fs.checkpoint.dir</name>
    <value>file:///tmp/hadoop/hdfs/checkpoint</value>
  </property>
  <property>
    <name>fs.checkpoint.edits.dir</name>
    <value>file:///tmp/hadoop/hdfs/checkpoint-edits</value>
  </property>
  <property>
      <name>dfs.datanode.data.dir</name>
      <value>file:///tmp/hadoop/hdfs/data</value>
  </property>
</configuration>