在 HDFS 中使用 -addMount 时找不到类 org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem

Question

我有以下设置：

$ export |grep HADOOP
declare -x HADOOP_HOME="/home/jesaremi/hadoop-3.1.3-bin"
declare -x HADOOP_OPTIONAL_TOOLS="hadoop-azure"

$ echo $PATH
/home/jesaremi/hadoop-3.1.3-bin/bin:/home/jesaremi/hadoop-3.1.3-bin/sbin:/opt/protobuf/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/jesaremi/.dotnet/tools:/opt/HP_Fortify/HP_Fortify_SCA_and_Apps_4.30/bin:/opt/HP_Fortify/HP_Fortify_SCA_and_Apps_4.30/bin


$ ls -l /home/jesaremi/hadoop-3.1.3-bin/share/hadoop/common/
total 7452
-rw-rw-r-- 1 jesaremi jesaremi  303043 Dec 19 17:30 hadoop-azure-3.1.3.jar
-rw-rw-r-- 1 jesaremi jesaremi 4092533 Dec 17 09:42 hadoop-common-3.1.3-SNAPSHOT.jar
-rw-rw-r-- 1 jesaremi jesaremi 2877294 Dec 17 09:42 hadoop-common-3.1.3-SNAPSHOT-tests.jar
-rw-rw-r-- 1 jesaremi jesaremi  130001 Dec 17 09:42 hadoop-kms-3.1.3-SNAPSHOT.jar
-rw-rw-r-- 1 jesaremi jesaremi  201637 Dec 17 09:42 hadoop-nfs-3.1.3-SNAPSHOT.jar
drwxrwxr-x 2 jesaremi jesaremi    4096 Dec 17 09:42 jdiff
drwxrwxr-x 2 jesaremi jesaremi    4096 Dec 17 09:42 lib
drwxrwxr-x 2 jesaremi jesaremi    4096 Dec 17 09:42 sources
drwxrwxr-x 3 jesaremi jesaremi    4096 Dec 17 09:42 webapps

然而，当我尝试添加 ABFS 文件系统时，我得到以下信息：

$ hdfs dfsadmin -addMount abfs://main@test1.dfs.core.windows.net/ /mnt/abfs/main "fs.azure.abfs.account.name=test1.dfs.core.windows.net,fs.azure.account.key.test1.dfs.core.windows.net=<key removed>"
addMount: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem not found

Answer 1

担心 HADOOP_OPTIONAL_TOOLS 环境变量还不够；你需要将 hadoop-azure JAR 和其他一些 JAR 放入 common/lib 中

从 share/hadoop/tools/lib 复制 hadoop-azure jar、azure-* 以及 wildfly-openssl.jar（如果存在）到 share/hadoop/common/lib

云存储JAR 具有诊断功能，因为它会告诉您缺少哪个 JAR，例如

bin/hadoop jar cloudstore-0.1-SNAPSHOT.jar storediag abfs://c@account.dfs.core.windows.net/