标签: hadoop2

YARN 中如何处理软件/硬件故障？具体来说，如果容器发生故障/崩溃，会发生什么？

我已经从NOAA下载了天气 .txt 文件，它看起来像：

WBAN,Date,Time,StationType,SkyCondition,SkyConditionFlag,Visibility,VisibilityFlag,WeatherType,WeatherTypeFlag,DryBulbFarenheit,DryBulbFarenheitFlag,DryBulbCelsius,DryBulbCelsiusFlag,WetBulbFarenheit,WetBulbFarenheitFlag,WetBulbCelsius,WetBulbCelsiusFlag,DewPointFarenheit,DewPointFarenheitFlag,DewPointCelsius,DewPointCelsiusFlag,RelativeHumidity,RelativeHumidityFlag,WindSpeed,WindSpeedFlag,WindDirection,WindDirectionFlag,ValueForWindCharacter,ValueForWindCharacterFlag,StationPressure,StationPressureFlag,PressureTendency,PressureTendencyFlag,PressureChange,PressureChangeFlag,SeaLevelPressure,SeaLevelPressureFlag,RecordType,RecordTypeFlag,HourlyPrecip,HourlyPrecipFlag,Altimeter,AltimeterFlag
00102,20150101,0001,0,OVC043, ,10.00, , , ,27, ,-2.8, ,26, ,-3.1, ,25, ,-3.9, , 92, , 0, ,000, , , ,30.05, , , , , ,30.36, ,AA, , , ,30.23, 
00102,20150101,0101,0,OVC045, ,10.00, , , ,27, ,-2.8, ,26, ,-3.1, ,25, ,-3.9, , 92, , 6, ,080, , , ,30.07, , , , , ,30.37, ,AA, , , ,30.25, 
00102,20150101,0201,0,OVC047, ,10.00, , , ,26, ,-3.3, ,25, ,-3.7, ,24, ,-4.4, , 92, , 6, ,090, , , ,30.08, , , …

我在 HDFS 中有以下数据（2 个文件）：

/a
  /b
    /c
      /f1.txt
      /f2.txt

我想将f1.txt和f2.txt的权限更改为644：例如hadoop fs -chmod 644 /a/b/c/*.txt

但是，为了真正授予对这些文件的访问权限，我需要将/b和:的权限更改/c为包含这些文件的目录。注意：我不拥有它，并且它已经是世界可读的。755+x/a

有没有hadoop fs命令让我这样做？Java/Scala 代码怎么样？

供参考：我通过在hadoop / share / hadoop / common中添加Netty 4.1.17解决了此问题

无论我尝试运行哪种jar（包括来自https://spark.apache.org/docs/latest/running-on-yarn.html的示例），在Yarn上运行Spark时，我都会不断收到有关容器故障的错误消息。我在命令提示符下收到此错误：

Diagnostics: Exception from container-launch.
Container id: container_1530118456145_0001_02_000001
Exit code: 1
Stack trace: ExitCodeException exitCode=1:
    at org.apache.hadoop.util.Shell.runCommand(Shell.java:585)
    at org.apache.hadoop.util.Shell.run(Shell.java:482)
    at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:776)
    at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212)
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

查看日志时，我发现此错误：

Exception in thread "main" java.lang.NoSuchMethodError:io.netty.buffer.PooledByteBufAllocator.metric()Lio/netty/buffer/PooledByteBufAllocatorMetric;
    at org.apache.spark.network.util.NettyMemoryMetrics.registerMetrics(NettyMemoryMetrics.java:80)
    at org.apache.spark.network.util.NettyMemoryMetrics.<init>(NettyMemoryMetrics.java:76)
    at org.apache.spark.network.client.TransportClientFactory.<init>(TransportClientFactory.java:109)
    at org.apache.spark.network.TransportContext.createClientFactory(TransportContext.java:99)
    at org.apache.spark.rpc.netty.NettyRpcEnv.<init>(NettyRpcEnv.scala:71)
    at org.apache.spark.rpc.netty.NettyRpcEnvFactory.create(NettyRpcEnv.scala:461)
    at org.apache.spark.rpc.RpcEnv$.create(RpcEnv.scala:57)
    at org.apache.spark.deploy.yarn.ApplicationMaster.runExecutorLauncher(ApplicationMaster.scala:530)
    at org.apache.spark.deploy.yarn.ApplicationMaster.org$apache$spark$deploy$yarn$ApplicationMaster$$runImpl(ApplicationMaster.scala:347)
    at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$run$2.apply$mcV$sp(ApplicationMaster.scala:260)
    at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$run$2.apply(ApplicationMaster.scala:260)
    at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$run$2.apply(ApplicationMaster.scala:260)
    at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$5.run(ApplicationMaster.scala:815)
    at java.security.AccessController.doPrivileged(Native Method) …

我正在将我的应用程序从hadoop 1.0.3迁移到hadoop 2.2.0,并且maven build将hadoop-core标记为依赖.因为hadoop 2.2.0不存在hadoop-core.我尝试用hadoop-client和hadoop-common替换它,但我仍然得到ant.filter的这个错误.任何人都可以建议使用哪种工件？

previous config :
<dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-core</artifactId>
    <version>1.0.3</version>
</dependency>

New Config:

<dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-client</artifactId>
    <version>2.2.0</version>
</dependency>

错误:

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on project event: Compilation failure: Compilation failure:

[ERROR] /opt/teamcity/buildAgent/work/c670ebea1992ec2f/event/src/main/java/com/intel/event/EventContext.java:[27,36] package org.apache.tools.ant.filters does not exist

[ERROR] /opt/teamcity/buildAgent/work/c670ebea1992ec2f/event/src/main/java/com/intel/event/EventContext.java:[27,36] package org.apache.tools.ant.filters does not exist

[ERROR] /opt/teamcity/buildAgent/work/c670ebea1992ec2f/event/src/main/java/com/intel/event/EventContext.java:[180,59] cannot find symbol

[ERROR] symbol: class StringInputStream

[ERROR] location: class com.intel.event.EventContext

据我所知,我知道Hadoop 1和2之间只有一个差异.

其主动和被动的二级名称节点.

有人可以列出Hadoop 1和2之间的区别吗？

我希望在我的一个项目场景中对Hadoop等大数据平台的使用有一些专家意见.虽然我很了解像MySQL这样的数据库,但我是这项技术的新手.

我们正在创建一种用于分析社交媒体数据的产品.因此输入数据将是大量的推文,Facebook帖子,用户配置文件,YouTube数据和来自博客的数据等​​.除此之外,我将有一个Web应用程序来帮助我查看和分析这些数据.正如要求所表明的那样,我需要一种实时系统.因此,如果我有一条推文,我想将它提供给我的网络应用程序,以便进行处理.批量数据处理可能不适合我的应用程序.

我的问题是:

• Hadoop引擎对我来说是个不错的选择吗？
• 我的决定应该以什么为参数？
• 与Hadoop相比,使用Multi Cluster MySQL引擎也是一个不错的选择吗？
• 在Hadoop成为一个好选择的数据的大小和速度方面是否有任何基准？

我有一个名为file name(1).zip（带有空格和括号的文件），我想将此文件放在HDFS上。但是每次我尝试通过时hadoop fs -put ...，都会出现异常。

我什至尝试在文件周围添加引号，甚至试图转义空格和括号，但是它不起作用。

hduser@localhost:/tmp$ hadoop fs -put file\ name\(1\).zip /tmp/one
15/06/05 15:57:46 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
put: unexpected URISyntaxException

hduser@localhost:/tmp$ hadoop fs -put "file\ name\(1\).zip" /tmp/one/
15/06/05 15:59:19 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
put: unexpected URISyntaxException

hduser@localhost:/tmp$ hadoop fs -put "file name(1).zip" /tmp/one/
15/06/05 16:00:36 WARN util.NativeCodeLoader: Unable to load native-hadoop library …

我按照本教程安装了Hadoop.不幸的是,当我运行start-all.sh脚本时 - 在控制台上打印了以下错误:

hduser@dennis-HP:/usr/local/hadoop/sbin$ start-all.sh
This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
hadoop config script is run...
hdfs script is run...
Config parameter : 
16/04/10 23:45:40 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting namenodes on [localhost]
localhost: mkdir: cannot create directory ‘/usr/local/hadoop/logs’: Permission denied
localhost: chown: cannot access ‘/usr/local/hadoop/logs’: No such file or directory
localhost: starting namenode, logging to /usr/local/hadoop/logs/hadoop-hduser-namenode-dennis-HP.out
localhost: /usr/local/hadoop/sbin/hadoop-daemon.sh: line 159: /usr/local/hadoop/logs/hadoop-hduser-namenode-dennis-HP.out: …

我有一个11个节点的集群,9个是奴隶,2个是主人,与我之前的问题相同.我正在该集群上执行TestDFSIO基准测试,该集群使用CDH 5.8.0.

我从TestDFSIO结果得到以下输出.这是吞吐量吗？或者我是否需要从中计算吞吐量,例如文件数量乘以TestDFSIO结果通过其他方式？

请告诉我如何获得整个群集的吞吐量.

----- TestDFSIO ----- : write
           Date & time: Mon Aug 29 07:28:01 MDT 2016
       Number of files: 10000
Total MBytes processed: 8000000.0
     Throughput mb/sec: 50.75090177850001
Average IO rate mb/sec: 85.83160400390625
 IO rate std deviation: 82.41435666074283
    Test exec time sec: 3149.755