我们有什么方法可以覆盖现有文件,同时HDFS使用:
hadoop fs -copyToLocal <HDFS PATH> <local path>
我试图看看是什么让Apache Tez和Hive比使用hive映射reduce更快.我无法理解DAG的概念.
任何人都可以很好地理解Apache TEZ的架构.
我试图用一个大表(10G)和小表(230 MB)执行地图方面.使用小i将在连接键列后使用所有列来生成输出记录
我用过以下设置
set hive.auto.convert.join = true;
set hive.mapjoin.smalltable.filesize = 262144000;
日志:
**2013-09-20 02:43:50 Starting to launch local task to process map join; maximum memory = 1065484288
2013-09-20 02:44:05 Processing rows: 200000 Hashtable size: 199999 Memory usage: 430269904 rate:0.404
2013-09-20 02:44:14 Processing rows: 300000 Hashtable size: 299999 Memory usage: 643070664 rate:0.604
Exception in thread "Thread-0" java.lang.OutOfMemoryError: Java heap space
at java.util.jar.Manifest$FastInputStream.<init>(Manifest.java:313)
at java.util.jar.Manifest$FastInputStream.<init>(Manifest.java:308)
at java.util.jar.Manifest.read(Manifest.java:176)
at java.util.jar.Manifest.<init>(Manifest.java:50)
at java.util.jar.JarFile.getManifestFromReference(JarFile.java:168)
at java.util.jar.JarFile.getManifest(JarFile.java:149)
at sun.misc.URLClassPath$JarLoader$2.getManifest(URLClassPath.java:696)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:228)
at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
at …我有运行配置单元查询,它运行正常的小数据集.但我正在运行2.5亿条记录,我已经在日志中遇到错误
FATAL org.apache.hadoop.mapred.Child: Error running child : java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:640)
at org.apache.hadoop.mapred.Task$TaskReporter.startCommunicationThread(Task.java:725)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:362)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1136)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
2013-03-18 14:12:58,907 WARN org.apache.hadoop.mapred.Child: Error running child
java.io.IOException: Cannot run program "ln": java.io.IOException: error=11, Resource temporarily unavailable
at java.lang.ProcessBuilder.start(ProcessBuilder.java:460)
at java.lang.Runtime.exec(Runtime.java:593)
at java.lang.Runtime.exec(Runtime.java:431)
at java.lang.Runtime.exec(Runtime.java:369)
at org.apache.hadoop.fs.FileUtil.symLink(FileUtil.java:567)
at org.apache.hadoop.mapred.TaskRunner.symlink(TaskRunner.java:787)
at org.apache.hadoop.mapred.TaskRunner.setupWorkDir(TaskRunner.java:752)
at org.apache.hadoop.mapred.Child.main(Child.java:225)
Caused by: java.io.IOException: java.io.IOException: error=11, Resource temporarily unavailable
at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
at java.lang.ProcessImpl.start(ProcessImpl.java:65)
at …