是否可以在没有Cygwin的Windows上以本地模式运行Hadoop作业(如WordCount示例)？

Question

我有Windows 7,Java 8,Maven和Eclipse.我创建了一个Maven项目和使用几乎一模一样的代码在这里.

这只是一个简单的"字数统计"样本.我尝试从Eclipse启动"驱动程序"程序,我提供命令行参数(输入文件和输出目录)并获得以下错误:

Exception in thread "main" java.lang.NullPointerException   at
java.lang.ProcessBuilder.start(ProcessBuilder.java:1012)    at
org.apache.hadoop.util.Shell.runCommand(Shell.java:404)     at
org.apache.hadoop.util.Shell.run(Shell.java:379)    at
org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589) at
org.apache.hadoop.util.Shell.execCommand(Shell.java:678)    at
org.apache.hadoop.util.Shell.execCommand(Shell.java:661)    at
org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:639) at
org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:435) at
org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:277) at
org.apache.hadoop.mapreduce.JobSubmissionFiles.getStagingDir(JobSubmissionFiles.java:125) at
org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:344) at
org.apache.hadoop.mapreduce.Job$10.run(Job.java:1268)   at
org.apache.hadoop.mapreduce.Job$10.run(Job.java:1265)   at
java.security.AccessController.doPrivileged(Native Method)  at
javax.security.auth.Subject.doAs(Subject.java:422)  at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491) at
org.apache.hadoop.mapreduce.Job.submit(Job.java:1265)   at
org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1286)    at
misc.projects.hadoop.exercises.WordCountDriverApp.main(WordCountDriverApp.java:29)

失败的行(WordCountDriverApp.java:29)包含启动作业的命令:

job.waitForCompletion(true)

我想让它工作,因此我想了解一些事情:

我是否必须提供任何hdfs-site.xml,yarn-site.xml,...所有这一切,如果我只想要本地模式(没有任何集群)？我现在没有这些XML配置文件.据我所知,本地模式的默认值都可以,也许我错了.

在Windows下完全可以(无论是启动任何Hadoop任务)还是整个Hadoop都只有Linux？

PS:Hadoop依赖关系如下:

<dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-client</artifactId>
    <version>2.2.0</version>
    <scope>provided</scope>
</dependency>

Answer 1

1. 下载为Windows编译的Hadoop 2.6.0或2.7.1
2. 创建指向解压缩目录的HADOOP_HOME环境变量
3. 将%HADOOP_HOME%\ bin添加到PATH env var

来源:https://stackoverflow.com/a/27394808/543836