今天我发现ASF退役了mrunit(参见https://blogs.apache.org/foundation/entry/the_apache_news_round_up85和https://issues.apache.org/jira/browse/HADOOP-3733和主页本身).除了"不活动"之外没有给出任何理由,所以我想必须有另一种选择吗？
将来应该使用什么代替mrunit来测试mapreduce工作呢？
我正在尝试使用MRunit来测试我的sortComparatorClass.似乎MRunit应该能够使用该setKeyOrderComparator方法执行此操作,但是当我运行mapReduceDriver它时,它不会调用类的compare()方法SortComparator.
很确定我在使用MRunit API做错了.
这是我的单元测试代码:
public class UnitTests {
private static transient Log log = LogFactory.getLog(UnitTests.class);
MapReduceDriver<Text, Text, Text, Text, Text, Text> mapReduceDriver;
MapDriver<Text, Text, Text, Text> mapDriver;
ReduceDriver<Text, Text, Text, Text> reduceDriver;
@Before
public void setUp() throws InterruptedException, IOException {
mapDriver = new MapDriver<Text, Text, Text, Text>();
mapDriver.setMapper(new TestMapper());
reduceDriver = new ReduceDriver<Text, Text, Text, Text>();
reduceDriver.setReducer(new TestReducer());
mapReduceDriver = new MapReduceDriver(new TestMapper(), new TestReducer());
mapReduceDriver.setKeyOrderComparator(new TestSortCompartor());
}
@Test
public …我想看看是否有办法在没有hadoop/hive/cygwin设置的情况下在Windows中运行Hive单元测试(即单元测试hive查询).遇到这些开源工具(Hive_runner,HiveTest),尝试在Eclipse中运行hiverunner作为maven项目,其JUnit测试失败并出现以下警告和错误
警告:无法为您的平台加载native-hadoop库....使用内置的java类(如果适用).错误:无法设置路径权限？到0700
谢谢Srivatsan Nallazhagappan
有没有一种方法来测试减少类MRUnit使用MultipleOutputFormat写入到多个输出文件？
我是Hadoop的新手.我想使用MRUnit Test单独测试我的映射器部件.我已经尝试了很多.但我不知道如何解决以下错误 -
"MapDriver类型中的方法setMapper(Mapper)不适用于参数(Recommand.IdIndexMapper)".我使用的是Hadoop-1.2.1,Eclipse Juno,mrunit-1.0.0-hadoop1.jar,junit-4.11,mockito-all-1.9.5.jar.贝娄是我的代码,
我的Mapper类:
类名:推荐,
public static class IdIndexMapper extends MapReduceBase implements Mapper<LongWritable, Text, Text, Text>{
public void map(LongWritable key, Text val, OutputCollector<Text, Text> output,Reporter reporter)throws IOException{
String[] ids;
String ln=val.toString();
ids=ln.split("\t");
output.collect(new Text(ids[0]),new Text(ids[1]));
// System.out.println(ids[0]+" "+ids[1]);
}
我的测试代码:
package org.apache.hadoop.examples;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
import org.apache.hadoop.classification.InterfaceAudience.Private;
import org.apache.hadoop.classification.InterfaceAudience.Public;
import org.apache.hadoop.examples.WordCount.IntSumReducer;
//import org.apache.hadoop.examples.WordCount.TokenizerMapper;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapred.JobConf;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mrunit.mapreduce.MapDriver;
import org.junit.Before;
import org.junit.Test;
import org.myorg.Recommand.IdIndexMapper;
public class RecomTest { …我找不到最新mrunit(1.1.0)的Cloudera存储库.可用的是0.8.0-incubating.以下是我的pom:
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.ma.hadoop</groupId>
<artifactId>MapReduce</artifactId>
<version>0.0.1-SNAPSHOT</version>
<properties>
<hadoop.version>2.3.0-cdh5.1.2</hadoop.version>
<hive.version>0.12.0-cdh5.1.2</hive.version>
<mrunit.version>0.8.0-incubating</mrunit.version>
</properties>
<dependencies>
<!-- For unit testing -->
<dependency>
<groupId>org.apache.mrunit</groupId>
<artifactId>mrunit</artifactId>
<version>${mrunit.version}</version>
</dependency>
<!-- This is sufficient for all -->
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>${hadoop.version}</version>
</dependency>
</dependencies>
<build>
<finalName>Mapred</finalName>
<pluginManagement>
<plugins>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>exec-maven-plugin</artifactId>
<version>1.2.1</version>
</plugin>
</plugins>
</pluginManagement>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.1</version>
<configuration>
<source>${jdk.version}</source>
<target>${jdk.version}</target>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
<version>2.4</version>
<configuration>
<outputDirectory>${basedir}</outputDirectory>
</configuration>
</plugin>
</plugins>
</build>
<repositories>
<repository> …我正在尝试使用MRUnit测试Hadoop .mapreduce Avro作业.我收到一个NullPointerException,如下所示.我附上了一部分pom和源代码.任何援助将不胜感激.
谢谢
我得到的错误是:
java.lang.NullPointerException
at org.apache.hadoop.mrunit.internal.io.Serialization.copy(Serialization.java:73)
at org.apache.hadoop.mrunit.internal.io.Serialization.copy(Serialization.java:91)
at org.apache.hadoop.mrunit.internal.io.Serialization.copyWithConf(Serialization.java:104)
at org.apache.hadoop.mrunit.TestDriver.copy(TestDriver.java:608)
at org.apache.hadoop.mrunit.MapDriverBase.setInputKey(MapDriverBase.java:64)
at org.apache.hadoop.mrunit.MapDriverBase.setInput(MapDriverBase.java:104)
at org.apache.hadoop.mrunit.MapDriverBase.withInput(MapDriverBase.java:218)
at org.lab41.project.mapreduce.ParseMetadataAsTextIntoAvroTest.testMap(ParseMetadataAsTextIntoAvroTest.java:115)
.....
pom片段:
<dependency>
<groupId>org.apache.mrunit</groupId>
<artifactId>mrunit</artifactId>
<version>0.9.0-incubating</version>
<classifier>hadoop2</classifier>
<scope>test</scope>
</dependency>
<avro.version>1.7.4</avro.version>
<hadoop.version>2.0.0-mr1-cdh4.1.3</hadoop.version>
<dependency>
<groupId>org.apache.avro</groupId>
<artifactId>avro</artifactId>
<version>${avro.version}</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>${hadoop.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-core</artifactId>
<version>${hadoop.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.avro</groupId>
<artifactId>avro-mapred</artifactId>
<version>${avro.version}</version>
<classifier>hadoop2</classifier>
</dependency>
以下是测试的摘录:
import static org.junit.Assert.*;
import java.io.IOException;
import java.nio.ByteBuffer;
import java.nio.CharBuffer;
import java.nio.charset.Charset;
import java.nio.charset.CharsetEncoder;
import java.text.ParseException;
import java.text.SimpleDateFormat;
import org.apache.avro.mapred.AvroKey;
import org.apache.avro.hadoop.io.AvroSerialization;
import org.apache.avro.mapred.AvroValue; …我只有mapper类来创建用于批量加载到HBase的文件,我已经为单元测试编写了一个MRUnit.
虽然预期结果和重新结果相同,但MRUnit失败并显示"缺少预期输出"消息 .
预期产量: (4b 65 79 31, {"totalColumns":1,"families":{"default":[{"timestamp":9223372036854775807,"qualifier":"default","vlen":6}]},"row":"Key1"})
实际产量: (4b 65 79 31, {"totalColumns":1,"families":{"default":[{"timestamp":9223372036854775807,"qualifier":"default","vlen":6}]},"row":"Key1"})
我哪里错了？是不是ImmutableBytesWritable或Put有问题？
我尝试在maven项目中使用mrunit测试我的MapReduce 2作业.我从Maven中央仓库搜索并找到了结果:http://search.maven.org/#search%7Cgav%7C1%7Cg%3A%22org.apache.mrunit%22%20AND%20a%3A%22mrunit%22
在上面的链接,最新mrunit 1.1.0有hadoop1.jar和hadoop2.jar,而不是一个罐子链接.正是由于这个原因,当我尝试添加相应的依赖项时
<dependency>
<groupId>org.apache.mrunit</groupId>
<artifactId>mrunit</artifactId>
<version>1.1.0</version>
</dependency>
以pom.xml和运行mvn package,我得到了以下编译错误: Could not find artifact org.apache.mrunit:mrunit:jar:1.1.0 in central (https://repo.maven.apache.org/maven2 ...
任何人都可以帮助解决这个问题吗？提前致谢
我正在使用MRUnit测试我的映射器。我正在传递键和值列表作为测试类向映射器的输入。问题是 ：
String key=1234_abc;
ArrayList<KeyValue> list = new ArrayList<KeyValue>();
KeyValue k1 = new KeyValue(Bytes.toBytes(key),"cf".getBytes(), "Val1".getBytes(),Bytes.toBytes("abc.com"));
KeyValue k2 = new KeyValue(Bytes.toBytes(key), "cf".getBytes(), "Val2".getBytes(),Bytes.toBytes("165"));
Result result = new Result(list);
mapDriver.withInput(key, result);
问题在于，在结果对象中仅保留了第一个键值。其他被存储为空。
mrunit ×10
hadoop ×7
mapreduce ×4
java ×3
hbase ×2
avro ×1
cloudera-cdh ×1
eclipse ×1
hadoop-2.7.2 ×1
hive ×1
junit ×1
maven ×1
reducers ×1
repository ×1
unit-testing ×1