如何将hadoop输入格式设置为NLineInputFormat？

Question

我试图限制每个Mappers获得的行数.我的代码是这样的:

    package com.iathao.mapreduce;

    import java.io.IOException;
    import java.net.MalformedURLException;

    import org.apache.hadoop.fs.Path;
    import org.apache.hadoop.io.IntWritable;
    import org.apache.hadoop.io.Text;
    import org.apache.hadoop.mapred.lib.NLineInputFormat;
    import org.apache.hadoop.mapreduce.Job;
    import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
    import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
    import org.apache.regexp.RESyntaxException;

    import com.gargoylesoftware.htmlunit.FailingHttpStatusCodeException;

    public class Main {


    public static void main(String[] args) throws FailingHttpStatusCodeException, MalformedURLException, IOException, RESyntaxException {

    try {
        if (args.length != 2) {
            System.err.println("Usage: NewMaxTemperature <input path> <output path>");
            System.exit(-1);
        }
        Job job = new Job();
        job.setJarByClass(Main.class);
        job.getConfiguration().set("mapred.max.map.failures.percent", "100");
        // job.getConfiguration().set("mapred.map.max.attempts", "10");
        //NLineInputFormat. .setNumLinesPerSplit(job, 1);
        job.setInputFormatClass(NLineInputFormat.class);

在示例的最后一行(job.setInputFormatClass(NLineInputFormat.class);)我收到以下错误:

The method setInputFormatClass(Class<? extends InputFormat>) in the type Job is not applicable for the arguments (Class<NLineInputFormat>)

我是否以某种方式获得了错误的NLineInputFormat类？

Answer 1

您正在混合旧API和新API.

import org.apache.hadoop.mapred.lib.NLineInputFormat;
import org.apache.hadoop.mapreduce.Job;

根据"Hadoop:The Definitive Guide"

新API位于org.apache.hadoop.mapreduce包(和子包)中.旧的API仍然可以在org.apache.hadoop.mapred中找到.

如果您打算使用新API,请使用org.apache.hadoop.mapreduce包中的NLineInputFormat.