如何阅读hadoop顺序文件？

Question

我有一个顺序文件,它是hadoop map-reduce作业的输出.在此文件中,数据以键值对形式写入,值本身是映射.我想将值作为MAP对象读取,以便我可以进一步处理它.

    Configuration config = new Configuration();
    Path path = new Path("D:\\OSP\\sample_data\\data\\part-00000");
    SequenceFile.Reader reader = new SequenceFile.Reader(FileSystem.get(config), path, config);
    WritableComparable key = (WritableComparable) reader.getKeyClass().newInstance();
    Writable value = (Writable) reader.getValueClass().newInstance();
    long position = reader.getPosition();

    while(reader.next(key,value))
    {
           System.out.println("Key is: "+textKey +" value is: "+val+"\n");
    }

程序输出:键是:[这是键]值是:{abc = 839177,xyz = 548498,lmn = 2,pqr = 1}

在这里我获得了作为字符串的价值,但我希望它作为地图的对象.

Answer 1

检查SequenceFile#next的API文档(可写,可写)

while(reader.next(key,value))
{
       System.out.println("Key is: "+textKey +" value is: "+val+"\n");
}

应该换成

while(reader.next(key,value))
{
       System.out.println("Key is: "+key +" value is: "+value+"\n");
}

使用SequenceFile.Reader#getValueClassName获取SequenceFile中的值类型.SequenceFile在文件头中具有键/值类型.