小编And*_*ssi的帖子

如何使用Hadoop WebHDFS读取和传输文件块?

我需要将大文件(至少14MB)从FIWARE Lab的Cosmos实例传输到我的后端.

我使用Spring RestTemplate作为此处描述的Hadoop WebHDFS REST API的客户端接口,但我遇到了IO异常:

Exception in thread "main" org.springframework.web.client.ResourceAccessException: I/O error on GET request for "http://cosmos.lab.fiware.org:14000/webhdfs/v1/user/<user.name>/<path>?op=open&user.name=<user.name>":Truncated chunk ( expected size: 14744230; actual size: 11285103); nested exception is org.apache.http.TruncatedChunkException: Truncated chunk ( expected size: 14744230; actual size: 11285103)
    at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:580)
    at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:545)
    at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:466)
Run Code Online (Sandbox Code Playgroud)

这是生成异常的实际代码:

RestTemplate restTemplate = new RestTemplate();
restTemplate.setRequestFactory(new HttpComponentsClientHttpRequestFactory());
restTemplate.getMessageConverters().add(new ByteArrayHttpMessageConverter()); 
HttpEntity<?> entity = new HttpEntity<>(headers);

UriComponentsBuilder builder = 
    UriComponentsBuilder.fromHttpUrl(hdfs_path)
        .queryParam("op", "OPEN")
        .queryParam("user.name", user_name);

ResponseEntity<byte[]> response =
    restTemplate
        .exchange(builder.build().encode().toUri(), HttpMethod.GET, entity, byte[].class);

FileOutputStream …
Run Code Online (Sandbox Code Playgroud)

hadoop httpclient webhdfs fiware fiware-cosmos

6
推荐指数
1
解决办法
934
查看次数

标签 统计

fiware ×1

fiware-cosmos ×1

hadoop ×1

httpclient ×1

webhdfs ×1