使用REST模板Java Spring MVC从服务器下载大文件

Question

我有一个REST服务,它向我发送一个大的ISO文件,REST服务中没有问题.现在我已经编写了一个Web应用程序,它调用其余服务来获取文件,在客户端(Web应用程序)端我收到Out Of memory Exception.Below是我的代码

HttpHeaders headers = new HttpHeaders();//1 Line

    headers.setAccept(Arrays.asList(MediaType.APPLICATION_OCTET_STREAM));//2 Line
    headers.set("Content-Type","application/json");//3 Line
    headers.set("Cookie", "session=abc");//4 Line
    HttpEntity statusEntity=new HttpEntity(headers);//5 Line
    String uri_status=new String("http://"+ip+":8080/pcap/file?fileName={name}");//6 Line

    ResponseEntity<byte[]>resp_status=rt.exchange(uri_status, HttpMethod.GET, statusEntity, byte[].class,"File5.iso");//7 Line

我在7行收到内存异常,我想我将不得不缓冲并获取部分内容,但不知道如何从服务器获取此文件,文件大小约为500到700 MB.谁能请你帮忙.

异常堆栈:

  org.springframework.web.util.NestedServletException: Handler processing failed; nested exception is java.lang.OutOfMemoryError: Java heap space
    org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:972)
    org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:852)
    org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:882)
    org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:778)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:622)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
    org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
root cause

java.lang.OutOfMemoryError: Java heap space
    java.util.Arrays.copyOf(Arrays.java:3236)
    java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118)
    java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
    java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
    org.springframework.util.FileCopyUtils.copy(FileCopyUtils.java:113)
    org.springframework.util.FileCopyUtils.copyToByteArray(FileCopyUtils.java:164)
    org.springframework.http.converter.ByteArrayHttpMessageConverter.readInternal(ByteArrayHttpMessageConverter.java:58)
    org.springframework.http.converter.ByteArrayHttpMessageConverter.readInternal(ByteArrayHttpMessageConverter.java:1)
    org.springframework.http.converter.AbstractHttpMessageConverter.read(AbstractHttpMessageConverter.java:153)
    org.springframework.web.client.HttpMessageConverterExtractor.extractData(HttpMessageConverterExtractor.java:81)
    org.springframework.web.client.RestTemplate$ResponseEntityResponseExtractor.extractData(RestTemplate.java:627)
    org.springframework.web.client.RestTemplate$ResponseEntityResponseExtractor.extractData(RestTemplate.java:1)
    org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:454)
    org.springframework.web.client.RestTemplate.execute(RestTemplate.java:409)
    org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:385)
    com.pcap.webapp.HomeController.getPcapFile(HomeController.java:186)

我的服务器端REST服务代码工作正常

@RequestMapping(value = URIConstansts.GET_FILE, produces = { MediaType.APPLICATION_OCTET_STREAM_VALUE}, method = RequestMethod.GET)
public void getFile(@RequestParam(value="fileName", required=false) String fileName,HttpServletRequest request,HttpServletResponse response) throws IOException{



    byte[] reportBytes = null;
    File result=new File("/home/arpit/Documents/PCAP/dummyPath/"+fileName);

    if(result.exists()){
        InputStream inputStream = new FileInputStream("/home/arpit/Documents/PCAP/dummyPath/"+fileName); 
        String type=result.toURL().openConnection().guessContentTypeFromName(fileName);
        response.setHeader("Content-Disposition", "attachment; filename=" + fileName);
        response.setHeader("Content-Type",type);

        reportBytes=new byte[100];//New change
        OutputStream os=response.getOutputStream();//New change
        int read=0;
        while((read=inputStream.read(reportBytes))!=-1){
            os.write(reportBytes,0,read);
        }
        os.flush();
        os.close();






    }

Answer 1

我就是这样做的.基于Spring Jira问题的提示.

RestTemplate restTemplate // = ...;

// Optional Accept header
RequestCallback requestCallback = request -> request.getHeaders()
        .setAccept(Arrays.asList(MediaType.APPLICATION_OCTET_STREAM, MediaType.ALL));

// Streams the response instead of loading it all in memory
ResponseExtractor<Void> responseExtractor = response -> {
    // Here I write the response to a file but do what you like
    Path path = Paths.get("some/path");
    Files.copy(response.getBody(), path);
    return null;
};
restTemplate.execute(URI.create("www.something.com"), HttpMethod.GET, requestCallback, responseExtractor);

从前面提到的Jira问题:

请注意,您不能简单地从提取器返回InputStream,因为在execute方法返回时,底层连接和流已经关闭.

春季5的更新

Spring 5引入了WebClient允许异步(例如非阻塞)http请求的类.来自doc:

与RestTemplate相比,WebClient是:

• 非阻塞,被动,并支持更高的并发性和更少的硬件资源.
• 提供了一个利用Java 8 lambdas的功能API.
• 支持同步和异步方案.
• 支持从服务器向上或向下流式传输.

Answer 2

正如@bernie提到的，您可以使用 WebClient 来实现这一点：

public Flux<DataBuffer> downloadFileUrl( ) throws IOException {

    WebClient webClient = WebClient.create();

    // Request service to get file data
    return Flux<DataBuffer> fileDataStream = webClient.get()
            .uri( this.fileUrl )
            .accept( MediaType.APPLICATION_OCTET_STREAM )
            .retrieve()
            .bodyToFlux( DataBuffer.class );
}

@GetMapping( produces = MediaType.APPLICATION_OCTET_STREAM_VALUE )
public void downloadFile( HttpServletResponse response ) throws IOException
{
    Flux<DataBuffer> dataStream = this.downloadFileUrl( );

    // Streams the stream from response instead of loading it all in memory
    DataBufferUtils.write( dataStream, response.getOutputStream() )
            .map( DataBufferUtils::release )
            .blockLast();
}

即使您没有 Reactive Server 堆栈，您仍然可以使用 WebClient - Rossen Stoyanchev（Spring 框架团队的成员）在Spring MVC Developers演示文稿的“Reactive”指南中对其进行了很好的解释。在这次演讲中，Rossen Stoyanchev 提到他们考虑过弃用 RestTemplate，但他们毕竟决定推迟它，但它可能会在未来发生！

到目前为止，使用 WebClient 的主要缺点是一个非常陡峭的学习曲线（反应式编程），但我认为将来没有办法避免，所以最好早点看一看。