将3GB文件转换为字节数组

Question

我试图将3GB文件转换为字节数组但获取OOM(OutOfMemoryError).

我们尝试了

RandomAccessFile randomAccessFile = new RandomAccessFile(sourceLocation.getAbsolutePath(), "r");
MappedByteBuffer mappedByteBuffer = randomAccessFile.getChannel().map(FileChannel.MapMode.READ_ONLY, 0, randomAccessFile.length()); //while it is large file, it threw 'mmap failed: ENOMEM (Out of memory)' exception.

byte[] data = new byte[1024 * 1024];  // 1MB read at time
while (mappedByteBuffer.hasRemaining()) {
    int remaining = data.length;

    if (mappedByteBuffer.remaining() < remaining)
        remaining = mappedByteBuffer.remaining();
        mappedByteBuffer.get(data, 0, remaining);
    }

    mappedByteBuffer.rewind();
    byte fileContent[] = mappedByteBuffer.array(); //while it is small file, it threw 'java.nio.ReadOnlyBufferException' exception.
    randomAccessFile.close();
}

我的自定义请求正文:我的请求正文准备的自定义请求正文类

import android.os.Looper;

import java.io.ByteArrayInputStream;
import java.io.File;
import java.io.IOException;

import okhttp3.MediaType;
import okhttp3.RequestBody;
import okio.BufferedSink;

public class ProgressRequestBodyTemp extends RequestBody
{
    private static final int DEFAULT_BUFFER_SIZE = 2048;
    private File mFile;
    private String mPath;
    private String mFileType;
    private int mItemIndex;
    private UploadCallbacks mListener;

    private byte[] encryptedData;
    private ByteArrayInputStream bis;

    public ProgressRequestBodyTemp(final String fileType, final File file, byte[] encryptedData, final int itemIndex, final UploadCallbacks listener)
    {
        this.mFile = file;
        this.mFileType = fileType;
        this.mItemIndex = itemIndex;
        this.mListener = listener;

        this.encryptedData = encryptedData;

        try
        {
            bis = new ByteArrayInputStream(encryptedData); // Convert byte array into input stream for send data to server
        }
        catch (Exception ex)
        {
            ex.printStackTrace();
        }
    }

    @Override
    public MediaType contentType()
    {
        try
        {
            return MediaType.parse(mFileType);
        }
        catch (Exception ex)
        {
            return MediaType.parse("");
        }
    }

    @Override
    public long contentLength() throws IOException
    {
        return encryptedData.length;
    }

    @Override
    public void writeTo(BufferedSink sink) throws IOException
    {
        long fileLength = mFile.length();
        byte[] buffer = new byte[DEFAULT_BUFFER_SIZE];

        long uploaded = 0;
        try
        {
            int read;
            android.os.Handler handler = new android.os.Handler(Looper.getMainLooper());
            while ((read = bis.read(buffer)) != -1)
            {

                // update progress on UI thread
                handler.post(new ProgressUpdater(uploaded, encryptedData.length));

                uploaded += read;
                sink.write(buffer, 0, read);
            }
        }
        finally
        {
            bis.close();
        }
    }

    public interface UploadCallbacks
    {
        void onProgressUpdate(int itemIndex, int percentage);
    }

    private class ProgressUpdater implements Runnable
    {
        private long mUploaded;
        private long mTotal;

        public ProgressUpdater(long uploaded, long total)
        {
            mUploaded = uploaded;
            mTotal = total;
        }

        @Override
        public void run()
        {
            if (mListener != null)
            {
                mListener.onProgressUpdate(mItemIndex, (int) (100 * mUploaded / mTotal));
            }
        }
    }
}

我的请求:请求上传文件

File sourceLocation = new File(".....");
byte fileContent[] = .... // byte array

ProgressRequestBody requestFile  = new ProgressRequestBody(fileType, sourceLocation, fileContent, itemIndex, this);
MultipartBody.Part body = MultipartBody.Part.createFormData("file", sourceLocation.getName(), requestFile); // MultipartBody.Part is used to send also the actual file name
Call<String> mediaCall = getRetrofit().addMedia("SOME STRING DATA", body);

它给出了OutOfMemoryError,请建议将大文件转换为字节数组的最佳方法.

任何帮助都很受欢迎.提前致谢.

Answer 1

使用InputStream和OutputStream.

它们适用于大量数据,例如,当您使用3GB数据并且无法将其加载到内存中时.

如果您要上传文件,请使用FileInputStream.创建一个File对象,将其传递给FileOutputStream构造函数并开始从其中读取字节InputStream到byte[]缓冲区,并将带有a 的字节发送OutputStream到服务器.

这种方法不会导致的OutOfMemoryError,因为你正在阅读只够字节填充缓冲区,这应该是约2KB - 8KB大小.缓冲区已满后,将字节写入服务器.在将缓冲区写入服务器之后,您再次读入缓冲区,并且该过程将一直持续到整个文件上载为止.

使用FileInputStream的示例

        File file = new File("yourfile.txt");
        FileInputStream fis = null;
        OutputStream outputStream = null;

        url = new URL(desiredUrl);
        HttpURLConnection connection = (HttpURLConnection) url.openConnection();


        try {
            fis = new FileInputStream(file);
            connection.setDoOutput(true);
            outputStream = connection.getOutputStream();

            int actuallyRead;
            byte[] buffer = new byte[2048];
            while ((actuallyRead = fis.read(buffer)) != -1) {
                //do something with bytes, for example, write to the server
            outputStream.write(buffer);

            }

        } catch (IOException e) {
            e.printStackTrace();
        } finally {
            try {
                if (fis != null)
                    fis.close();
            } catch (IOException ex) {
                ex.printStackTrace();
            }
            try {
                if (outputStream != null)
                    outputStream.close();
            } catch (IOException ex) {
                ex.printStackTrace();
            }

        }

注意:此方法并不意味着每次填充缓冲区时都需要重新连接到服务器.它会不断写入服务器,直到处理完文件为止.这将在相同的单一连接下发生.