F#:如何正确枚举多个文件？

Question

我有一堆文件,几个MiB的大小非常简单:

• 它们的大小为8的倍数
• 它们只包含小尾数双打,所以可以被读取BinaryReader的ReadDouble()方法

按字典顺序排序,它们包含所需序列中的所有值.

我不能保持在内存中的一切作为float list或float array所以我需要float seq通过必要的文件去实际被访问时.通过序列的部分实际上使用命令式样式,GetEnumerator()因为我不希望任何资源泄漏并且想要正确关闭所有文件.

我的第一个功能方法是:

let readFile file = 
    let rec readReader (maybeReader : BinaryReader option) = 
        match maybeReader with
        | None -> 
            let openFile() = 
                printfn "Opening the file"
                new BinaryReader(new FileStream(file, FileMode.Open, FileAccess.Read, FileShare.Read))
                |> Some
                |> readReader
            seq { yield! openFile() }
        | Some reader when reader.BaseStream.Position >= reader.BaseStream.Length -> 
            printfn "Closing the file"
            reader.Dispose()
            Seq.empty
        | Some reader -> 
            reader.BaseStream.Position |> printfn "Reading from position %d"
            let bytesToRead = Math.Min(1048576L, reader.BaseStream.Length - reader.BaseStream.Position) |> int
            let bytes = reader.ReadBytes bytesToRead
            let doubles = Array.zeroCreate<float> (bytesToRead / 8)
            Buffer.BlockCopy(bytes, 0, doubles, 0, bytesToRead)
            seq { 
                yield! doubles
                yield! readReader maybeReader
            }
    readReader None

然后,当我有一个string list包含所有文件时,我可以这样说:

let values = files |> Seq.collect readFile
use ve = values.GetEnumerator()
// Do stuff that only gets partial data from one file

但是,这只会在阅读器到达末尾时关闭文件(在查看该功能时很清楚).因此,作为第二种方法,我实施了强制枚举的文件:

type FileEnumerator(file : string) = 
    let reader = new BinaryReader(new FileStream(file, FileMode.Open, FileAccess.Read, FileShare.Read))
    let mutable _current : float = Double.NaN
    do file |> printfn "Enumerator active for %s"

    interface IDisposable with
        member this.Dispose() = 
            reader.Dispose()
            file |> printfn "Enumerator disposed for %s"

    interface IEnumerator with
        member this.Current = _current :> obj
        member this.Reset() = reader.BaseStream.Position <- 0L
        member this.MoveNext() = 
            let stream = reader.BaseStream
            if stream.Position >= stream.Length then false
            else 
                _current <- reader.ReadDouble()
                true

    interface IEnumerator<float> with
        member this.Current = _current

type FileEnumerable(file : string) = 

    interface IEnumerable with
        member this.GetEnumerator() = new FileEnumerator(file) :> IEnumerator

    interface IEnumerable<float> with
        member this.GetEnumerator() = new FileEnumerator(file) :> IEnumerator<float>

let readFile' file = new FileEnumerable(file) :> float seq

现在,当我说

let values = files |> Seq.collect readFile'
use ve = values.GetEnumerator()
// do stuff with the enumerator

正确处理枚举器会将气泡传递给我的命令枚举器.

虽然这对于我想要实现的目标是一个可行的解决方案(我可以通过像第一个功能方法那样块状地读取它来加快速度但是为了简洁起见我没有在这里做到)我想知道是否有一个真正的功能方法来避免这种情况枚举器中的可变状态.

Answer 1

当你说使用GetEnumerator()可以防止资源泄漏并允许正确关闭所有文件时,我不明白你的意思.以下是我的尝试(忽略块复制部分用于演示目的),我认为它会导致文件正确关闭.

let eof (br : BinaryReader) = 
  br.BaseStream.Position = br.BaseStream.Length  

let readFileAsFloats filePath = 
    seq{
        use file = new FileStream(filePath, FileMode.Open, FileAccess.Read, FileShare.Read)
        use reader = new BinaryReader(file)
        while (not (eof reader)) do
            yield reader.ReadDouble()
    }

let readFilesAsFloats filePaths = 
    filePaths |> Seq.collect readFileAsFloats

let floats = readFilesAsFloats ["D:\\floatFile1.txt"; "D:\\floatFile2.txt"]

那是你的想法吗？