go中逐行读取文件

Question

我无法file.ReadLine在Go中找到功能.我可以弄清楚如何快速写一个,但只是想知道我是否在这里忽略了一些东西.如何逐行读取文件？

Answer 1

在Go 1.1和更新版本中,最简单的方法是使用a bufio.Scanner.这是一个从文件中读取行的简单示例:

package main

import (
    "bufio"
    "fmt"
    "log"
    "os"
)

func main() {
    file, err := os.Open("/path/to/file.txt")
    if err != nil {
        log.Fatal(err)
    }
    defer file.Close()

    scanner := bufio.NewScanner(file)
    for scanner.Scan() {
        fmt.Println(scanner.Text())
    }

    if err := scanner.Err(); err != nil {
        log.Fatal(err)
    }
}

这是Reader逐行读取的最简洁方法.

有一点需要注意:扫描仪不能很好地处理超过65536个字符的行.如果这对你来说是一个问题,那么你应该自己动手Reader.Read().

Answer 2

包中有ReadLine功能bufio.

请注意,如果该行不适合读取缓冲区,则该函数将返回不完整的行.如果你想通过一次调用函数来总是在程序中读取整行,你需要将ReadLine函数封装到你自己的函数中,该函数调用ReadLinefor循环.

bufio.ReadString('\n')并不完全等同于ReadLine因为ReadString当文件的最后一行没有以换行符结尾时无法处理这种情况.

Answer 3

使用:

• reader.ReadString('\n')
  • 如果你不介意线路可能很长(即使用大量的RAM).它保持\n在返回的字符串的末尾.
• reader.ReadLine()
  • 如果您关心限制RAM消耗,并且不介意处理线路大于读取器缓冲区大小的情况的额外工作.

我通过编写程序来测试建议的各种解决方案,以测试在其他答案中被识别为问题的场景:

• 一个4MB行的文件.
• 不以换行符结尾的文件.

我找到:

• 该Scanner解决方案无法处理长线.
• 该ReadLine解决方案实施起来很复杂.
• 该ReadString解决方案是最简单,适用于大排长龙.

这是演示每个解决方案的代码,它可以通过go run main.go以下方式运行:

package main

import (
    "bufio"
    "bytes"
    "fmt"
    "io"
    "os"
)

func readFileWithReadString(fn string) (err error) {
    fmt.Println("readFileWithReadString")

    file, err := os.Open(fn)
    defer file.Close()

    if err != nil {
        return err
    }

    // Start reading from the file with a reader.
    reader := bufio.NewReader(file)

    var line string
    for {
        line, err = reader.ReadString('\n')

        fmt.Printf(" > Read %d characters\n", len(line))

        // Process the line here.
        fmt.Println(" > > " + limitLength(line, 50))

        if err != nil {
            break
        }
    }

    if err != io.EOF {
        fmt.Printf(" > Failed!: %v\n", err)
    }

    return
}

func readFileWithScanner(fn string) (err error) {
    fmt.Println("readFileWithScanner - this will fail!")

    // Don't use this, it doesn't work with long lines...

    file, err := os.Open(fn)
    defer file.Close()

    if err != nil {
        return err
    }

    // Start reading from the file using a scanner.
    scanner := bufio.NewScanner(file)

    for scanner.Scan() {
        line := scanner.Text()

        fmt.Printf(" > Read %d characters\n", len(line))

        // Process the line here.
        fmt.Println(" > > " + limitLength(line, 50))
    }

    if scanner.Err() != nil {
        fmt.Printf(" > Failed!: %v\n", scanner.Err())
    }

    return
}

func readFileWithReadLine(fn string) (err error) {
    fmt.Println("readFileWithReadLine")

    file, err := os.Open(fn)
    defer file.Close()

    if err != nil {
        return err
    }

    // Start reading from the file with a reader.
    reader := bufio.NewReader(file)

    for {
        var buffer bytes.Buffer

        var l []byte
        var isPrefix bool
        for {
            l, isPrefix, err = reader.ReadLine()
            buffer.Write(l)

            // If we've reached the end of the line, stop reading.
            if !isPrefix {
                break
            }

            // If we're just at the EOF, break
            if err != nil {
                break
            }
        }

        if err == io.EOF {
            break
        }

        line := buffer.String()

        fmt.Printf(" > Read %d characters\n", len(line))

        // Process the line here.
        fmt.Println(" > > " + limitLength(line, 50))
    }

    if err != io.EOF {
        fmt.Printf(" > Failed!: %v\n", err)
    }

    return
}

func main() {
    testLongLines()
    testLinesThatDoNotFinishWithALinebreak()
}

func testLongLines() {
    fmt.Println("Long lines")
    fmt.Println()

    createFileWithLongLine("longline.txt")
    readFileWithReadString("longline.txt")
    fmt.Println()
    readFileWithScanner("longline.txt")
    fmt.Println()
    readFileWithReadLine("longline.txt")
    fmt.Println()
}

func testLinesThatDoNotFinishWithALinebreak() {
    fmt.Println("No linebreak")
    fmt.Println()

    createFileThatDoesNotEndWithALineBreak("nolinebreak.txt")
    readFileWithReadString("nolinebreak.txt")
    fmt.Println()
    readFileWithScanner("nolinebreak.txt")
    fmt.Println()
    readFileWithReadLine("nolinebreak.txt")
    fmt.Println()
}

func createFileThatDoesNotEndWithALineBreak(fn string) (err error) {
    file, err := os.Create(fn)
    defer file.Close()

    if err != nil {
        return err
    }

    w := bufio.NewWriter(file)
    w.WriteString("Does not end with linebreak.")
    w.Flush()

    return
}

func createFileWithLongLine(fn string) (err error) {
    file, err := os.Create(fn)
    defer file.Close()

    if err != nil {
        return err
    }

    w := bufio.NewWriter(file)

    fs := 1024 * 1024 * 4 // 4MB

    // Create a 4MB long line consisting of the letter a.
    for i := 0; i < fs; i++ {
        w.WriteRune('a')
    }

    // Terminate the line with a break.
    w.WriteRune('\n')

    // Put in a second line, which doesn't have a linebreak.
    w.WriteString("Second line.")

    w.Flush()

    return
}

func limitLength(s string, length int) string {
    if len(s) < length {
        return s
    }

    return s[:length]
}

我测试过:

• 去版本go1.7 windows/amd64
• 去版本go1.6.3 linux/amd64
• 去版本go1.7.4 darwin/amd64

测试程序输出:

Long lines

readFileWithReadString
 > Read 4194305 characters
 > > aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
 > Read 12 characters
 > > Second line.

readFileWithScanner - this will fail!
 > Failed!: bufio.Scanner: token too long

readFileWithReadLine
 > Read 4194304 characters
 > > aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
 > Read 12 characters
 > > Second line.

No linebreak

readFileWithReadString
 > Read 28 characters
 > > Does not end with linebreak.

readFileWithScanner - this will fail!
 > Read 28 characters
 > > Does not end with linebreak.

readFileWithReadLine
 > Read 28 characters
 > > Does not end with linebreak.

Answer 4

编辑:从go1.1开始,惯用的解决方案是使用bufio.Scanner

我写了一种方法来轻松读取文件中的每一行.Readln(*bufio.Reader)函数从底层的bufio.Reader结构返回一行(sans \n).

// Readln returns a single line (without the ending \n)
// from the input buffered reader.
// An error is returned iff there is an error with the
// buffered reader.
func Readln(r *bufio.Reader) (string, error) {
  var (isPrefix bool = true
       err error = nil
       line, ln []byte
      )
  for isPrefix && err == nil {
      line, isPrefix, err = r.ReadLine()
      ln = append(ln, line...)
  }
  return string(ln),err
}

您可以使用Readln读取文件中的每一行.以下代码读取文件中的每一行,并将每行输出到stdout.

f, err := os.Open(fi)
if err != nil {
    fmt.Printf("error opening file: %v\n",err)
    os.Exit(1)
}
r := bufio.NewReader(f)
s, e := Readln(r)
for e == nil {
    fmt.Println(s)
    s,e = Readln(r)
}

干杯!

Answer 5

有两种常见的逐行读取文件的方法.

1. 使用bufio.Scanner
2. 在bufio.Reader中使用ReadString/ReadBytes/....

在我的测试用例中,~250MB,~2,500,000行,bufio.Scanner(使用时间:0.395491384s)比bufio.Reader.ReadString(time_used:0.446867622s)快.

源代码:https://github.com/xpzouying/go-practice/tree/master/read_file_line_by_line

读取文件使用bufio.Scanner,

func scanFile() {
    f, err := os.OpenFile(logfile, os.O_RDONLY, os.ModePerm)
    if err != nil {
        log.Fatalf("open file error: %v", err)
        return
    }
    defer f.Close()

    sc := bufio.NewScanner(f)
    for sc.Scan() {
        _ = sc.Text()  // GET the line string
    }
    if err := sc.Err(); err != nil {
        log.Fatalf("scan file error: %v", err)
        return
    }
}

读取文件使用bufio.Reader,

func readFileLines() {
    f, err := os.OpenFile(logfile, os.O_RDONLY, os.ModePerm)
    if err != nil {
        log.Fatalf("open file error: %v", err)
        return
    }
    defer f.Close()

    rd := bufio.NewReader(f)
    for {
        line, err := rd.ReadString('\n')
        if err != nil {
            if err == io.EOF {
                break
            }

            log.Fatalf("read file line error: %v", err)
            return
        }
        _ = line  // GET the line string
    }
}

Answer 6

您还可以使用ReadString和\n作为分隔符:

  f, err := os.Open(filename)
  if err != nil {
    fmt.Println("error opening file ", err)
    os.Exit(1)
  }
  defer f.Close()
  r := bufio.NewReader(f)
  for {
    path, err := r.ReadString(10) // 0x0A separator = newline
    if err == io.EOF {
      // do something here
      break
    } else if err != nil {
      return err // if you return error
    }
  }

Answer 7

Example from this gist

func readLine(path string) {
  inFile, err := os.Open(path)
  if err != nil {
     fmt.Println(err.Error() + `: ` + path)
     return
  }
  defer inFile.Close()

  scanner := bufio.NewScanner(inFile)
  for scanner.Scan() {
    fmt.Println(scanner.Text()) // the line
  }
}

but this gives an error when there is a line that larger than Scanner's buffer.

When that happened, what I do is use reader := bufio.NewReader(inFile) create and concat my own buffer either using ch, err := reader.ReadByte() or len, err := reader.Read(myBuffer)

Answer 8

另一种方法是使用io/ioutil和strings库读取整个文件的字节，将它们转换为字符串，并使用“ \n”（换行符）字符作为分隔符来分割它们，例如：

import (
    "io/ioutil"
    "strings"
)

func main() {
    bytesRead, _ := ioutil.ReadFile("something.txt")
    fileContent := string(bytesRead)
    lines := strings.Split(fileContent, "\n")
}

从技术上讲，您不是逐行读取文件，但是您可以使用此技术解析每一行。此方法适用于较小的文件。如果您尝试解析大型文件，请使用逐行读取的技术之一。

Answer 9

bufio.Reader.ReadLine()运行良好。但是，如果您想通过字符串读取每一行，请尝试使用ReadString('\n')。它不需要重新发明轮子。