如何用C语言读取大数据？

Question

我练习用 C 语言处理文件。我需要读取一个记录了大约 9000000 个数据的文件。当我读取前 9000 条数据时它起作用了。但是，当我设置 NUM_DATA = 9000000 时，它总是崩溃。我该如何解决这个问题？

#include <stdio.h>
#include <string.h>
#include <ctype.h>
#include <stdlib.h>

#define MAX_LENGTH 10
#define NUM_DATA 9000000

int main(int argc, char **argv){

    char **signal = malloc( NUM_DATA * MAX_LENGTH * sizeof(char *) );
    char **svid = malloc( NUM_DATA * MAX_LENGTH * sizeof(char *) );

    char GetString[110];
    int num;
    double tw[NUM_DATA], ip[NUM_DATA], qp[NUM_DATA]
    char a[20], b[20], c[20], d[20], e[20], f[20], g[20], h[20], z[20], x[20], w[20];

    FILE *fIn;

    fIn = fopen(argv[1], "r");

    // read header
    fgets(GetString, 109, fIn);
    fgets(GetString, 109, fIn);

    // read data
    num = 0;
    while (!feof(fIn)){

        fscanf(fIn, "%[^','],%[^','],%[^','],%[^','],%[^','],%[^','],%[^','],%[^','],%[^','],%[^','],%f",
               a, b, c, d, e, f, g, h, z, x, w);

        tw[num] = atof(a);
        ip[num] = atof(z);
        qp[num] = atof(x);
        signal[num] = malloc(MAX_LENGTH * sizeof(char *));
        strcpy(signal[num], f);
        strcpy(svid[num], h);
        num++;
    }
    fclose(fIn);
}

文件的格式：

353700.000,1352,0.020000,0.000000,1,ISR1,主要,E11,173,14,56.102000 353700.000,1353,0.020000,0.000000,1,ISR1,主要,E11,156,7,14.367 000 353700.000,1354,0.020000,0.000000, 1,ISR1,主,E11,161,14,40.136000 353700.000,1355,0.020000,0.000000,1,ISR1,主,E11,316,23,31.326000 353700.000,1356,0.020000,0.00 0000,1,ISR1,主,E11, 152,-1,3.806000 353700.000,1357,0.020000,0.000000,2,ISR1,主,G14,-101,10,51.833000 353700.000,1358,0.020000,0.000000,3,ISR1,主,G24,434,-3,56.414000 353700.000,1359,0.020000,0.000000,3,ISR1,主,G24,171,-4,17.890000 353700.000,1360,0.020000,0.000000,3,ISR1,主,G24,474,6,52.542 000

......

Answer 1

该大小的本地数组会使堆栈溢出。相反，动态分配它们。

double *tw = malloc(NUM_DATA * sizeof(double));
double *ip = malloc(NUM_DATA * sizeof(double));
double *qp = malloc(NUM_DATA * sizeof(double));