如何克服OutOfMemoryException从API中提取大型xml文档？

Question

我从API中提取1M +记录.拉动工作正常,但在尝试ReadToEnd进入字符串变量时,我遇到内存不足异常.

这是代码:

        XDocument xmlDoc = new XDocument();

        HttpWebRequest client = (HttpWebRequest)WebRequest.Create(uri);
        client.Timeout = 2100000;//35 minutes
        WebResponse apiResponse = client.GetResponse();

        Stream receivedStream = apiResponse.GetResponseStream();
        StreamReader reader = new StreamReader(receivedStream);

        string s = reader.ReadToEnd();

堆栈跟踪:

at System.Text.StringBuilder.ToString()
at System.IO.StreamReader.ReadToEnd()
at MyApplication.DataBuilder.getDataFromAPICall(String uri) in
    c:\Users\RDESLONDE\Documents\Projects\MyApplication\MyApplication\DataBuilder.cs:line 578
at MyApplication.DataBuilder.GetDataFromAPIAsXDoc(String uri) in
c:\Users\RDESLONDE\Documents\Projects\MyApplication\MyApplication\DataBuilder.cs:line 543

我该怎么做才能解决这个问题？

Answer 1

听起来你的文件对你的环境来说太大了.为大文件加载DOM可能会有问题,尤其是在使用win32平台时(您没有说明是否是这种情况).

您可以将XmlReader的速度和内存效率与XElement/Xnode等的便利性结合起来,并使用XStreamingElement在处理后保存转换后的内容.这对于大文件来说更具内存效率

这是伪代码中的一个例子:

    // use a XStreamingElement for writing
    var st = new XStreamingElement("root"); 
    using(var xr = new XmlTextReader(stream))
    {
        while (xr.Read())
        {
            // whatever you're interested in
            if (xr.NodeType == XmlNodeType.Element) 
            {
                var node = XNode.ReadFrom(xr) as XElement;
                if (node != null)
                {
                    ProcessNode(node);
                    st.Add(node);
                }
            }

        }
    }
    st.Save(outstream); // or st.WriteTo(xmlwriter);