从C#网站获取HTML代码

Question

如何从网站获取HTML代码,保存它,并通过LINQ表达式查找一些文本？

我正在使用以下代码来获取网页的来源:

public static String code(string Url)
{
    HttpWebRequest myRequest = (HttpWebRequest)WebRequest.Create(Url);
    myRequest.Method = "GET";
    WebResponse myResponse = myRequest.GetResponse();
    StreamReader sr = new StreamReader(myResponse.GetResponseStream(), System.Text.Encoding.UTF8);
    string result = sr.ReadToEnd();
    sr.Close();
    myResponse.Close();

    return result;
 }

如何在网页源中的div中找到文本？

Answer 1

更好的是,您可以使用Webclient类来简化任务:

using System.Net;

using (WebClient client = new WebClient())
{
    string htmlCode = client.DownloadString("http://somesite.com/default.html");
}

Answer 2

从网站获取HTML代码.你可以使用这样的代码.

string urlAddress = "http://google.com";

HttpWebRequest request = (HttpWebRequest)WebRequest.Create(urlAddress);
HttpWebResponse response = (HttpWebResponse)request.GetResponse();

if (response.StatusCode == HttpStatusCode.OK)
{
  Stream receiveStream = response.GetResponseStream();
  StreamReader readStream = null;

  if (String.IsNullOrWhiteSpace(response.CharacterSet))
     readStream = new StreamReader(receiveStream);
  else
     readStream = new StreamReader(receiveStream, Encoding.GetEncoding(response.CharacterSet));

  string data = readStream.ReadToEnd();

  response.Close();
  readStream.Close();
}

这将为您提供网站上返回的HTML代码.但是通过LINQ查找文本并不容易.也许最好使用正则表达式,但这不适合HTML代码

Answer 3

最好的用途是HTMLAgilityPack.您还可以根据从检索到的页面中选择元素的需要来研究使用Fizzler或CSQuery.使用LINQ或Regukar表达式只是容易出错,特别是当HTML可能格式错误,缺少结束标记,嵌套子元素等时.

您需要将页面流式传输到HtmlDocument对象,然后选择所需的元素.

// Call the page and get the generated HTML
var doc = new HtmlAgilityPack.HtmlDocument();
HtmlAgilityPack.HtmlNode.ElementsFlags["br"] = HtmlAgilityPack.HtmlElementFlag.Empty;
doc.OptionWriteEmptyNodes = true;

try
{
    var webRequest = HttpWebRequest.Create(pageUrl);
    Stream stream = webRequest.GetResponse().GetResponseStream();
    doc.Load(stream);
    stream.Close();
}
catch (System.UriFormatException uex)
{
    Log.Fatal("There was an error in the format of the url: " + itemUrl, uex);
    throw;
}
catch (System.Net.WebException wex)
{
    Log.Fatal("There was an error connecting to the url: " + itemUrl, wex);
    throw;
}

//get the div by id and then get the inner text 
string testDivSelector = "//div[@id='test']";
var divString = doc.DocumentNode.SelectSingleNode(testDivSelector).InnerHtml.ToString();

[编辑]其实,报废.最简单的方法是使用FizzlerEx,这是原始Fizzler项目的更新jQuery/CSS3选择器实现.

直接从其网站代码示例:

using HtmlAgilityPack;
using Fizzler.Systems.HtmlAgilityPack;

//get the page
var web = new HtmlWeb();
var document = web.Load("http://example.com/page.html");
var page = document.DocumentNode;

//loop through all div tags with item css class
foreach(var item in page.QuerySelectorAll("div.item"))
{
    var title = item.QuerySelector("h3:not(.share)").InnerText;
    var date = DateTime.Parse(item.QuerySelector("span:eq(2)").InnerText);
    var description = item.QuerySelector("span:has(b)").InnerHtml;
}

我认为它不会比这更简单.

Answer 4

我正在使用AngleSharp并且对它非常满意。

这是一个如何获取页面的简单示例：

var config = Configuration.Default.WithDefaultLoader();
var document = await BrowsingContext.New(config).OpenAsync("https://www.google.com");

现在您在文档变量中有一个网页。然后就可以很方便的通过LINQ或者其他方式访问了。例如，如果您想从 HTML 表中获取字符串值：

var someStringValue = document.All.Where(m =>
        m.LocalName == "td" &&
        m.HasAttribute("class") &&
        m.GetAttribute("class").Contains("pid-1-bid")
    ).ElementAt(0).TextContent.ToString();

要使用 CSS 选择器，请参阅AngleSharp 示例。

Answer 5

这是使用HttpWebRequest类获取URL 的示例

private void buttonl_Click(object sender, EventArgs e) 
{ 
    String url = TextBox_url.Text;
    HttpWebRequest request = (HttpWebRequest) WebRequest.Create(url); 
    HttpWebResponse response = (HttpWebResponse) request.GetResponse(); 
    StreamReader sr = new StreamReader(response.GetResponseStream()); 
    richTextBox1.Text = sr.ReadToEnd(); 
    sr.Close(); 
}