检查远程/外部网页是否可用并获取状态代码

Question

我知道有很多关于使用System.Net.WebRequest来检查远程网页是否可用的问题和答案,但我没有发现这些方法对我的情况100%有用.

我需要检查页面是否可用,不返回404或500错误,如果有重定向,那么我想跟踪它们,直到找到工作页面.如果远程页面需要身份验证(401未经授权),我想知道,因为在某些情况下这可能是可以接受的.

发出请求并且远程服务器返回内部服务器错误(500)时,WebRequest引发异常.

我还发现301重定向也会引发异常,但在我的情况下,我想检查重定向是否是有效页面.

有没有其他方法可以检查页面是否存在,并且最好只有在获取标题时出现错误(即无效的域名等)才能获得实际的HTTP状态代码或异常？

这就是我现在正在做的事情,这还不够好......

protected bool URLExists(string url)
{
    bool result = false;

    try
    {
        HttpWebRequest webRequest = (HttpWebRequest)WebRequest.Create(url);
        webRequest.Timeout = 1200;
        webRequest.Method = "GET";
        //webRequest.AllowAutoRedirect = true;
        //webRequest.UserAgent = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/49.0.2623.110 Safari/537.36";

        HttpWebResponse response = null;

        try
        {
            response = (HttpWebResponse)webRequest.GetResponse();
            //result = true;

            int statusCode = (int)response.StatusCode;
            if (statusCode >= 100 && statusCode < 400) //Good requests
            {
                return true;
            }
            else if (statusCode >= 500 && statusCode <= 510) //Server Errors
            {
                return false;
            }
        }
        catch (WebException webException)
        {

        }
        finally
        {
            if (response != null)
            {
                response.Close();
            }
        }
    }
    catch (Exception ex)
    {

    }

    return result;
}

Answer 1

考虑一下这段代码,您需要获得200个响应状态代码以确保网页可用,如果重定向发生,请遵循终点URL响应状态代码.

public static bool URLExists(string url)
        {
            HttpStatusCode result = default(HttpStatusCode);

            var request =(HttpWebRequest)WebRequest.Create(url);
            request.AllowAutoRedirect = false;
            request.Method = "HEAD";
            try
            {
                using (var response = request.GetResponse() as HttpWebResponse)
                {
                    if (response != null)
                    {
                        if (response.StatusCode == HttpStatusCode.OK)
                            return true;
                        if (response.StatusCode == HttpStatusCode.Redirect)
                        {
                            string uriString = response.Headers["Location"];
                            return URLExists(uriString);
                        }

                        response.Close();
                    }
                }
                return false;
            }
            catch (WebException e)
            {
                using (WebResponse response = e.Response)
                {
                    HttpWebResponse httpResponse = (HttpWebResponse)response;
                    Console.WriteLine("Error code: {0}", httpResponse.StatusCode);
                    return false;
                }
            }
        }

够了吗？;)