Zbo*_*one 1 c# parallel-processing foreach plinq
我正在开发一款可在Google搜索结果网址中搜索电子邮件地址的应用.问题是它需要将每个页面中找到的值+它找到电子邮件的URL返回到包含2列的数据网格视图:电子邮件和URL.我正在使用Parallel.ForEach这个,但它当然返回随机URL,而不是它真正找到的电子邮件.
public static string htmlcon; //htmlsource
public static List<string> emailList = new List<string>();
public static string Get(string url, bool proxy)
{
htmlcon = "";
try
{
HttpWebRequest req = (HttpWebRequest)WebRequest.Create(url);
if (proxy)
req.Proxy = new WebProxy(proxyIP + ":" + proxyPort);
req.Method = "GET";
req.UserAgent = Settings1.Default.UserAgent;
if (Settings1.Default.EnableCookies == true)
{
CookieContainer cont = new CookieContainer();
req.CookieContainer = cont;
}
WebResponse resp = req.GetResponse();
StreamReader SR = new StreamReader(resp.GetResponseStream());
htmlcon = SR.ReadToEnd();
Thread.Sleep(400);
resp.Close();
SR.Close();
}
catch (Exception)
{
Thread.Sleep(500);
}
return htmlcon;
}
private void copyMails(string url)
{
string emailPat = @"(\b[a-zA-Z0-9._%-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,4}\b)";
MatchCollection mailcol = Regex.Matches(htmlcon, emailPat, RegexOptions.Singleline);
foreach (Match mailMatch in mailcol)
{
email = mailMatch.Groups[1].Value;
if (!emailList.Contains(email))
{
emailList.Add(email);
Action dgeins = () => mailDataGrid.Rows.Insert(0, email, url);
mailDataGrid.BeginInvoke(dgeins);
}
}
}
private void SEbgWorker_DoWork(object sender, DoWorkEventArgs e)
{
//ALOT OF IRRELEVAMT STUFF BEING RUN
Parallel.ForEach(allSElist.OfType<string>(), (s) =>
{
//Get URL
Get(s, Settings1.Default.Proxyset);
//match mails 1st page
copyMails(s);
});
}
所以就是这样:我执行一个Get请求(其中"s"是列表中的URL),然后从URL的html源执行copyMails.它使用正则表达式来复制电子邮件.如果我没有并行执行,则会返回datagridview中每封电子邮件的正确URL.如何在数据网格视图中仍然获得正确的匹配？
谢谢
你最好使用PLINQ Where来过滤(伪代码):
var results = from i in input.AsParallel()
let u = get the URL from i
let d = get the data from u
let v = try get the value from d
where v is found
select new {
Url = u,
Value = v
};
下面的AsParallel是TPL的执行手段LINQ运营商(Select,Where,...)被使用.
更新:现在有更多信息
首先,您的代码中存在许多问题:
该变量htmlcon仅static由多个线程直接使用.这可能是您潜在的问题.只考虑两个输入值.第一个Get完成设置htmlcon,在该线程的调用copyMails开始之前,第二个线程Get完成其HTML GET并写入htmlcon.随着'电子邮件
该列表emailList也可以在没有多个线程锁定的情况下访问..NET(以及任何其他编程平台)中的大多数集合类型都不是线程安全的,您需要一次限制对单个线程的访问.
您正在混合各种方法中的各种活动.考虑应用单一责任原则.
Thread.Sleep处理异常？!如果你不能处理异常(即解决条件),那么什么都不做.在这种情况下,如果操作抛出,那么Parallel.Foreach将抛出:直到您定义如何处理HTML GET失败为止.
三点建议:
根据我的经验,干净的代码(以强迫程度)使事情变得更容易:格式的细节无关紧要(一个真正的支撑风格更好,但一致性是关键).只是通过并清理格式显示问题#1和#2.
良好的命名.除非这是域名的重要术语,否则不要缩写超过几行代码使用的任何内容.例如.s对于并行循环中的action参数实际上是一个url所以称之为.这种事情立即使代码更容易遵循.
考虑一下电子邮件的正则表达式:有许多有效的电子邮件不匹配(例如,使用+提供多个逻辑地址:exmaple+one@gamil.com将被传递到example@gmail.com,然后可以用于本地规则).另外一个撇号(" '")是一个有效的角色(而且已知的人因为网站错误而拒绝了他们的地址而感到沮丧).
第二:相对直接的清理:
public static string Get(string url, bool proxy) {
HttpWebRequest req = (HttpWebRequest)WebRequest.Create(url);
if (proxy) {
req.Proxy = new WebProxy(proxyIP + ":" + proxyPort);
}
req.Method = "GET";
req.UserAgent = Settings1.Default.UserAgent;
if (Settings1.Default.EnableCookies == true) {
CookieContainer cont = new CookieContainer();
req.CookieContainer = cont;
}
using (WebResponse resp = req.GetResponse())
using (StreamReader SR = new StreamReader(resp.GetResponseStream())) {
return SR.ReadToEnd();
}
}
private static Regex emailMatcher = new Regex(@"(\b[a-zA-Z0-9._%-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,4}\b)", RegexOptions.Singleline);
private static string[] ExtractEmails(string htmlContent) {
return emailMatcher.Matches(htmlContent).OfType<Match>
.Select(m => m.Groups[1].Value)
.Distinct()
.ToArray();
}
private void SEbgWorker_DoWork(object sender, DoWorkEventArgs e) {
Parallel.ForEach(allSElist.OfType<string>(), url => {
var htmlContent = Get(url, Settings1.Default.Proxyset);
var emails = ExtractEmails(htmlContent);
foreach (var email in emails) {
Action dgeins = () => mailDataGrid.Rows.Insert(0, email, url);
mailDataGrid.BeginInvoke(dgeins);
}
}
我在这里:
using语句自动清理资源.Regex明确记录为具有线程安全的实例方法.所以我只需要一个实例.ExtractEmails因为提取不使用URL.Get现在只执行HTML get,ExtreactEMail只执行提取第三:上面将阻止最慢操作的线程:HTML GET.
真正的并发好处是HttpWebRequest.GetResponse用它们的异步等价物替换和读取响应流.
使用Task将是.NET 4中的答案,但您需要直接使用Stream和编码自己,因为StreamReader不提供任何BeginABC/ EndABC方法对.但.NET 4.5几乎就在这里,所以应用一些async/ await:
ExtractEMails.Get 现在是异步的,既不阻止HTTP GET也不读取结果.SEbgWorker_DoWork直接使用Tasks来避免混合太多不同的方式来使用TPL.因为Get返回a Task<string>可以简单地继续(当它没有失败时 - 除非你指定,否则ContinueWith只有在前一个任务成功完成时才会继续):这应该适用于.NET 4.5,但如果没有一组有效的URL,我将无法测试.
public static async Task<string> Get(string url, bool proxy) {
HttpWebRequest req = (HttpWebRequest)WebRequest.Create(url);
if (proxy) {
req.Proxy = new WebProxy(proxyIP + ":" + proxyPort);
}
req.Method = "GET";
req.UserAgent = Settings1.Default.UserAgent;
if (Settings1.Default.EnableCookies == true) {
CookieContainer cont = new CookieContainer();
req.CookieContainer = cont;
}
using (WebResponse resp = await req.GetResponseAsync())
using (StreamReader SR = new StreamReader(resp.GetResponseStream())) {
return await SR.ReadToEndAsync();
}
}
private static Regex emailMatcher = new Regex(@"(\b[a-zA-Z0-9._%-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,4}\b)", RegexOptions.Singleline);
private static string[] ExtractEmails(string htmlContent) {
return emailMatcher.Matches(htmlContent).OfType<Match>
.Select(m => m.Groups[1].Value)
.Distinct()
.ToArray();
}
private void SEbgWorker_DoWork(object sender, DoWorkEventArgs e) {
tasks = allSElist.OfType<string>()
.Select(url => {
return Get(url, Settings1.Default.Proxyset)
.ContinueWith(htmlContentTask => {
// No TaskContinuationOptions, so know always OK here
var htmlContent = htmlContentTask.Result;
var emails = ExtractEmails(htmlContent);
foreach (var email in emails) {
// No InvokeAsync on WinForms, so do this the old way.
Action dgeins = () => mailDataGrid.Rows.Insert(0, email, url);
mailDataGrid.BeginInvoke(dgeins);
}
});
});
tasks.WaitAll();
}