使用R,RCurl进行多个Web表挖掘

Question

首先,提前感谢任何回复.

我需要通过在各自的网页中加入一些较小的表来获取表格.到目前为止,我已经能够提取信息,但未能使用循环自动完成.到目前为止,我的命令是:

library(RCurl)
library(XML)
# index <- toupper(letters)
# EDIT:
index <- LETTERS

index[1] <- "0-A"
url <- paste("www.citefactor.org/journal-impact-factor-list-2014_", index, ".html", sep="", collapse=";")
urls <- strsplit(url, ";") [[1]]

这是我的循环尝试:

read.html.tab <- function(url){
 require(RCurl)
 require(XML)
 uri <- url
 tabs <- NULL
 for (i in uri){
  tabs <- getURL(uri)
  tabs <- readHTMLTable(tabs, stringsAsFactors = F)
  tab1 <- as.data.frame(tabs)
  }
 tab1
 }

如果我尝试使用该read.html.tab功能:

tab0 <- read.html.tab(urls)

我收到以下错误: Error in data.frame(`Search Journal Impact Factor List 2014` = list(`0-A` = "N", : arguments imply differing number of rows: 1, 1100, 447, 874, 169, 486, 201, 189, 172, 837....

但是,如果urls只有一个元素,则该函数有效:

tabA <- read.html.tab(urls[1])
tabB <- read.html.tab(urls[2]) 
tab.if <- rbind(tabA,tabB)

ifacs <- tab.if[,27:ncol(tab.if)]
View(ifacs)

好像我不明白循环是如何工作的......

Answer 1

强制性的Hadleyverse回答:

library(rvest)
library(dplyr)
library(magrittr)
library(pbapply)

urls <- sprintf("http://www.citefactor.org/journal-impact-factor-list-2014_%s.html", 
                c("0-A", LETTERS[-1]))

dat <- urls %>%
  pblapply(function(url) 
    html(url) %>% html_table(header=TRUE) %>% extract2(2)) %>%
  bind_rows()

glimpse(dat)

## Observations: 1547
## Variables:
## $ INDEX     (int) 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14,...
## $ JOURNAL   (chr) "4OR-A Quarterly Journal of Operations Researc...
## $ ISSN      (chr) "1619-4500", "0891-0162", "0149-1423", "1550-7...
## $ 2013/2014 (chr) "0.918", "0.608", "1.832", "3.905", "1.776", "...
## $ 2012      (chr) "0.73", "0.856", "1.768", "4.386", "1.584", "0...
## $ 2011      (chr) "0.323", "0.509", "1.831", "5.086", "1.432", "...
## $ 2010      (chr) "0.69", "0.56", "1.964", "3.942", "1.211", "0....
## $ 2009      (chr) "0.75", "-", "1.448", "3.54", "1.19", "0.293",...
## $ 2008      (chr) "-", "-", "1.364", "-", "1.445", "0.352", "1.4...

rvest给了我们html和html_table

我magrittr只使用extract2它,它只是包装[[和读取更好(IMO).

该pbapply包装包裹*apply功能并为您提供免费的进度条.

注意:bind_rows是最新的dplyr,所以在使用它之前抓住它.