那些遇到过的问题

HtmlUnit不创建HtmlPage对象

qua*_*285 4 html javascript htmlunit web-scraping headless-browser

我是HtmlUnit的新手，我正在尝试抓取一个使用Javascript编辑代码的网站。我听说HtmlUnit是最好的方法，因为它使用无头浏览器返回最终代码。

但是，正如您将看到的那样，我什至无法创建一个HtmlPage对象，而不会抛出一个巨大且无法理解的异常（至少考虑到我对HtmlUnit几乎为空的经验）。

这是我的代码：

import com.gargoylesoftware.htmlunit.*;
import com.gargoylesoftware.htmlunit.html.HtmlPage;

public class Main {

    public static void main(String[] args) {
        Main scraper = new Main();
        scraper.testingGargoyle();


    }

    private void testingGargoyle() {
        String myUrl = "https://www.wearvr.com/#game_id=game_4";
        WebClient webClient = new WebClient();
        try {
            HtmlPage myPage = ((HtmlPage) webClient.getPage(myUrl));
        } catch (FailingHttpStatusCodeException | IOException e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        }
    }
}
Run Code Online (Sandbox Code Playgroud)

这是抛出的异常：

Apr 30, 2015 5:43:50 PM com.gargoylesoftware.htmlunit.IncorrectnessListenerImpl notify
WARNING: Obsolete content type encountered: 'application/x-javascript'.
Apr 30, 2015 5:43:50 PM com.gargoylesoftware.htmlunit.javascript.StrictErrorReporter runtimeError
SEVERE: runtimeError: message=[The data necessary to complete this operation is not yet available.] sourceName=[https://load.sumome.com/] line=[1] lineSource=[null] lineOffset=[0]
Exception in thread "main" ======= EXCEPTION START ========
EcmaError: lineNumber=[19] column=[0] lineSource=[<no source>] name=[TypeError] sourceName=[https://www.wearvr.com/assets/scripts/bundle.b4038a088bb1abfcf55c.js] message=[TypeError: Cannot find function bind in object function (e, n, r) {...}. (https://www.wearvr.com/assets/scripts/bundle.b4038a088bb1abfcf55c.js#19)]
com.gargoylesoftware.htmlunit.ScriptException: TypeError: Cannot find function bind in object function (e, n, r) {...}. (https://www.wearvr.com/assets/scripts/bundle.b4038a088bb1abfcf55c.js#19)
    at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine$HtmlUnitContextAction.run(JavaScriptEngine.java:847)
    at net.sourceforge.htmlunit.corejs.javascript.Context.call(Context.java:620)
    at net.sourceforge.htmlunit.corejs.javascript.ContextFactory.call(ContextFactory.java:513)
    at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine.execute(JavaScriptEngine.java:733)
    at com.gargoylesoftware.htmlunit.html.HtmlPage.loadExternalJavaScriptFile(HtmlPage.java:1096)
    at com.gargoylesoftware.htmlunit.html.HtmlScript.executeScriptIfNeeded(HtmlScript.java:395)
    at com.gargoylesoftware.htmlunit.html.HtmlScript$3.execute(HtmlScript.java:270)
    at com.gargoylesoftware.htmlunit.html.HtmlScript.onAllChildrenAddedToPage(HtmlScript.java:290)
    at com.gargoylesoftware.htmlunit.html.HTMLParser$HtmlUnitDOMBuilder.endElement(HTMLParser.java:793)
    at org.apache.xerces.parsers.AbstractSAXParser.endElement(Unknown Source)
    at com.gargoylesoftware.htmlunit.html.HTMLParser$HtmlUnitDOMBuilder.endElement(HTMLParser.java:751)
    at org.cyberneko.html.HTMLTagBalancer.callEndElement(HTMLTagBalancer.java:1170)
    at org.cyberneko.html.HTMLTagBalancer.endElement(HTMLTagBalancer.java:1072)
    at org.cyberneko.html.filters.DefaultFilter.endElement(DefaultFilter.java:206)
    at org.cyberneko.html.filters.NamespaceBinder.endElement(NamespaceBinder.java:330)
    at org.cyberneko.html.HTMLScanner$ContentScanner.scanEndElement(HTMLScanner.java:3126)
    at org.cyberneko.html.HTMLScanner$ContentScanner.scan(HTMLScanner.java:2093)
    at org.cyberneko.html.HTMLScanner.scanDocument(HTMLScanner.java:920)
    at org.cyberneko.html.HTMLConfiguration.parse(HTMLConfiguration.java:499)
    at org.cyberneko.html.HTMLConfiguration.parse(HTMLConfiguration.java:452)
    at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
    at com.gargoylesoftware.htmlunit.html.HTMLParser$HtmlUnitDOMBuilder.parse(HTMLParser.java:1017)
    at com.gargoylesoftware.htmlunit.html.HTMLParser.parse(HTMLParser.java:248)
    at com.gargoylesoftware.htmlunit.html.HTMLParser.parseHtml(HTMLParser.java:194)
    at com.gargoylesoftware.htmlunit.DefaultPageCreator.createHtmlPage(DefaultPageCreator.java:268)
    at com.gargoylesoftware.htmlunit.DefaultPageCreator.createPage(DefaultPageCreator.java:156)
    at com.gargoylesoftware.htmlunit.WebClient.loadWebResponseInto(WebClient.java:471)
    at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:345)
    at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:410)
    at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:395)
    at Main.testingGargoyle(Main.java:19)
    at Main.main(Main.java:10)
Caused by: net.sourceforge.htmlunit.corejs.javascript.EcmaError: TypeError: Cannot find function bind in object function (e, n, r) {...}. (https://www.wearvr.com/assets/scripts/bundle.b4038a088bb1abfcf55c.js#19)
    at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.constructError(ScriptRuntime.java:3629)
    at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.constructError(ScriptRuntime.java:3613)
    at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.typeError(ScriptRuntime.java:3634)
    at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.typeError2(ScriptRuntime.java:3650)
    at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.notFunctionError(ScriptRuntime.java:3714)
    at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.getPropFunctionAndThisHelper(ScriptRuntime.java:2233)
    at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.getPropFunctionAndThis(ScriptRuntime.java:2215)
    at net.sourceforge.htmlunit.corejs.javascript.Interpreter.interpretLoop(Interpreter.java:1333)
    at net.sourceforge.htmlunit.corejs.javascript.Interpreter.interpret(Interpreter.java:798)
    at net.sourceforge.htmlunit.corejs.javascript.InterpretedFunction.call(InterpretedFunction.java:105)
    at net.sourceforge.htmlunit.corejs.javascript.ContextFactory.doTopCall(ContextFactory.java:411)
    at com.gargoylesoftware.htmlunit.javascript.HtmlUnitContextFactory.doTopCall(HtmlUnitContextFactory.java:309)
    at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:3057)
    at net.sourceforge.htmlunit.corejs.javascript.InterpretedFunction.exec(InterpretedFunction.java:115)
    at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine$3.doRun(JavaScriptEngine.java:724)
    at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine$HtmlUnitContextAction.run(JavaScriptEngine.java:832)
    ... 31 more
Enclosed exception: 
net.sourceforge.htmlunit.corejs.javascript.EcmaError: TypeError: Cannot find function bind in object function (e, n, r) {...}. (https://www.wearvr.com/assets/scripts/bundle.b4038a088bb1abfcf55c.js#19)
    at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.constructError(ScriptRuntime.java:3629)
    at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.constructError(ScriptRuntime.java:3613)
    at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.typeError(ScriptRuntime.java:3634)
    at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.typeError2(ScriptRuntime.java:3650)
    at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.notFunctionError(ScriptRuntime.java:3714)
    at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.getPropFunctionAndThisHelper(ScriptRuntime.java:2233)
    at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.getPropFunctionAndThis(ScriptRuntime.java:2215)
    at net.sourceforge.htmlunit.corejs.javascript.Interpreter.interpretLoop(Interpreter.java:1333)
    at script(https://www.wearvr.com/assets/scripts/bundle.b4038a088bb1abfcf55c.js:19)
    at script.r(https://www.wearvr.com/assets/scripts/bundle.b4038a088bb1abfcf55c.js:16)
    at script.r(https://www.wearvr.com/assets/scripts/bundle.b4038a088bb1abfcf55c.js:384)
    at script(https://www.wearvr.com/assets/scripts/bundle.b4038a088bb1abfcf55c.js:16)
    at script(https://www.wearvr.com/assets/scripts/bundle.b4038a088bb1abfcf55c.js:16)
    at script.t(https://www.wearvr.com/assets/scripts/bundle.b4038a088bb1abfcf55c.js:1)
    at script(https://www.wearvr.com/assets/scripts/bundle.b4038a088bb1abfcf55c.js:16)
    at script(https://www.wearvr.com/assets/scripts/bundle.b4038a088bb1abfcf55c.js:16)
    at script.t(https://www.wearvr.com/assets/scripts/bundle.b4038a088bb1abfcf55c.js:1)
    at script(https://www.wearvr.com/assets/scripts/bundle.b4038a088bb1abfcf55c.js:7)
    at script.t(https://www.wearvr.com/assets/scripts/bundle.b4038a088bb1abfcf55c.js:1)
    at script(https://www.wearvr.com/assets/scripts/bundle.b4038a088bb1abfcf55c.js:463)
    at script(https://www.wearvr.com/assets/scripts/bundle.b4038a088bb1abfcf55c.js:463)
    at script.t(https://www.wearvr.com/assets/scripts/bundle.b4038a088bb1abfcf55c.js:1)
    at script(https://www.wearvr.com/assets/scripts/bundle.b4038a088bb1abfcf55c.js:1)
    at script.t(https://www.wearvr.com/assets/scripts/bundle.b4038a088bb1abfcf55c.js:1)
    at script(https://www.wearvr.com/assets/scripts/bundle.b4038a088bb1abfcf55c.js:1)
    at script(https://www.wearvr.com/assets/scripts/bundle.b4038a088bb1abfcf55c.js:1)
    at net.sourceforge.htmlunit.corejs.javascript.Interpreter.interpret(Interpreter.java:798)
    at net.sourceforge.htmlunit.corejs.javascript.InterpretedFunction.call(InterpretedFunction.java:105)
    at net.sourceforge.htmlunit.corejs.javascript.ContextFactory.doTopCall(ContextFactory.java:411)
    at com.gargoylesoftware.htmlunit.javascript.HtmlUnitContextFactory.doTopCall(HtmlUnitContextFactory.java:309)
    at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:3057)
    at net.sourceforge.htmlunit.corejs.javascript.InterpretedFunction.exec(InterpretedFunction.java:115)
    at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine$3.doRun(JavaScriptEngine.java:724)
    at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine$HtmlUnitContextAction.run(JavaScriptEngine.java:832)
    at net.sourceforge.htmlunit.corejs.javascript.Context.call(Context.java:620)
    at net.sourceforge.htmlunit.corejs.javascript.ContextFactory.call(ContextFactory.java:513)
    at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine.execute(JavaScriptEngine.java:733)
    at com.gargoylesoftware.htmlunit.html.HtmlPage.loadExternalJavaScriptFile(HtmlPage.java:1096)
    at com.gargoylesoftware.htmlunit.html.HtmlScript.executeScriptIfNeeded(HtmlScript.java:395)
    at com.gargoylesoftware.htmlunit.html.HtmlScript$3.execute(HtmlScript.java:270)
    at com.gargoylesoftware.htmlunit.html.HtmlScript.onAllChildrenAddedToPage(HtmlScript.java:290)
    at com.gargoylesoftware.htmlunit.html.HTMLParser$HtmlUnitDOMBuilder.endElement(HTMLParser.java:793)
    at org.apache.xerces.parsers.AbstractSAXParser.endElement(Unknown Source)
    at com.gargoylesoftware.htmlunit.html.HTMLParser$HtmlUnitDOMBuilder.endElement(HTMLParser.java:751)
    at org.cyberneko.html.HTMLTagBalancer.callEndElement(HTMLTagBalancer.java:1170)
    at org.cyberneko.html.HTMLTagBalancer.endElement(HTMLTagBalancer.java:1072)
    at org.cyberneko.html.filters.DefaultFilter.endElement(DefaultFilter.java:206)
    at org.cyberneko.html.filters.NamespaceBinder.endElement(NamespaceBinder.java:330)
    at org.cyberneko.html.HTMLScanner$ContentScanner.scanEndElement(HTMLScanner.java:3126)
    at org.cyberneko.html.HTMLScanner$ContentScanner.scan(HTMLScanner.java:2093)
    at org.cyberneko.html.HTMLScanner.scanDocument(HTMLScanner.java:920)
    at org.cyberneko.html.HTMLConfiguration.parse(HTMLConfiguration.java:499)
    at org.cyberneko.html.HTMLConfiguration.parse(HTMLConfiguration.java:452)
    at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
    at com.gargoylesoftware.htmlunit.html.HTMLParser$HtmlUnitDOMBuilder.parse(HTMLParser.java:1017)
    at com.gargoylesoftware.htmlunit.html.HTMLParser.parse(HTMLParser.java:248)
    at com.gargoylesoftware.htmlunit.html.HTMLParser.parseHtml(HTMLParser.java:194)
    at com.gargoylesoftware.htmlunit.DefaultPageCreator.createHtmlPage(DefaultPageCreator.java:268)
    at com.gargoylesoftware.htmlunit.DefaultPageCreator.createPage(DefaultPageCreator.java:156)
    at com.gargoylesoftware.htmlunit.WebClient.loadWebResponseInto(WebClient.java:471)
    at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:345)
    at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:410)
    at com.gargoylesoftware.htmlunit.WebClient.getPage(WebClient.java:395)
    at Main.testingGargoyle(Main.java:19)
    at Main.main(Main.java:10)
======= EXCEPTION END ========
Run Code Online (Sandbox Code Playgroud)

我告诉过你，它很大。我如何解决这个问题并获得此页面的最终资源以便进行抓取？

提前致谢！

小智 5

引发异常的原因有多种，例如错误的html，脚本页面上的错误，找不到诸如CSS，脚本文件或图像文件之类的资源（例如 <img src="bla.gif"> <- bla.gif not found HTML404)

因此，我们使用这些选项来保持html导航，而不会停止我们使用的第一个错误/问题：

webClient.getOptions().setThrowExceptionOnScriptError(false);
webClient.getOptions().setThrowExceptionOnFailingStatusCode(false);
Run Code Online (Sandbox Code Playgroud)

您还可以使用以下方法实现空类，以停止htmlUnity在控制台上出现关于css / javaScript错误的详细信息：

webClient.setCssErrorHandler(new SilentCssErrorHandler());    
webClient.setJavaScriptErrorListener(new JavaScriptErrorListener(){});
Run Code Online (Sandbox Code Playgroud)

小样本测试用例：

@Test
public void TestCall() throws FailingHttpStatusCodeException, MalformedURLException, IOException {      
    WebClient webClient = new WebClient(BrowserVersion.CHROME);
    webClient.getOptions().setUseInsecureSSL(true); //ignore ssl certificate
    webClient.getOptions().setThrowExceptionOnScriptError(false);
    webClient.getOptions().setThrowExceptionOnFailingStatusCode(false);
    String url = "https://www.wearvr.com/#game_id=game_4";
    HtmlPage myPage = webClient.getPage(url);
    webClient.waitForBackgroundJavaScriptStartingBefore(200);
    webClient.waitForBackgroundJavaScript(20000);
    //do stuff on page ex: myPage.getElementById("main")
    //myPage.asXml() <- tags and elements
    System.out.println(myPage.asText());

}
Run Code Online (Sandbox Code Playgroud)

  • webClient.setJavaScriptErrorListener(new SilentJavaScriptErrorListener()); (2认同)

归档时间：

查看次数：

3607 次

最近记录：

10 年，7 月 前
相关归档
我可以将变量设置为undefined或将undefined作为参数传递吗？ 296
删除某个角色后的所有内容 252
设置功能参数的类型？ 133
使锚链接在其链接的上方的某些像素处 105
可读性用于从URL中提取文本的算法是什么？ 102
是否优化了任何Javascript引擎尾调用？ 90
JQuery Ajax - 如何在进行Ajax调用时检测网络连接错误 84
我如何有条件地包装React组件？ 66
为什么4不是Number的实例？ 64
使用边距设置跨距之间的距离 18
难疑归档
Python有三元条件运算符吗？ 5591
使用jQuery从下拉列表(选择框)中获取所选文本 2240
如何使div不大于其内容？ 1960
如何格式化Microsoft JSON日期？ 1954
如何仅列出两次提交之间更改的文件名？ 1807
在GitHub上使用https://时有没有办法跳过密码输入？ 1806
如何在没有换行或空格的情况下打印？ 1760
家谱软件中的循环 1594
如何完全删除使用init创建的git存储库？ 1358
这是什么意思？"'NSUnknownKeyException',原因:...此类不是键值X的键值编码兼容" 1143