小编Wil*_*sus的帖子

谷歌数据流“找不到方案 gs 的文件系统”

我正在尝试执行 Google Dataflow 应用程序,但它抛出此异常

java.lang.IllegalArgumentException: No filesystem found for scheme gs
    at org.apache.beam.sdk.io.FileSystems.getFileSystemInternal(FileSystems.java:459)
    at org.apache.beam.sdk.io.FileSystems.matchNewResource(FileSystems.java:529)
    at org.apache.beam.sdk.io.FileBasedSink.convertToFileResourceIfPossible(FileBasedSink.java:213)
    at org.apache.beam.sdk.io.TextIO$TypedWrite.to(TextIO.java:700)
    at org.apache.beam.sdk.io.TextIO$Write.to(TextIO.java:1028)
    at br.com.sulamerica.mecsas.ExportacaoDadosPipeline.main(ExportacaoDadosPipeline.java:52)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:282)
    at java.lang.Thread.run(Thread.java:748)
Run Code Online (Sandbox Code Playgroud)

这是我的流水线代码的一部分

Pipeline.create()
        .apply(PubsubIO.readStrings().fromSubscription(subscription))
        .apply(new KeyExportacaoDadosToEntityTransform())
        .apply(new ListKeyEmpresaSelecionadasTransform())
        .apply(ParDo.of(new DoFn<List<Entity>, String>() {
            @ProcessElement
            public void processElement(ProcessContext c){
                c.output(
                    c.element().stream()
                        .map(e-> e.getString("dscRazaoSocial"))
                        .collect(Collectors.joining("\r\n"))
                );
            }
        }))
        .apply(TextIO.write().to("gs://<my bucket>"))
        .getPipeline()
    .run();
Run Code Online (Sandbox Code Playgroud)

这是用于执行我的管道的命令

mvn -Pdataflow-runner compile exec:java \
  -Dexec.mainClass=br.com.xpto.foo.ExportacaoDadosPipeline \
  -Dexec.args="--project=<projectID>\
  --stagingLocation=gs://dataflow-xpto/exportacao/staging \
  --output=gs://dataflow-xpto/exportacao/output \
  --runner=DataflowRunner"  
Run Code Online (Sandbox Code Playgroud)

java exception google-cloud-storage google-cloud-platform google-cloud-dataflow

5
推荐指数
1
解决办法
1871
查看次数