使用带有大量压缩文本文件(1000多个文件,大小在100MB和1.5GB之间)的TextIO.Read转换,我们有时会收到以下错误:
java.util.zip.ZipException: too many length or distance symbols at
java.util.zip.InflaterInputStream.read(InflaterInputStream.java:164) at
java.util.zip.GZIPInputStream.read(GZIPInputStream.java:117) at
java.io.BufferedInputStream.fill(BufferedInputStream.java:246) at
java.io.BufferedInputStream.read1(BufferedInputStream.java:286) at
java.io.BufferedInputStream.read(BufferedInputStream.java:345) at
java.io.FilterInputStream.read(FilterInputStream.java:133) at
java.io.PushbackInputStream.read(PushbackInputStream.java:186) at
com.google.cloud.dataflow.sdk.runners.worker.TextReader$ScanState.readBytes(TextReader.java:261) at
com.google.cloud.dataflow.sdk.runners.worker.TextReader$TextFileIterator.readElement(TextReader.java:189) at
com.google.cloud.dataflow.sdk.runners.worker.FileBasedReader$FileBasedIterator.computeNextElement(FileBasedReader.java:265) at
com.google.cloud.dataflow.sdk.runners.worker.FileBasedReader$FileBasedIterator.hasNext(FileBasedReader.java:165) at
com.google.cloud.dataflow.sdk.util.common.worker.ReadOperation.runReadLoop(ReadOperation.java:169) at
com.google.cloud.dataflow.sdk.util.common.worker.ReadOperation.start(ReadOperation.java:118) at
com.google.cloud.dataflow.sdk.util.common.worker.MapTaskExecutor.execute(MapTaskExecutor.java:66) at
com.google.cloud.dataflow.sdk.runners.worker.DataflowWorker.executeWork(DataflowWorker.java:204) at
com.google.cloud.dataflow.sdk.runners.worker.DataflowWorker.doWork(DataflowWorker.java:151) at
com.google.cloud.dataflow.sdk.runners.worker.DataflowWorker.getAndPerformWork(DataflowWorker.java:118) at
com.google.cloud.dataflow.sdk.runners.worker.DataflowWorkerHarness$WorkerThread.call(DataflowWorkerHarness.java:139) at
com.google.cloud.dataflow.sdk.runners.worker.DataflowWorkerHarness$WorkerThread.call(DataflowWorkerHarness.java:124) at
java.util.concurrent.FutureTask.run(FutureTask.java:266) at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at
java.lang.Thread.run(Thread.java:745)
Run Code Online (Sandbox Code Playgroud)
在线搜索相同的ZipException,只会导致此回复:
当热部署程序在将应用程序完全复制到deploy目录之前尝试部署应用程序时,通常会发生Zip文件错误.如果复制文件需要几秒钟,这是相当常见的.解决方案是将文件复制到与应用程序服务器相同的磁盘分区上的临时目录,然后将该文件移动到deploy目录.
有没有其他人遇到类似的例外?或者无论如何我们可以解决这个问题?
是否可以创建从Pub/Sub读取数据并写入数据存储区的管道?在我的代码中,我将PubsubIO指定为输入,并应用窗口来获取有界PCollection,但似乎无法使用带有options.setStreaming的DatastoreIO.writeTo为true,而这是必需的以便使用PubsubIO作为输入.有没有解决的办法?或者是不可能从pubsub读取并写入数据存储区?
这是我的代码:
DataflowPipelineOptions options = PipelineOptionsFactory.create()
.as(DataflowPipelineOptions.class);
options.setRunner(DataflowPipelineRunner.class);
options.setProject(projectName);
options.setStagingLocation("gs://my-staging-bucket/staging");
options.setStreaming(true);
Pipeline p = Pipeline.create(options);
PCollection<String> input = p.apply(PubsubIO.Read.topic("projects/"+projectName+"/topics/event-streaming"));
PCollection<String> inputWindow = input.apply(Window.<String>into(FixedWindows.of(Duration.standardSeconds(5))).triggering(AfterPane.elementCountAtLeast(1)).discardingFiredPanes().withAllowedLateness(Duration.standardHours(1)));
PCollection<String> inputDecode = inputWindow.apply(ParDo.of(new DoFn<String, String>() {
private static final long serialVersionUID = 1L;
public void processElement(ProcessContext c) {
String msg = c.element();
byte[] decoded = Base64.decodeBase64(msg.getBytes());
String outmsg = new String(decoded);
c.output(outmsg);
}
}));
PCollection<DatastoreV1.Entity> inputEntity = inputDecode.apply(ParDo.of(new CreateEntityFn("stream", "events")));
inputEntity.apply(DatastoreIO.writeTo(datasetid));
p.run();
Run Code Online (Sandbox Code Playgroud)
这是我得到的例外:
Exception in thread "main" java.lang.UnsupportedOperationException: The Write transform is not supported by the …Run Code Online (Sandbox Code Playgroud) google-cloud-datastore google-cloud-pubsub google-cloud-dataflow
我正在阅读“ 使用自定义机器创建实例”,这看起来确实很有趣,但是我不知道如何使用自定义机器(例如8个CPU和15GB)来设置Dataflow流作业。可能吗?
我的Cloud Dataflow作业运行了4个小时后,神秘地失败了,因为一个工作人员四次(在一小时的时间内)抛出了该异常。异常堆栈如下所示。
java.io.IOException: com.google.api.client.googleapis.json.GoogleJsonResponseException: 410 Gone { "code" : 500, "errors" : [ { "domain" : "global", "message" : "Backend Error", "reason" : "backendError" } ], "message" : "Backend Error" }
at com.google.cloud.hadoop.util.AbstractGoogleAsyncWriteChannel.waitForCompletionAndThrowIfUploadFailed(AbstractGoogleAsyncWriteChannel.java:431)
at com.google.cloud.hadoop.util.AbstractGoogleAsyncWriteChannel.close(AbstractGoogleAsyncWriteChannel.java:289)
at com.google.cloud.dataflow.sdk.io.FileBasedSink$FileBasedWriter.close(FileBasedSink.java:516)
at com.google.cloud.dataflow.sdk.io.FileBasedSink$FileBasedWriter.close(FileBasedSink.java:419)
at com.google.cloud.dataflow.sdk.io.Write$Bound$2.finishBundle(Write.java:201) Caused by: com.google.api.client.googleapis.json.GoogleJsonResponseException: 410 Gone { "code" : 500, "errors" : [ { "domain" : "global", "message" : "Backend Error", "reason" : "backendError" } ], "message" : "Backend Error" }
at com.google.api.client.googleapis.json.GoogleJsonResponseException.from(GoogleJsonResponseException.java:146)
at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:113)
at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40)
at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:432)
at …Run Code Online (Sandbox Code Playgroud) 我正在尝试使用来自Eclipse的DataFlowRunner在Windows上使用MinimalWordCount运行MinimalWordCount示例 - >在eclipse中运行为Java应用程序,它使用我的gcs存储桶的示例中的相同股票代码,但是我一直得到以下异常,有人可以告诉我这里的问题吗?
线程"main"中的异常java.lang.RuntimeException:无法在org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod中从工厂方法DataflowRunner#fromOptions(interface org.apache.beam.sdk.options.PipelineOptions)构造实例(InstanceBuilder.java:233)org.apache.beam.sdk.util.InstanceBuilder.build(InstanceBuilder.java:162)org.apache.beam.sdk.runners.PipelineRunner.fromOptions(PipelineRunner.java:56)at at org.apache.beam.sdk.Pipeline.create(Pipeline.java:135)at com.dynaobject.apachebeamexperiment.MinimalWordCount.main(MinimalWordCount.java:77)引起:sun.reflect的java.lang.reflect.InvocationTargetException.位于org.apache的java.lang.reflect.Method.invoke(未知来源)的sun.reflect.DelegatingMethodAccessorImpl.invoke(未知来源)的sun.reflect.NativeMethodAccessorImpl.invoke(未知来源)的NativeMethodAccessorImpl.invoke0(本地方法). beam.sdk.util.InstanceBuilder.buildFromMethod(InstanceBuilder.java:222)... 4更多引起:java.lang.IllegalA rgumentException:DataflowRunner需要gcpTempLocation,但无法从org.apache.beam.runners.dataflow.DataflowRunner.fromOptions(DataflowRunner.java:212)中的PipelineOptions检索值... 9更多原因:java.lang.IllegalArgumentException:Error构造gcpTempLocation的默认值:tempLocation不是有效的GCS路径,gs:// tempxyz. org.apache.beam.sdk.options.GcpOptions $ GcpTempLocationFactory.create(GcpOptions.java:219)org.apache.beam.sdk.options.GcpOptions $ GcpTempLocationFactory.create(GcpOptions.java:205)at org.apache位于org.apache.beam.sdk.options.ProxyInvocationHandler的org.apache.beam.sdk.options.ProxyInvocationHandler.getDefault(ProxyInvocationHandler.java:516)中的.beam.sdk.options.ProxyInvocationHandler.returnDefaultHelper(ProxyInvocationHandler.java:575) .invoke(ProxyInvocationHandler.java:155)位于com.sun.proxy.$ Proxy15.getGcpTempLocation(未知来源)org.apache.beam.runners.dataflow.DataflowRunner.fromOptions(DataflowRunner.java:210)... 9更多 引发者:java.lang.IllegalArgumentException:输出路径不存在或不可写:gs:// tempxyz at org.apache.beam.sdk.repackaged.com.google.common.base.Preconditions.checkArgument(Preconditions.java) :191)位于org.apache.beam.sdk.util.GcsPathValidator的org.apache.beam.sdk.util.GcsPathValidator.verifyPathIsAccessible(GcsPathValidator.java:78).validateOutputFilePrefixSupported(GcsPathValidator.java:62)at org.apache.beam.sdk.options.GcpOptions $ GcpTempLocationFactory.create(GcpOptions.java:217)... 15更多
我正在尝试DirectRunner在Google Cloud Dataflow上运行能够成功运行的管道。当我执行此Maven命令时:
mvn compile exec:java \
-Dexec.mainClass=com.example.Pipeline \
-Dexec.args="--project=project-name \
--stagingLocation=gs://bucket-name/staging/ \
... custom arguments ...
--runner=DataflowRunner"
Run Code Online (Sandbox Code Playgroud)
我收到以下错误:
No Runner was specified and the DirectRunner was not found on the classpath.
[ERROR] Specify a runner by either:
[ERROR] Explicitly specifying a runner by providing the 'runner' property
[ERROR] Adding the DirectRunner to the classpath
[ERROR] Calling 'PipelineOptions.setRunner(PipelineRunner)' directly
Run Code Online (Sandbox Code Playgroud)
我有意DirectRunner从我的收藏夹中删除pom.xml并添加了以下内容:
<dependency>
<groupId>org.apache.beam</groupId>
<artifactId>beam-runners-google-cloud-dataflow-java</artifactId>
<version>2.0.0</version>
<scope>runtime</scope>
</dependency>
Run Code Online (Sandbox Code Playgroud)
我继续删除了<scope>标签,然后调用了options.setRunner(DataflowRunner.class),但没有帮助。PipelineOptions从扩展我自己的界面DataflowPipelineOptions …
我有一个Beam应用程序,可以使用directrunner在本地成功运行,并向我提供本地控制台中代码中包含的所有日志信息。但是,当我尝试在google cloud数据流环境中运行它时,我只在本地控制台上看到了这些日志信息,但是它们并没有显示在Google Cloud Console上的数据流作业中,也没有在其StackDriver日志记录页面中显示。
这是我从本地控制台为数据流运行程序运行代码的操作:
mvn compile exec:java -Dexec.mainClass= ... \
-Dexec.args=... " \
-Pdataflow-runner
Run Code Online (Sandbox Code Playgroud)
并且所有日志都将在此本地控制台上返回。但是,当我在浏览器上转到Google Cloud Console并搜索我的数据流作业日志时,在LOGGER.info(msg)任何地方的代码中都看不到这些日志。我只看到与数据流管道相关的日志。
因此,我想知道我的Beam应用程序是否以这样的方式单独运行,即不在管道内部的主类的一部分在本地运行,并且仅管道代码的一部分会发送到google cloud以便在那里执行。因此,不在管道代码中的那些日志信息将不会在Google Cloud Dataflow日志中提供。
带命令的气流安装失败
sudo pip3 install apache-airflow[gcp_api]
Run Code Online (Sandbox Code Playgroud)
昨天一切都很好.今天我看到以下错误:
Could not find a version that satisfies the requirement apache-beam[gcp]==2.3.0 (from google-cloud-dataflow->apache-airflow[gcp_api]) (from versions: 0.6.0, 2.0.0, 2.1.0, 2.1.1, 2.2.0)
No matching distribution found for apache-beam[gcp]==2.3.0 (from google-cloud-dataflow->apache-airflow[gcp_api])
Run Code Online (Sandbox Code Playgroud)
有人可以帮我吗?
提前致谢
google-cloud-platform google-cloud-dataflow airflow apache-beam
我正在使用FlinkRunner在本地运行以下Beam管道代码。PubsubIO用于读取主题中的消息。
我有一个单独的线程,该线程以固定的时间间隔(每30秒)将消息发布到主题,并且还设置了“ ts”属性,该属性稍后将用于导出事件时间。
自定义转换以转换为KV对-
private static class PubSubMessageGrouper extends DoFn<PubsubMessage, KV<String, PubsubMessage>> {
@ProcessElement
public void processElement(ProcessContext c) {
PubsubMessage element = c.element();
KV<String, PubsubMessage> kv = KV.of(element.getAttribute("key"), element);
c.output(kv);
}
}
Run Code Online (Sandbox Code Playgroud)
请注意,“密钥”是发布者线程中较早的消息属性中设置的密钥。目的是通过此键将消息下游分组。
管道代码-
PCollection<PubsubMessage> pubsubColl = p
.apply(PubsubIO.readMessagesWithAttributes()
.withTimestampAttribute("ts")
.fromTopic("projects/" + projectName + "/topics/beamtest")
);
PCollection<KV<String, PubsubMessage>> idfied =
pubsubColl.apply(ParDo.of(new PubSubMessageGrouper()));
PCollection<KV<String, PubsubMessage>> windowed = idfied
.apply(Window.<KV<String, PubsubMessage>>into(FixedWindows.of(Duration.standardSeconds(15)))
.triggering(
Repeatedly.forever(
AfterWatermark.pastEndOfWindow()
)
)
.withAllowedLateness(Duration.standardSeconds(15))
.discardingFiredPanes());
PCollection<KV<String, Iterable<PubsubMessage>>> grouped = windowed.apply(GroupByKey.create());
grouped.apply(ParDo.of(new KVPrinter()));
Run Code Online (Sandbox Code Playgroud)
转换没有链接以便于阅读。最后,KVPrinter转换只是打印出从group by接收到的消息,一旦我运行此消息,它将随后由实际代码替换。
当我运行此命令时,我没有在相当长的时间内(几分钟或更长时间)执行触发器。当它最终触发时,我看到某些消息没有收到(在最后一步中)。这是由于PubsubIO使用的内部水印吗?我的目的是确保所有消息都在groupby中进行处理,包括允许的延迟窗口中的延迟消息。
我看了相关的问题,但没有得到很多帮助。无论我等待多长时间,使用DirectRunner运行相同的东西都不会产生任何输出。还要注意的另一点是,如果删除管道的GroupBy部分,则会触发触发器。
在Beam Python DoFn中进行昂贵的一次性初始化的推荐方法是什么?Java SDK具有DoFn.Setup,但在Beam Python中似乎没有等效的东西。
当前是threading.local()在DoFn初始化程序中将对象附加到的最佳方法吗?