Firebase 数据传输至 Google BigQuery

Question

Firebase 在 Google Cloud Storage 上提供私人备份。特色用例之一是“摄入分析产品”：

\n\n

Private Backups provides a perfect pipeline into cloud analytics products such as Google\xe2\x80\x99s BigQuery. Cloud Analytics products often prefer to ingest data through Cloud Storage buckets rather than directly from the application.\n

\n\n

我在 Firebase 中有大量数据（导出到 Cloud Storage 存储桶时超过 1GB），并且如 Firebase 产品中所述，我想将这些数据放入 Big Query 中。

\n\n

但真的有可能编写适合 Firebase 原始数据的表架构吗？\n让我们以 Firebase 文档中的恐龙事实数据库为例。\nJSON 如下所示：

\n\n\n\n

Private Backups provides a perfect pipeline into cloud analytics products such as Google\xe2\x80\x99s BigQuery. Cloud Analytics products often prefer to ingest data through Cloud Storage buckets rather than directly from the application.\n

\n\n\n\n

要列出所有恐龙，我想唯一的方法是在 bigQuery 架构中使用 RECORD 字段。但通常 BigQuery 中的 RECORDS 对应于导入的 JSON 中的数组。Firebase 中没有数组，只有一个以恐龙名称作为键名称的对象。

\n\n

因此，像这样的 BigQuery 表架构不起作用：\n

\n\n

[\n    {\n        "name": "dinosaurs",\n        "type": "RECORD",\n        "mode": "REQUIRED",\n        "fields": [\n            {\n                "name": "dinosaur",\n                "type": "RECORD",\n                "mode": "REPEATED",\n                "fields": [\n                    {\n                        "name": "appeared",\n                        "type": "INTEGER"\n                    },\n                    {\n                        "name": "height",\n                        "type": "INTEGER"\n                    },\n                    {\n                        "name": "length",\n                        "type": "INTEGER"\n                    },\n                    {\n                        "name": "order",\n                        "type": "STRING"\n                    },\n                    {\n                        "name": "vanished",\n                        "type": "INTEGER"\n                    },\n                    {\n                        "name": "weight",\n                        "type": "INTEGER"\n                    }\n                ]\n            },\n            {\n                "name": "scores",\n                "type": "RECORD",\n                "mode": "REPEATED",\n                "fields": [\n                    {\n                        "name": "dinosaur",\n                        "type": "INTEGER"\n                    }\n                ]\n            }\n        ]\n    }\n]\n

\n\n\n\n

是否可以编写适合 Firebase 原始数据的表架构？或者我们应该首先准备数据以使其与 BigQuery 兼容？

\n

Answer 1

由于上面的数据只是 JSON，因此您应该能够使其与 Firebase 一起使用。不过，我认为备份后准备数据会容易得多。

您提到 Firebase 数据中没有数组。Firebase 确实支持数组，但它们必须满足一定的标准。

// we send this
['a', 'b', 'c', 'd', 'e']
// Firebase stores this
{0: 'a', 1: 'b', 2: 'c', 3: 'd', 4: 'e'}
// since the keys are numeric and sequential,
// if we query the data, we get this
['a', 'b', 'c', 'd', 'e']

尽管它可能看起来像 Firebase 数据库中的对象，但在查询时它会作为数组返回。

因此，在 Firebase 数据库中创建架构是可行的，但它可能会给您的应用程序带来大量开销。