如何为单个表获取BigQuery存储大小

Question

我想计算Google Big Query Storage的表格明智成本,但我不知道如何分别查看每个表的存储大小.

Answer 1

或者从GUI中,您可以使用元数据内部表__TABLES__,例如,这将为您提供GB的大小:

select 
  sum(size_bytes)/pow(10,9) as size
from
  <your_dataset>.__TABLES__
where 
  table_id = '<your_table>'

Answer 2

有几种方法可以做到这一点,但要注意表格的大小(以字节为单位)属性对于主动接收流式插入的表不可用.

A.使用BQ命令行工具和JQ linux库来解析JSON.

bq --format=json show publicdata:samples.gsod | jq '.numBytes | tonumber'

这个出局:

17290009238

B.使用REST API执行表:获取调用

GET https://www.googleapis.com/bigquery/v2/projects/projectId/datasets/datasetId/tables/tableId

这将返回一个完整的JSON,您可以解析并获取numBytes.

{
    "kind": "bigquery#table",
    "description": "This dataset contains weather information collected by NOAA, such a…",
    "creationTime": "1335916040125",
    "tableReference": {
        "projectId": "publicdata",
        "tableId": "gsod",
        "datasetId": "samples"
    },
    "numRows": "114420316",
    "numBytes": "17290009238",
    "etag": "\"Gn3Hpo5WaKnpFuT457VBDNMgZBw/MTQxMzkzNzk4Nzg0Ng\"",
    "location": "US",
    "lastModifiedTime": "1413937987846",
    "type": "TABLE",
    "id": "publicdata:samples.gsod",
    "selfLink": "https://www.googleapis.com/bigquery/v2/projects/publicdata/datasets…",
    "schema": {
        "fields": [
            {
                "description": "The World Meteorological Organization (WMO) / DATSAV3 station numbe…",
                "type": "INTEGER",
                "name": "station_number",
                "mode": "REQUIRED"
            },
            {
                "description": "The Weather-Bureau-Army-Navy (WBAN) station number where the data w…",
                "type": "INTEGER",
                "name": "wban_number",
                "mode": "NULLABLE"
            },
            {
                "description": "The year the data was collected in",
                "type": "INTEGER",
                "name": "year",
                "mode": "REQUIRED"
            },
            {
                "description": "The month the data was collected in",
                "type": "INTEGER",
                "name": "month",
                "mode": "REQUIRED"
            },
            {
                "description": "The day the data was collected in.",
                "type": "INTEGER",
                "name": "day",
                "mode": "REQUIRED"
            },
            {
                "description": "The mean temperature of the day in degrees Fahrenheit, accurate to …",
                "type": "FLOAT",
                "name": "mean_temp",
                "mode": "NULLABLE"
            },
            {
                "description": "The number of observations used to calculate mean_temp.",
                "type": "INTEGER",
                "name": "num_mean_temp_samples",
                "mode": "NULLABLE"
            },
            {
                "description": "The mean dew point of the day in degrees Fahrenheit, accurate to on…",
                "type": "FLOAT",
                "name": "mean_dew_point",
                "mode": "NULLABLE"
            },
            {
                "description": "The number of observations used to calculate mean_dew_point.",
                "type": "INTEGER",
                "name": "num_mean_dew_point_samples",
                "mode": "NULLABLE"
            },
            {
                "description": "The mean sea level pressure of the day in millibars, accurate to on…",
                "type": "FLOAT",
                "name": "mean_sealevel_pressure",
                "mode": "NULLABLE"
            },
            {
                "description": "The number of observations used to calculate mean_sealevel_pressure…",
                "type": "INTEGER",
                "name": "num_mean_sealevel_pressure_samples",
                "mode": "NULLABLE"
            },
            {
                "description": "The mean station pressure of the day in millibars, accurate to one …",
                "type": "FLOAT",
                "name": "mean_station_pressure",
                "mode": "NULLABLE"
            },
            {
                "description": "The number of observations used to calculate mean_station_pressure.…",
                "type": "INTEGER",
                "name": "num_mean_station_pressure_samples",
                "mode": "NULLABLE"
            },
            {
                "description": "The mean visibility of the day in miles, accurate to one tenth of a…",
                "type": "FLOAT",
                "name": "mean_visibility",
                "mode": "NULLABLE"
            },
            {
                "description": "The number of observations used to calculate mean_visibility.",
                "type": "INTEGER",
                "name": "num_mean_visibility_samples",
                "mode": "NULLABLE"
            },
            {
                "description": "The mean wind speed of the day in knots, accurate to one tenth of a…",
                "type": "FLOAT",
                "name": "mean_wind_speed",
                "mode": "NULLABLE"
            },
            {
                "description": "The number of observations used to calculate mean_wind_speed.",
                "type": "INTEGER",
                "name": "num_mean_wind_speed_samples",
                "mode": "NULLABLE"
            },
            {
                "description": "The maximum sustained wind speed reported on the day in knots, accu…",
                "type": "FLOAT",
                "name": "max_sustained_wind_speed",
                "mode": "NULLABLE"
            },
            {
                "description": "The maximum wind gust speed reported on the day in knots, accurate …",
                "type": "FLOAT",
                "name": "max_gust_wind_speed",
                "mode": "NULLABLE"
            },
            {
                "description": "The maximum temperature of the day in degrees Fahrenheit, accurate …",
                "type": "FLOAT",
                "name": "max_temperature",
                "mode": "NULLABLE"
            },
            {
                "description": "Indicates the source of max_temperature.",
                "type": "BOOLEAN",
                "name": "max_temperature_explicit",
                "mode": "NULLABLE"
            },
            {
                "description": "The minimum temperature of the day in degrees Fahrenheit, accurate …",
                "type": "FLOAT",
                "name": "min_temperature",
                "mode": "NULLABLE"
            },
            {
                "description": "Indicates the source of min_temperature.",
                "type": "BOOLEAN",
                "name": "min_temperature_explicit",
                "mode": "NULLABLE"
            },
            {
                "description": "The total precipitation of the day in inches, accurate to one hundr…",
                "type": "FLOAT",
                "name": "total_precipitation",
                "mode": "NULLABLE"
            },
            {
                "description": "The snow depth of the day in inches, accurate to one tenth of an in…",
                "type": "FLOAT",
                "name": "snow_depth",
                "mode": "NULLABLE"
            },
            {
                "description": "Indicates if fog was reported on this day.",
                "type": "BOOLEAN",
                "name": "fog",
                "mode": "NULLABLE"
            },
            {
                "description": "Indicates if rain was reported on this day.",
                "type": "BOOLEAN",
                "name": "rain",
                "mode": "NULLABLE"
            },
            {
                "description": "Indicates if snow was reported on this day.",
                "type": "BOOLEAN",
                "name": "snow",
                "mode": "NULLABLE"
            },
            {
                "description": "Indicates if hail was reported on this day.",
                "type": "BOOLEAN",
                "name": "hail",
                "mode": "NULLABLE"
            },
            {
                "description": "Indicates if thunder was reported on this day.",
                "type": "BOOLEAN",
                "name": "thunder",
                "mode": "NULLABLE"
            },
            {
                "description": "Indicates if a tornado was reported on this day.",
                "type": "BOOLEAN",
                "name": "tornado",
                "mode": "NULLABLE"
            }
        ]
    }
}

C.有称为__TABLES__和的元词__TABLES_SUMMARY__

您可以运行如下查询:

SELECT size_bytes FROM <dataset>.__TABLES__ WHERE table_id='mytablename'

该__TABLES__查询的部分可能看起来不熟悉.__TABLES_SUMMARY__是一个元表,包含有关数据集中表的信息.您可以自己使用此元表.例如,查询SELECT * FROM publicdata:samples.__TABLES_SUMMARY__将返回有关publicdata:samples数据集中表的元数据.你也可以SELECT * FROM publicdata:samples.__TABLES__

可用字段:

__TABLES_SUMMARY__元表的字段(在TABLE_QUERY查询中都可用)包括:

• table_id:表的名称.
• creation_time:表格创建后的时间(自UTC时间1/1/1970起以毫秒为单位).这creation_time与表中的字段相同.
• type:无论是视图(2)还是常规表(1).

以下字段不可用,TABLE_QUERY()因为它们是成员__TABLES__但不是成员__TABLES_SUMMARY__.他们留在这里是为了历史的兴趣,并部分记录__TABLES__metatable:

• last_modified_time:自1970年1月1日UTC以来的时间(以毫秒为单位)表格已更新(元数据或表格内容).请注意,如果您使用tabledata.insertAll()流式传输记录到您的表,这可能是几分钟过时.
• row_count:表中的行数.
• size_bytes:表的总大小(以字节为单位).

Answer 3

您可以使用命令行工具执行此操作

bq show ds_name.table_name

它将在表格中显示一些信息,包括"总字节数".请参阅此处https://cloud.google.com/bigquery/bq-command-line-tool