在查询执行错误期间,Google BigQuery超出了资源

Question

有关如何使此查询返回Google BigQuery结果的任何想法？我收到资源超出错误...数据集中有大约2B行.我正在尝试获取每个user_id最多的艺术家ID.

select user_id, artist, count(*) as count
from [legacy20130831.merged_data] as d
group each by user_id, artist
order by user_id ASC, count DESC

Answer 1

对公共数据的等效查询会引发相同的错误:

SELECT actor, repository_name, count(*) AS count
FROM [githubarchive:github.timeline] AS d
GROUP EACH BY actor, repository_name
ORDER BY actor, count desc

与同一查询比较,加上要返回的结果的限制.这个工作(我14秒):

SELECT actor, repository_name, count(*) as count
FROM [githubarchive:github.timeline] as d
GROUP EACH BY actor, repository_name
ORDER BY actor, count desc
LIMIT 100

您可以使用一小部分user_ids,而不是使用LIMIT.在我的情况下,1/3工作:

SELECT actor, repository_name, count(*) as count
FROM [githubarchive:github.timeline] as d
WHERE ABS(HASH(actor) % 3)  = 0
GROUP EACH BY actor, repository_name

但你真正想要的是"获得每个user_id最多的艺术家ID".让我们走得更远,并得到:

SELECT actor, repository_name, count FROM (
  SELECT actor, repository_name, count, ROW_NUMBER() OVER (PARTITION BY actor ORDER BY count DESC) rank FROM (
    SELECT actor, repository_name, count(*) as count
    FROM [githubarchive:github.timeline] as d
    WHERE ABS(HASH(actor) % 10) = 0
    GROUP EACH BY actor, repository_name
))
WHERE rank=1

请注意,这次我使用%10,因为它让我的结果更快.但你可能想知道"我希望用一个查询得到我的结果,而不是10".

你可以做两件事:

• 联合分区表(FROM表达式中的逗号执行联合,而不是BigQuery中的联接).
• 如果仍然超出资源,则可能需要实现表.运行原始查询并将结果保存到新表.在该表上运行RANK()算法,而不是在内存中的GROUP上运行.

如果您愿意与我分享您的数据集,我可以提供特定于数据集的建议(很大程度上取决于基数).