查询执行期间超出资源

Question

我正在尝试运行一个连接2个大型数据集的查询,并且我在查询执行错误期间遇到了超出的资源.我已经读过,使用Join Each和Group Each时会有解决方法但不是那些解决方法.

SELECT 
  year(users.firstseen) as first_year,
  month(users.firstseen) as first_month, 
  DATEDIFF(orders.timestamp,users.firstseen) as days_elapsed,
  count(orders.user_key) as count_orders
FROM 
  [project.orders] as orders
JOIN EACH
  [project.users] AS users
ON
  orders.user_key = users.user_key
WHERE orders.store = 'ios'
GROUP EACH BY 1,2,3

编辑:以下工作:

SELECT
  year(users.firstseen) as firstyear,
  month(users.firstseen) as firstmonth,
  DATEDIFF(orders.timestamp, users.firstseen) as days_elapsed,
  COUNT(users.firstseen) AS count_orders FROM [project.orders] as orders
JOIN EACH( SELECT user_key, firstseen FROM [project.users]
WHERE store_key = 'ios') as users ON orders.user_key = users.user_key
GROUP BY firstyear, firstmonth, days_elapsed
ORDER BY firstyear, firstmonth, days_elapsed

Answer 1

如果您的连接键(在本例中为user_key)分布不均匀,则JOIN EACH可能会失败.例如,如果您有一个经常出现异常的user_key,则会从处理该密钥的节点获得"资源超出"错误.或者,您可以尝试通过在连接之前过滤掉部分用户密钥来在较小的用户密钥集上运行查询.

如果您有太多不同的组密钥,GROUP EACH BY可能会失败.您可以尝试通过添加一些WHERE子句来减少连接输出,以查看是否是这种情况.

基本上,我建议将输入缩小到JOIN EACH或GROUP EACH BY,直到你的查询生效为止,然后你就会更好地理解你遇到的限制.一旦你知道了,你就可以(希望)构建你的查询以充分利用可用资源.

(顺便说一句,我们希望在不久的将来调整这些操作,以消除您可能遇到的一些限制!)