BigQuery - DELETE语句删除重复项

Question

SQL上有很多很棒的帖子可以选择唯一的行并写入(截断)表格,以便删除dus.例如

WITH ev AS (
  SELECT
    *,
    ROW_NUMBER() OVER (PARTITION BY id ORDER BY loadTime DESC) AS rowNum
  FROM `duplicates`
 )
SELECT
  * EXCEPT(rowNum)
FROM
  ev
WHERE rowNum = 1

我试图使用DML和DELETE稍微改变一下(例如,如果你不想使用BQ savedQuery,只需执行SQL).我想做的大致是:

WITH dup_events AS (
  SELECT
        *,
        ROW_NUMBER() OVER (PARTITION BY id ORDER BY loadTime DESC) AS rowNum
      FROM `duplicates`
 )
DELETE FROM
  dup_events
WHERE rowNum > 1

但在控制台中出现此错误:

Syntax error: Expected "(" or keyword SELECT but got keyword DELETE at [10:1]

可以使用DELETE实现(standardSQL)吗？

谢谢!

Answer 1

实际下面:o)工作

#standardSQL
DELETE FROM `yourproject.yourdataset.duplicates`
WHERE STRUCT(id, loadTime) NOT IN (
        SELECT AS STRUCT id, MAX(loadTime) loadTime 
        FROM `yourproject.yourdataset.duplicates` 
        GROUP BY id)  

注意:它假设loadTime也是唯一的 - 这意味着如果对于给定的id,有多个记录包含最新的loadTime - 它们都将被保留

Answer 2

从语法文档来看，to 的参数DELETE需要是一个表，并且没有规定使用WITH子句。鉴于您无法从本质上是逻辑视图（CTE）中删除，这是有道理的。您可以通过将逻辑放在过滤器中来表达您想要的内容，例如

DELETE
FROM duplicates AS d
WHERE (SELECT ROW_NUMBER() OVER (PARTITION BY id ORDER BY loadTime DESC)
       FROM `duplicates` AS d2
       WHERE d.id = d2.id AND d.loadTime = d2.loadTime) > 1;

Answer 3

这必须是最简单的方法：

create or replace table `myproject.mydataset.duplicates` as (
select distinct *
from `myproject.mydataset.duplicates`)

如果您有数组数据类型，请尝试以下操作：

-- build a test table with a duplicate and an array datatype column --
create or replace table DW.pmoTest as (
select 1 as ID, 'peter' as firstname,ARRAY<INT64>[1, 2, 3]  as int_array, current_date as createdate
union all
select 1 as ID, 'peter' as firstname,ARRAY<INT64>[1, 7, 3] as int_array, current_date as createdate
union all
select 2 as ID, 'chamri' as firstname,ARRAY<INT64>[1, 2, 39, 4] as int_array, current_date as createdate
);

-- recreate table without duplicate row
create or replace table DW.pmoTest as (
SELECT col.* FROM (
  SELECT ARRAY_AGG(tbl ORDER BY createdate LIMIT 1)[OFFSET(0)]  col
  FROM DW.pmoTest tbl
  GROUP BY ID
  )
);