如何删除重复的行？

Question

从相当大的SQL Server表中删除重复行的最佳方法是什么(即300,000多行)？

当然,由于RowID身份字段的存在,行不会是完美的重复.

MyTable的

RowID int not null identity(1,1) primary key,
Col1 varchar(20) not null,
Col2 varchar(2048) not null,
Col3 tinyint not null

Answer 1

假设没有空,你GROUP BY的唯一列,并SELECT在MIN (or MAX)RowId的为行,以保持.然后,只删除没有行id的所有内容:

DELETE FROM MyTable
LEFT OUTER JOIN (
   SELECT MIN(RowId) as RowId, Col1, Col2, Col3 
   FROM MyTable 
   GROUP BY Col1, Col2, Col3
) as KeepRows ON
   MyTable.RowId = KeepRows.RowId
WHERE
   KeepRows.RowId IS NULL

如果您有GUID而不是整数,则可以替换

MIN(RowId)

同

CONVERT(uniqueidentifier, MIN(CONVERT(char(36), MyGuidColumn)))

Answer 2

另一种可行的方法是

; 

--Ensure that any immediately preceding statement is terminated with a semicolon above
WITH cte
     AS (SELECT ROW_NUMBER() OVER (PARTITION BY Col1, Col2, Col3 
                                       ORDER BY ( SELECT 0)) RN
         FROM   #MyTable)
DELETE FROM cte
WHERE  RN > 1;

我在ORDER BY (SELECT 0)上面使用,因为它是任意哪一行在出现平局时保留.

例如,为了保留最新的一个,RowID你可以使用ORDER BY RowID DESC

执行计划

对于此,执行计划通常比接受的答案更简单,更有效,因为它不需要自联接.

但情况并非总是如此.GROUP BY可能首选解决方案的地方是优先选择散列聚合而不是流聚合的情况.

的ROW_NUMBER,而该解决方案将永远给几乎相同的计划GROUP BY策略更加灵活.

可能有利于散列聚合方法的因素是

• 分区列上没有有用的索引
• 相对较少的群体,每组中具有相对较多的重复

在第二种情况的极端版本​​中(如果每个组中都有很多重复的组很少),也可以考虑简单地插入行以保存到新表中然后TRUNCATE复制它们并将其复制回以最小化日志记录与删除行的比例非常高.

Answer 3

有一篇关于删除 Microsoft支持站点上的重复项的文章.这是相当保守的 - 他们让你在不同的步骤中做所有事情 - 但它应该适用于大表.

我过去曾经使用过自联接来做这个,虽然它可能会被HAVING子句搞定:

DELETE dupes
FROM MyTable dupes, MyTable fullTable
WHERE dupes.dupField = fullTable.dupField 
AND dupes.secondDupField = fullTable.secondDupField 
AND dupes.uniqueField > fullTable.uniqueField

Answer 4

以下查询对于删除重复行很有用.本例中的表具有ID作为标识列和具有重复数据的列是Column1,Column2并且Column3.

DELETE FROM TableName
WHERE  ID NOT IN (SELECT MAX(ID)
                  FROM   TableName
                  GROUP  BY Column1,
                            Column2,
                            Column3
                  /*Even if ID is not null-able SQL Server treats MAX(ID) as potentially
                    nullable. Because of semantics of NOT IN (NULL) including the clause
                    below can simplify the plan*/
                  HAVING MAX(ID) IS NOT NULL) 

下面的脚本说明用法GROUP BY,HAVING,ORDER BY在一个查询中,并返回带有重复列,其计结果.

SELECT YourColumnName,
       COUNT(*) TotalCount
FROM   YourTableName
GROUP  BY YourColumnName
HAVING COUNT(*) > 1
ORDER  BY COUNT(*) DESC 

Answer 5

delete t1
from table t1, table t2
where t1.columnA = t2.columnA
and t1.rowid>t2.rowid

Postgres的:

delete
from table t1
using table t2
where t1.columnA = t2.columnA
and t1.rowid > t2.rowid

Answer 6

DELETE LU 
FROM   (SELECT *, 
               Row_number() 
                 OVER ( 
                   partition BY col1, col1, col3 
                   ORDER BY rowid DESC) [Row] 
        FROM   mytable) LU 
WHERE  [row] > 1 

Answer 7

这将删除第一行以外的重复行

DELETE
FROM
    Mytable
WHERE
    RowID NOT IN (
        SELECT
            MIN(RowID)
        FROM
            Mytable
        GROUP BY
            Col1,
            Col2,
            Col3
    )

参考(http://www.codeproject.com/Articles/157977/Remove-Duplicate-Rows-from-a-Table-in-SQL-Server)

Answer 8

我更喜欢CTE从sql server表中删除重复的行

强烈建议遵循这篇文章:: http://codaffection.com/sql-server-article/delete-duplicate-rows-in-sql-server/

保持原创

WITH CTE AS
(
SELECT *,ROW_NUMBER() OVER (PARTITION BY col1,col2,col3 ORDER BY col1,col2,col3) AS RN
FROM MyTable
)

DELETE FROM CTE WHERE RN<>1

没有保持原创

WITH CTE AS
(SELECT *,R=RANK() OVER (ORDER BY col1,col2,col3)
FROM MyTable)
 
DELETE CTE
WHERE R IN (SELECT R FROM CTE GROUP BY R HAVING COUNT(*)>1)

Answer 9

获取重复行:

SELECT
name, email, COUNT(*)
FROM 
users
GROUP BY
name, email
HAVING COUNT(*) > 1

删除重复行:

DELETE users 
WHERE rowid NOT IN 
(SELECT MIN(rowid)
FROM users
GROUP BY name, email);      

Answer 10

快速和脏删除精确重复的行(对于小表):

select  distinct * into t2 from t1;
delete from t1;
insert into t1 select *  from t2;
drop table t2;

Answer 11

我更喜欢子查询\具有count(*)> 1的内部连接解决方​​案,因为我发现它更容易阅读,并且很容易变成SELECT语句来验证在运行之前将删除的内容.

--DELETE FROM table1 
--WHERE id IN ( 
     SELECT MIN(id) FROM table1 
     GROUP BY col1, col2, col3 
     -- could add a WHERE clause here to further filter
     HAVING count(*) > 1
--)

Answer 12

SELECT  DISTINCT *
      INTO tempdb.dbo.tmpTable
FROM myTable

TRUNCATE TABLE myTable
INSERT INTO myTable SELECT * FROM tempdb.dbo.tmpTable
DROP TABLE tempdb.dbo.tmpTable

Answer 13

我想我会分享我的解决方案,因为它在特殊情况下有效.我的情况下,具有重复值的表没有外键(因为这些值是从另一个数据库复制的).

begin transaction
-- create temp table with identical structure as source table
Select * Into #temp From tableName Where 1 = 2

-- insert distinct values into temp
insert into #temp 
select distinct * 
from  tableName

-- delete from source
delete from tableName 

-- insert into source from temp
insert into tableName 
select * 
from #temp

rollback transaction
-- if this works, change rollback to commit and execute again to keep you changes!!

PS:在处理这样的事情时,我总是使用一个事务,这不仅可以确保所有内容都作为一个整体执行,而且还允许我在不冒任何风险的情况下进行测试.但是当然你应该备份,以确保......

Answer 14

使用CTE.我们的想法是加入一个或多个形成重复记录的列,然后删除您喜欢的任何一个:

;with cte as (
    select 
        min(PrimaryKey) as PrimaryKey
        UniqueColumn1,
        UniqueColumn2
    from dbo.DuplicatesTable 
    group by
        UniqueColumn1, UniqueColumn1
    having count(*) > 1
)
delete d
from dbo.DuplicatesTable d 
inner join cte on 
    d.PrimaryKey > cte.PrimaryKey and
    d.UniqueColumn1 = cte.UniqueColumn1 and 
    d.UniqueColumn2 = cte.UniqueColumn2;

Answer 15

这个查询对我来说表现非常好:

DELETE tbl
FROM
    MyTable tbl
WHERE
    EXISTS (
        SELECT
            *
        FROM
            MyTable tbl2
        WHERE
            tbl2.SameValue = tbl.SameValue
        AND tbl.IdUniqueValue < tbl2.IdUniqueValue
    )

它从2M的表中删除了超过30秒的1M行(50%重复)

Answer 16

另外一个简单的解决方案可以在这里粘贴的链接找到.这个容易掌握,似乎对大多数类似的问题都有效.它适用于SQL Server,但使用的概念是可接受的.

以下是链接页面的相关部分:

考虑这些数据:

EMPLOYEE_ID ATTENDANCE_DATE
A001    2011-01-01
A001    2011-01-01
A002    2011-01-01
A002    2011-01-01
A002    2011-01-01
A003    2011-01-01

那么我们如何删除那些重复数据呢？

首先,使用以下代码在该表中插入标识列:

ALTER TABLE dbo.ATTENDANCE ADD AUTOID INT IDENTITY(1,1)  

使用以下代码解决它:

DELETE FROM dbo.ATTENDANCE WHERE AUTOID NOT IN (SELECT MIN(AUTOID) _
    FROM dbo.ATTENDANCE GROUP BY EMPLOYEE_ID,ATTENDANCE_DATE) 

Answer 17

这是关于删除重复项的另一篇好文章.

它讨论了为什么它很难:" SQL基于关系代数,并且重复在关系代数中不会发生,因为在集合中不允许重复. "

临时表解决方案,以及两个mysql示例.

将来您是要在数据库级别还是从应用程序的角度来防止它.我建议数据库级别,因为您的数据库应该负责维护参照完整性,开发人员只会导致问题;)

Answer 18

行,可以.使用临时表.如果你想要一个"工作"的单一,不是非常高效的陈述,你可以使用:

DELETE FROM MyTable WHERE NOT RowID IN
    (SELECT 
        (SELECT TOP 1 RowID FROM MyTable mt2 
        WHERE mt2.Col1 = mt.Col1 
        AND mt2.Col2 = mt.Col2 
        AND mt2.Col3 = mt.Col3) 
    FROM MyTable mt)

基本上,对于表中的每一行,子选择查找与所考虑的行完全相同的所有行的顶部RowID.因此,您最终会得到一个表示"原始"非重复行的RowID列表.

Answer 19

我有一个表,我需要保留不重复的行.我不确定速度或效率.

DELETE FROM myTable WHERE RowID IN (
  SELECT MIN(RowID) AS IDNo FROM myTable
  GROUP BY Col1, Col2, Col3
  HAVING COUNT(*) = 2 )

Answer 20

用这个

WITH tblTemp as
(
SELECT ROW_NUMBER() Over(PARTITION BY Name,Department ORDER BY Name)
   As RowNumber,* FROM <table_name>
)
DELETE FROM tblTemp where RowNumber >1

Answer 21

另一种方法是创建一个具有相同字段和唯一索引的新表.然后将所有数据从旧表移动到新表.自动SQL SERVER忽略(如果存在重复值,还有一个关于该怎么做的选项:ignore,interrupt或sth)重复值.所以我们有相同的表没有重复的行.如果您不想要唯一索引,则可以在传输数据之后删除它.

特别是对于较大的表,您可以使用DTS(SSIS包来导入/导出数据),以便将所有数据快速传输到新的唯一索引表.对于700万行,它只需要几分钟.

Answer 22

1. 创建具有相同结构的新空白表

2. 像这样执行查询

   INSERT INTO tc_category1
   SELECT *
   FROM tc_category
   GROUP BY category_id, application_id
   HAVING count(*) > 1
   

3. 然后执行此查询

   INSERT INTO tc_category1
   SELECT *
   FROM tc_category
   GROUP BY category_id, application_id
   HAVING count(*) = 1
   

Answer 23

通过使用以下查询,我们可以根据单列或多列删除重复记录.以下查询是基于两列删除.表名是:testing和列名empno,empname

DELETE FROM testing WHERE empno not IN (SELECT empno FROM (SELECT empno, ROW_NUMBER() OVER (PARTITION BY empno ORDER BY empno) 
AS [ItemNumber] FROM testing) a WHERE ItemNumber > 1)
or empname not in
(select empname from (select empname,row_number() over(PARTITION BY empno ORDER BY empno) 
AS [ItemNumber] FROM testing) a WHERE ItemNumber > 1)

Answer 24

这是删除重复记录的最简单方法

 DELETE FROM tblemp WHERE id IN 
 (
  SELECT MIN(id) FROM tblemp
   GROUP BY  title HAVING COUNT(id)>1
 )

http://askme.indianyouth.info/details/how-to-dumplicate-record-from-table-in-using-sql-105

Answer 25

从应用程序级别(不幸的).我同意防止重复的正确方法是在数据库级别通过使用唯一索引,但在SQL Server 2005中,索引只允许900字节,我的varchar(2048)字段将其吹走.

我不知道它的表现如何,但我认为你可以编写一个触发器来强制执行此操作,即使你不能直接使用索引.就像是:

-- given a table stories(story_id int not null primary key, story varchar(max) not null)
CREATE TRIGGER prevent_plagiarism 
ON stories 
after INSERT, UPDATE 
AS 
    DECLARE @cnt AS INT 

    SELECT @cnt = Count(*) 
    FROM   stories 
           INNER JOIN inserted 
                   ON ( stories.story = inserted.story 
                        AND stories.story_id != inserted.story_id ) 

    IF @cnt > 0 
      BEGIN 
          RAISERROR('plagiarism detected',16,1) 

          ROLLBACK TRANSACTION 
      END 

另外,varchar(2048)对我来说听起来很腥(生活中有些东西是2048字节,但这种情况非常罕见); 它真的不是varchar(max)吗？

Answer 26

我会提到这种方法,它可能会有所帮助,适用于所有SQL服务器:通常只有一个 - 两个重复,并且Ids和重复计数是已知的.在这种情况下:

SET ROWCOUNT 1 -- or set to number of rows to be deleted
delete from myTable where RowId = DuplicatedID
SET ROWCOUNT 0

Answer 27

DELETE
FROM
    table_name T1
WHERE
    rowid > (
        SELECT
            min(rowid)
        FROM
            table_name T2
        WHERE
            T1.column_name = T2.column_name
    );

Answer 28

CREATE TABLE car(Id int identity(1,1), PersonId int, CarId int)

INSERT INTO car(PersonId,CarId)
VALUES(1,2),(1,3),(1,2),(2,4)

--SELECT * FROM car

;WITH CTE as(
SELECT ROW_NUMBER() over (PARTITION BY personid,carid order by personid,carid) as rn,Id,PersonID,CarId from car)

DELETE FROM car where Id in(SELECT Id FROM CTE WHERE rn>1)

Answer 29

DELETE 
FROM MyTable
WHERE NOT EXISTS (
              SELECT min(RowID)
              FROM Mytable
              WHERE (SELECT RowID 
                     FROM Mytable
                     GROUP BY Col1, Col2, Col3
                     ))
               );

Answer 30

我想要预览要删除的行,并控制要保留的重复行.见http://developer.azurewebsites.net/2014/09/better-sql-group-by-find-duplicate-data/

with MYCTE as (
  SELECT ROW_NUMBER() OVER (
    PARTITION BY DuplicateKey1
                ,DuplicateKey2 -- optional
    ORDER BY CreatedAt -- the first row among duplicates will be kept, other rows will be removed
  ) RN
  FROM MyTable
)
DELETE FROM MYCTE
WHERE RN > 1

Answer 31

另一种方法: -

DELETE A
FROM   TABLE A,
       TABLE B
WHERE  A.COL1 = B.COL1
       AND A.COL2 = B.COL2
       AND A.UNIQUEFIELD > B.UNIQUEFIELD