如何一次性将40亿条记录从MySQL加载到SQL Server

Question

We have a need to do the initial data copy on a table that has 4+ billion records to target SQL Server (2014) from source MySQL (5.5). The table in question is pretty wide with 55 columns, however none of them are LOB. I'm looking for options for copying this data in the most efficient way possible.

We've tried loading via Attunity Replicate (which has worked wonderfully for tables not this large) but if the initial data copy with Attunity Replicate fails then it starts over from scratch ... losing whatever time was spent copying the data. With patching and the possibility of this table taking 3+ months to load Attunity wasn't the solution.

We've also tried smaller batch loads with a linked server. This is working but doesn't seem efficient at all.

Once the data is copied we will be using Attunity Replicate to handle CDC.

Answer 1

对于这样的事情，我认为 SSIS 是最简单的。它专为高达 1TB 的大型插入而设计。事实上，我推荐这篇 MSDN 文章“我们在 30 分钟内加载了 1TB，您也可以”。

做一些简单的事情（例如删除索引）和执行其他优化（例如分区）将使您的加载速度更快。虽然 30 分钟不是一个可行的时间，但在工作时间之外运行 SSIS 包将是一项非常简单的任务。

我的业务没有您那样的负载规模，但我们每晚都会刷新超过 100M 的数据库，即使优化得很差，也不会超过 45 分钟。