使用.NET将1,000,000条记录更新(填充)到数据库中的最快方法

Question

我使用此代码将100万条记录插入数据库中的空表.好吧没有太多代码我将从我已经与数据交互的点开始,并将模式读入DataTable:

所以:

DataTable returnedDtViaLocalDbV11 = DtSqlLocalDb.GetDtViaConName(strConnName, queryStr, strReturnedDtName);

现在我们已经returnedDtViaLocalDbV11创建了一个new DataTable来成为源数据库表的克隆:

DataTable NewDtForBlkInsert = returnedDtViaLocalDbV11.Clone();

Stopwatch SwSqlMdfLocalDb11 = Stopwatch.StartNew();
NewDtForBlkInsert.BeginLoadData();

for (int i = 0; i < 1000000; i++)
{
   NewDtForBlkInsert.LoadDataRow(new object[] { null, "NewShipperCompanyName"+i.ToString(), "NewShipperPhone" }, false);
}
NewDtForBlkInsert.EndLoadData();

DBRCL_SET.UpdateDBWithNewDtUsingSQLBulkCopy(NewDtForBlkInsert, tblClients._TblName, strConnName);

SwSqlMdfLocalDb11.Stop();

var ResSqlMdfLocalDbv11_0 = SwSqlMdfLocalDb11.ElapsedMilliseconds;

此代码在5200ms内将 100万条记录填充到嵌入式SQL数据库(localDb)中.其余的代码只是实现了bulkCopy,但无论如何我都会发布它.

 public string UpdateDBWithNewDtUsingSQLBulkCopy(DataTable TheLocalDtToPush, string TheOnlineSQLTableName, string WebConfigConName)
 {
    //Open a connection to the database. 
    using (SqlConnection connection = new SqlConnection(ConfigurationManager.ConnectionStrings[WebConfigConName].ConnectionString))
    {
       connection.Open();

       // Perform an initial count on the destination table.
       SqlCommand commandRowCount = new SqlCommand("SELECT COUNT(*) FROM "+TheOnlineSQLTableName +";", connection);
       long countStart = System.Convert.ToInt32(commandRowCount.ExecuteScalar());

       var nl = "\r\n";
       string retStrReport = "";
       retStrReport = string.Concat(string.Format("Starting row count = {0}", countStart), nl);
       retStrReport += string.Concat("==================================================", nl);
       // Create a table with some rows. 
       //DataTable newCustomers = TheLocalDtToPush;

       // Create the SqlBulkCopy object.  
       // Note that the column positions in the source DataTable  
       // match the column positions in the destination table so  
       // there is no need to map columns.  
       using (SqlBulkCopy bulkCopy = new SqlBulkCopy(connection))
       {
          bulkCopy.DestinationTableName = TheOnlineSQLTableName;

          try
          {
             // Write from the source to the destination.
             for (int colIndex = 0; colIndex < TheLocalDtToPush.Columns.Count; colIndex++)
             {
                bulkCopy.ColumnMappings.Add(colIndex, colIndex);
             }
             bulkCopy.WriteToServer(TheLocalDtToPush);
          }

          catch (Exception ex)
          {
             Console.WriteLine(ex.Message);
          }
       }

       // Perform a final count on the destination  
       // table to see how many rows were added. 
       long countEnd = System.Convert.ToInt32(
       commandRowCount.ExecuteScalar());

       retStrReport += string.Concat("Ending row count = ", countEnd, nl);
       retStrReport += string.Concat("==================================================", nl);
       retStrReport += string.Concat((countEnd - countStart)," rows were added.", nl);
       retStrReport += string.Concat("New Customers Was updated successfully", nl, "END OF PROCESS !");
       //Console.ReadLine();
       return retStrReport;
   }
}

通过与SQL服务器的连接尝试它大约7000毫秒(充其量)和平均约7700毫秒.另外通过一个随机的kv nosql数据库花了大约40秒(实际上我甚至没有记录它,因为它通过x2的sql变种).那么...有没有比我在代码中测试更快的方式？

编辑

我正在使用win7 x64 8gb ram,最重要的是我应该认为(因为i5 3ghz)不是那么好,现在raid-0上的x3 500Gb Wd工作做得更好但我只是说如果你会检查你的电脑虽然只需将其与配置中的任何其他方法进行比较即可

Answer 1

你尝试过SSIS吗？我从未编写过带有 loacldb 连接的 SSIS 包，但 SSIS 应该非常适合这种活动。

如果您的数据源是 SQL Server，另一个想法是设置链接服务器。不确定这是否适用于 localdb。如果您可以设置链接服务器，则可以完全绕过 C# 并使用 INSERT .. SELECT ... FROM ... SQL 语句加载数据。