Vah*_*iri 110 entity entity-framework
我想按实体框架在表中插入20000条记录,大约需要2分钟.除了使用SP来改善其性能之外,还有什么方法.这是我的代码:
foreach (Employees item in sequence)
{
t = new Employees ();
t.Text = item.Text;
dataContext.Employees.AddObject(t);
}
dataContext.SaveChanges();
Rom*_*ias 208
有几次改进的机会(如果您正在使用DbContext):
组:
yourContext.Configuration.AutoDetectChangesEnabled = false;
yourContext.Configuration.ValidateOnSaveEnabled = false;
做SaveChanges()100个插入的包......或者你可以用的1000个项目包试试,看看性能的变化.
因为在所有这些插入期间,上下文是相同的并且它变得越来越大,您可以每1000次插入重建上下文对象. var yourContext = new YourContext();我认为这是一大收获.
在我的导入数据进程中进行此改进,从7分钟到6秒.
在您的情况下,实际数字...不能是100或1000 ...尝试并调整它.
Lad*_*nka 41
以这种方式执行时,无法强制EF提高性能.问题是EF在单独的往返数据库中执行每个插入.太棒了不是吗？甚至DataSet都支持批处理.查看本文以获取一些解决方法.另一种解决方法是使用自定义存储过程接受表值参数,但您需要原始ADO.NET.
Mån*_*ryd 25
使用下面的代码,您可以使用一个方法扩展部分上下文类,该方法将获取实体对象的集合并将它们批量复制到数据库.只需将类的名称从MyEntities替换为您的实体类的名称,并将其添加到项目的正确名称空间中.之后,您需要做的就是调用BulkInsertAll方法移交要插入的实体对象.不要重复使用上下文类,而是每次使用它时创建一个新实例.这是必需的,至少在某些版本的EF中是必需的,因为与此处使用的SQLConnection相关联的认证数据在使用该类一次后会丢失.我不知道为什么.
此版本适用于EF 5
public partial class MyEntities
{
public void BulkInsertAll<T>(T[] entities) where T : class
{
var conn = (SqlConnection)Database.Connection;
conn.Open();
Type t = typeof(T);
Set(t).ToString();
var objectContext = ((IObjectContextAdapter)this).ObjectContext;
var workspace = objectContext.MetadataWorkspace;
var mappings = GetMappings(workspace, objectContext.DefaultContainerName, typeof(T).Name);
var tableName = GetTableName<T>();
var bulkCopy = new SqlBulkCopy(conn) { DestinationTableName = tableName };
// Foreign key relations show up as virtual declared
// properties and we want to ignore these.
var properties = t.GetProperties().Where(p => !p.GetGetMethod().IsVirtual).ToArray();
var table = new DataTable();
foreach (var property in properties)
{
Type propertyType = property.PropertyType;
// Nullable properties need special treatment.
if (propertyType.IsGenericType &&
propertyType.GetGenericTypeDefinition() == typeof(Nullable<>))
{
propertyType = Nullable.GetUnderlyingType(propertyType);
}
// Since we cannot trust the CLR type properties to be in the same order as
// the table columns we use the SqlBulkCopy column mappings.
table.Columns.Add(new DataColumn(property.Name, propertyType));
var clrPropertyName = property.Name;
var tableColumnName = mappings[property.Name];
bulkCopy.ColumnMappings.Add(new SqlBulkCopyColumnMapping(clrPropertyName, tableColumnName));
}
// Add all our entities to our data table
foreach (var entity in entities)
{
var e = entity;
table.Rows.Add(properties.Select(property => GetPropertyValue(property.GetValue(e, null))).ToArray());
}
// send it to the server for bulk execution
bulkCopy.BulkCopyTimeout = 5 * 60;
bulkCopy.WriteToServer(table);
conn.Close();
}
private string GetTableName<T>() where T : class
{
var dbSet = Set<T>();
var sql = dbSet.ToString();
var regex = new Regex(@"FROM (?<table>.*) AS");
var match = regex.Match(sql);
return match.Groups["table"].Value;
}
private object GetPropertyValue(object o)
{
if (o == null)
return DBNull.Value;
return o;
}
private Dictionary<string, string> GetMappings(MetadataWorkspace workspace, string containerName, string entityName)
{
var mappings = new Dictionary<string, string>();
var storageMapping = workspace.GetItem<GlobalItem>(containerName, DataSpace.CSSpace);
dynamic entitySetMaps = storageMapping.GetType().InvokeMember(
"EntitySetMaps",
BindingFlags.GetProperty | BindingFlags.NonPublic | BindingFlags.Instance,
null, storageMapping, null);
foreach (var entitySetMap in entitySetMaps)
{
var typeMappings = GetArrayList("TypeMappings", entitySetMap);
dynamic typeMapping = typeMappings[0];
dynamic types = GetArrayList("Types", typeMapping);
if (types[0].Name == entityName)
{
var fragments = GetArrayList("MappingFragments", typeMapping);
var fragment = fragments[0];
var properties = GetArrayList("AllProperties", fragment);
foreach (var property in properties)
{
var edmProperty = GetProperty("EdmProperty", property);
var columnProperty = GetProperty("ColumnProperty", property);
mappings.Add(edmProperty.Name, columnProperty.Name);
}
}
}
return mappings;
}
private ArrayList GetArrayList(string property, object instance)
{
var type = instance.GetType();
var objects = (IEnumerable)type.InvokeMember(property, BindingFlags.GetProperty | BindingFlags.NonPublic | BindingFlags.Instance, null, instance, null);
var list = new ArrayList();
foreach (var o in objects)
{
list.Add(o);
}
return list;
}
private dynamic GetProperty(string property, object instance)
{
var type = instance.GetType();
return type.InvokeMember(property, BindingFlags.GetProperty | BindingFlags.NonPublic | BindingFlags.Instance, null, instance, null);
}
}
此版本适用于EF 6
public partial class CMLocalEntities
{
public void BulkInsertAll<T>(T[] entities) where T : class
{
var conn = (SqlConnection)Database.Connection;
conn.Open();
Type t = typeof(T);
Set(t).ToString();
var objectContext = ((IObjectContextAdapter)this).ObjectContext;
var workspace = objectContext.MetadataWorkspace;
var mappings = GetMappings(workspace, objectContext.DefaultContainerName, typeof(T).Name);
var tableName = GetTableName<T>();
var bulkCopy = new SqlBulkCopy(conn) { DestinationTableName = tableName };
// Foreign key relations show up as virtual declared
// properties and we want to ignore these.
var properties = t.GetProperties().Where(p => !p.GetGetMethod().IsVirtual).ToArray();
var table = new DataTable();
foreach (var property in properties)
{
Type propertyType = property.PropertyType;
// Nullable properties need special treatment.
if (propertyType.IsGenericType &&
propertyType.GetGenericTypeDefinition() == typeof(Nullable<>))
{
propertyType = Nullable.GetUnderlyingType(propertyType);
}
// Since we cannot trust the CLR type properties to be in the same order as
// the table columns we use the SqlBulkCopy column mappings.
table.Columns.Add(new DataColumn(property.Name, propertyType));
var clrPropertyName = property.Name;
var tableColumnName = mappings[property.Name];
bulkCopy.ColumnMappings.Add(new SqlBulkCopyColumnMapping(clrPropertyName, tableColumnName));
}
// Add all our entities to our data table
foreach (var entity in entities)
{
var e = entity;
table.Rows.Add(properties.Select(property => GetPropertyValue(property.GetValue(e, null))).ToArray());
}
// send it to the server for bulk execution
bulkCopy.BulkCopyTimeout = 5*60;
bulkCopy.WriteToServer(table);
conn.Close();
}
private string GetTableName<T>() where T : class
{
var dbSet = Set<T>();
var sql = dbSet.ToString();
var regex = new Regex(@"FROM (?<table>.*) AS");
var match = regex.Match(sql);
return match.Groups["table"].Value;
}
private object GetPropertyValue(object o)
{
if (o == null)
return DBNull.Value;
return o;
}
private Dictionary<string, string> GetMappings(MetadataWorkspace workspace, string containerName, string entityName)
{
var mappings = new Dictionary<string, string>();
var storageMapping = workspace.GetItem<GlobalItem>(containerName, DataSpace.CSSpace);
dynamic entitySetMaps = storageMapping.GetType().InvokeMember(
"EntitySetMaps",
BindingFlags.GetProperty | BindingFlags.Public | BindingFlags.Instance,
null, storageMapping, null);
foreach (var entitySetMap in entitySetMaps)
{
var typeMappings = GetArrayList("EntityTypeMappings", entitySetMap);
dynamic typeMapping = typeMappings[0];
dynamic types = GetArrayList("Types", typeMapping);
if (types[0].Name == entityName)
{
var fragments = GetArrayList("MappingFragments", typeMapping);
var fragment = fragments[0];
var properties = GetArrayList("AllProperties", fragment);
foreach (var property in properties)
{
var edmProperty = GetProperty("EdmProperty", property);
var columnProperty = GetProperty("ColumnProperty", property);
mappings.Add(edmProperty.Name, columnProperty.Name);
}
}
}
return mappings;
}
private ArrayList GetArrayList(string property, object instance)
{
var type = instance.GetType();
var objects = (IEnumerable)type.InvokeMember(
property,
BindingFlags.GetProperty | BindingFlags.Public | BindingFlags.Instance, null, instance, null);
var list = new ArrayList();
foreach (var o in objects)
{
list.Add(o);
}
return list;
}
private dynamic GetProperty(string property, object instance)
{
var type = instance.GetType();
return type.InvokeMember(property, BindingFlags.GetProperty | BindingFlags.Public | BindingFlags.Instance, null, instance, null);
}
}
最后,为你Linq-To-Sql爱好者提供一些东西.
partial class MyDataContext
{
partial void OnCreated()
{
CommandTimeout = 5 * 60;
}
public void BulkInsertAll<T>(IEnumerable<T> entities)
{
entities = entities.ToArray();
string cs = Connection.ConnectionString;
var conn = new SqlConnection(cs);
conn.Open();
Type t = typeof(T);
var tableAttribute = (TableAttribute)t.GetCustomAttributes(
typeof(TableAttribute), false).Single();
var bulkCopy = new SqlBulkCopy(conn) {
DestinationTableName = tableAttribute.Name };
var properties = t.GetProperties().Where(EventTypeFilter).ToArray();
var table = new DataTable();
foreach (var property in properties)
{
Type propertyType = property.PropertyType;
if (propertyType.IsGenericType &&
propertyType.GetGenericTypeDefinition() == typeof(Nullable<>))
{
propertyType = Nullable.GetUnderlyingType(propertyType);
}
table.Columns.Add(new DataColumn(property.Name, propertyType));
}
foreach (var entity in entities)
{
table.Rows.Add(properties.Select(
property => GetPropertyValue(
property.GetValue(entity, null))).ToArray());
}
bulkCopy.WriteToServer(table);
conn.Close();
}
private bool EventTypeFilter(System.Reflection.PropertyInfo p)
{
var attribute = Attribute.GetCustomAttribute(p,
typeof (AssociationAttribute)) as AssociationAttribute;
if (attribute == null) return true;
if (attribute.IsForeignKey == false) return true;
return false;
}
private object GetPropertyValue(object o)
{
if (o == null)
return DBNull.Value;
return o;
}
}
您的代码有两个主要的性能问题：
使用添加方法
在您添加的每个实体中，Add 方法只会变得越来越慢。
请参阅：http : //entityframework.net/improve-ef-add-performance
例如，通过以下方式添加 10,000 个实体：
注意：实体尚未保存在数据库中！
问题是 Add 方法尝试在添加的每个实体上进行 DetectChanges，而 AddRange 在所有实体都添加到上下文后执行一次。
常见的解决方法有：
使用 SaveChanges
尚未为批量操作创建实体框架。对于您保存的每个实体，都会执行一次数据库往返。
所以，如果你想插入 20,000 条记录，你将执行 20,000 次数据库往返，这是INSANE！
有一些支持批量插入的第三方库可用：
请参阅：实体框架批量插入库
选择批量插入库时要小心。只有 Entity Framework Extensions 支持所有类型的关联和继承，并且它是唯一仍然受支持的。
免责声明：我是实体框架扩展的所有者
该库允许您执行场景所需的所有批量操作：
例子
// Easy to use
context.BulkSaveChanges();
// Easy to customize
context.BulkSaveChanges(bulk => bulk.BatchSize = 100);
// Perform Bulk Operations
context.BulkDelete(customers);
context.BulkInsert(customers);
context.BulkUpdate(customers);
// Customize Primary Key
context.BulkMerge(customers, operation => {
operation.ColumnPrimaryKeyExpression =
customer => customer.Code;
});
编辑：在评论中回答问题
您创建的库的每个批量插入是否有推荐的最大大小
不要太高，也不要太低。没有适合所有场景的特定值，因为它取决于多种因素，例如行大小、索引、触发器等。
一般建议在4000左右。
还有一种方法可以将它全部绑定在一个事务中，而不必担心它会超时
您可以使用实体框架事务。我们的库在启动时使用该事务。但是要小心，花费太多时间的事务也会带来一些问题，例如某些行/索引/表锁。
| 归档时间： |
|
| 查看次数： |
94273 次 |
| 最近记录： |