我有两个CSV文件,两个文件都有多个列和行,我期待在两个文件的所有行之间获得差异.让我们假设Asset Tag Number文件之间是否存在差异,然后突出显示任何形式的差异(可能是粗略的值或适当的东西),而且,我们在这里有一个密钥,Serial Number它在两个文件上都是唯一的.因此,最好将行的差异转换为new.csv文件,并在删除相同行时突出显示差异.
仅仅为了refe,我有两个文件有超过100列..
我的实际数据列如下所示在csv文件中.
Columns: [Asset Tag Number_a, Serial Number_a, System Name_a, Domain_a, System manufacturer_a, Model Name_a, System Type_a, Critical Level_a, Purpose Level 1_a, Purpose2_a, ShareIndv_a, Site_a, Building_a, Room_a, Rack_a, serverCostCenter_a, User ID BU Grp Mgr_a, OS Name_a, OS Version_a, OS Type_a, Service Pack_a, Notification Group_a, Off The Network_a, First Name_a, Last Name_a, Manager Name_a, Status_a, BU Cost Center_a, BU CC Description_a, Organization Name_a, Higher Level BU_a, Business Contact_a, Description_a, Asset Type_a, System Type SW_a, Server _a, Host ID(Unix)_a, IP Address_a, MAC Address_a, Installed RAM_a, Disk Capacity_a, Installed Disk_a, Server Status _a, High Level Status_a, Lifecycle Status_a, EndOfLifeDate_a, Last Audit_a, AltVersion_a, BIOS Vendor_a, BIOS Version_a, BIOS Release Date_a, SMBIOS Enabled_a, SMBios Version_a, Region_a, Currency_a, Acquisition Cost USD_a, Net Book Value USD_a, CPU Type_a, CPU Speed_a, Acquisition Date_a, Age_a, DateModified_a, Altiris Exception_a, Inventory Owner_a, Last Logon User_a, Inventory Owner Last Logon User_a, Client Date_a, Reporting Status_a, Contact Status_a, Comments_a, Exception Reason_a, DNR_a, Asset Tag Number_b, Serial Number_b, System Name_b, Domain_b, System manufacturer_b, Model Name_b, System Type_b, Critical Level_b, Purpose Level 1_b, Purpose2_b, ShareIndv_b, Site_b, Building_b, Room_b, Rack_b, serverCostCenter_b, User ID BU Grp Mgr_b, OS Name_b, OS Version_b, OS Type_b, Service Pack_b, Notification Group_b, Off The Network_b, First Name_b, Last Name_b, Manager Name_b, Status_b, BU Cost Center_b, ...]
Index: []
作为一个新手大熊猫学习者我应用了很少的代码方法,但似乎并不适合附近,因此寻求慷慨的帮助和建议..
1)第一个代码试过..
#!/grid/common/pkgs/python/v3.6.1/bin/python3
import pandas as pd
A = pd.read_csv('a.csv', index_col=0)
B = pd.read_csv('b.csv', index_col=0)
C = pd.merge(left=A,right=B, how='outer', left_index=True, right_index=True, suffixes=['_a', '_b'])
not_in_a = C.drop( A.index )
not_in_b = C.drop( B.index )
not_in_a.to_csv('not_in_a.csv')
not_in_b.to_csv('not_in_b.csv')
2)尝试了另一个代码,但是输出的宽度很大,难以阅读,而这个代码片段应该删除重复项,并且只会打印出差异的那个.
from __future__ import print_function
from signal import signal, SIGPIPE, SIG_DFL
signal(SIGPIPE,SIG_DFL)
import csv
import pandas as pd
##### Python pandas, widen output display to see more columns. ####
pd.set_option('display.height', None)
pd.set_option('display.max_rows', None)
pd.set_option('display.max_columns', None)
pd.set_option('display.width', None)
pd.set_option('expand_frame_repr', True)
a = pd.read_csv('a.csv')
b = pd.read_csv('b.csv')
c = pd.concat([a,b], axis=0)
c.drop_duplicates(keep='first', inplace=True)
c.reset_index(drop=True, inplace=True)
print(c)
我做了一些谷歌搜索,发现了一些关于该主题的堆栈溢出讨论.但是,在线程中有一些不错的解决方案,但我认为没有任何东西符合我的要求,因此我发布在这里.
3)应用python集的另一个代码部分工作..
#!/grid/common/pkgs/python/v3.6.1/bin/python3
import os
orig = open('aa.csv','r')
new = open('bb.csv','r')
bigb = set(new) - set(orig)
print(bigb)
# Write to output file
with open('different.csv', 'w') as file_out:
for line in bigb:
file_out.write(line)
orig.close()
new.close()
file_out.close()
我有两个样本文件供参考,它看起来类似于我的数据,我们可以把它Serial Number作为输出逻辑和代码的键.
下面是我的两个csv文件file1.csv和file2.csv
文件1:
wrkStaId Asset Tag Number Serial Number System Name
mac-ymatsuok2
PC-ABNER-W10
PC-ADAMLIN-W10
{ED0CCFFD-28D6-4170-9DE9-0DFB83F49193} 1234 ser123 sfreder
{8AEAF485-A4FF-460C-91FA-0DFCAD79DD24} 3456 ser124 10210277
{E6204B69-DABB-4A1E-906B-0DFD2BCEDA41} 456 ser345 A313819
{445EC096-A70C-47D1-91FF-0DFE747F762A} 4485 ser900 dgs1sj
示例文件2:
wrkStaId Asset Tag Number Serial Number System Name
mac-ymatsuok2
PC-Karn-W10
PC-ADAMLIN-W10
PC-ADRIANA-W10
{ED0CCFFD-28D6-4170-9DE9-0DFB83F49193} 1234 ser123 sfreder
{8AEAF485-A4FF-460C-91FA-0DFCAD79DD24} 3456 ser124 10210277
{E6204B69-DABB-4A1E-906B-0DFD2BCEDA41} 1709 ser345 A313819
{445EC096-A70C-47D1-91FF-0DFE747F762A} 4485 ser900 dgs1sj
期望的结果:您希望如何表示差异,因为这些是非数字值.是否要打印两行,以防它们与新文件不同,如果它们相同则删除它们？
ANS: Yes
期望的输出..
File1中的文件不同于file2
wrkStaId Asset Tag Number Serial Number System Name
PC-ABNER-W10
{E6204B69-DABB-4A1E-906B-0DFD2BCEDA41} 456 ser345 A313819
与File1不同的File2不同
wrkStaId Asset Tag Number Serial Number System Name
PC-Karn-W10
PC-ADRIANA-W10
{E6204B69-DABB-4A1E-906B-0DFD2BCEDA41} 1709 ser345 A313819
非常感谢@wm,但是我仍然希望从SO的专家那里获得更多的想法.
您的数据似乎包含两部分：一个列表System Name，然后是一个行表。由于结构完全不同，我建议您将数据拆分为 s 列表System Name和完整行，并分别处理它们。
首先提取System Name列表：
l1 = df1[df1.wrkStaId == ""].System_Name
l2 = df2[df2.wrkStaId == ""].System_Name
可以通过Python设置差异代码得到差异：
>>> set(l1).difference(set(l2))
{'PC-ABNER-W10'}
>>> set(l2).difference(set(l1))
{'PC-ADRIANA-W10', 'PC-Karn-W10'}
现在删除空的 wrkStaId 条目：
df1 = df1[df1.wrkStaId != ""].set_index("wrkStaId")
df2 = df2[df1.wrkStaId != ""].set_index("wrkStaId")
其余数据现在包含wrkStaId作为索引的完整行。
df1:
Asset_Tag_Number Serial_Number System_Name
wrkStaId
{ED0CCFFD-28D6-4170-9DE9-0DFB83F49193} 1234.0 ser123 sfreder
{8AEAF485-A4FF-460C-91FA-0DFCAD79DD24} 3456.0 ser124 10210277
{E6204B69-DABB-4A1E-906B-0DFD2BCEDA41} 456.0 ser345 A313819
{445EC096-A70C-47D1-91FF-0DFE747F762A} 4485.0 ser900 dgs1sj
df2:
Asset_Tag_Number Serial_Number System_Name
wrkStaId
{ED0CCFFD-28D6-4170-9DE9-0DFB83F49193} 1234.0 ser123 sfreder
{8AEAF485-A4FF-460C-91FA-0DFCAD79DD24} 3456.0 ser124 10210277
{E6204B69-DABB-4A1E-906B-0DFD2BCEDA41} 1709.0 ser345 A313819
{445EC096-A70C-47D1-91FF-0DFE747F762A} 4485.0 ser900 dgs1sj
您现在可以像这样对 pandas df 进行设置差异：
>>> df1[~df1.isin(df2).all(1)]
Asset_Tag_Number Serial_Number System_Name
wrkStaId
{E6204B69-DABB-4A1E-906B-0DFD2BCEDA41} 456.0 ser345 A313819
>>> df2[~df2.isin(df1).all(1)]
Asset_Tag_Number Serial_Number System_Name
wrkStaId
{E6204B69-DABB-4A1E-906B-0DFD2BCEDA41} 1709.0 ser345 A313819
您可能需要稍微调整代码才能得到您想要的，但我希望这能让您继续前进。