根据第二个文本文件从文本文件中删除重复项

Question

如何main.txt通过检查第二个文本文件()从文本文件()中删除所有行removethese.txt.什么是有效的方法,如果文件大于10-100mb更大.[使用mac]

例:

main.txt
3
1
2
5

删除这些行

removethese.txt
3
2
9

输出:

output.txt
1
5

示例行(这些是我正在使用的实际行 - 顺序无关紧要):

ChIJW3p7Xz8YyIkRBD_TjKGJRS0
ChIJ08x-0kMayIkR5CcrF-xT6ZA
ChIJIxbjOykFyIkRzugZZ6tio1U
ChIJiaF4aOoEyIkR2c9WYapWDxM
ChIJ39HoPKDix4kRcfdIrxIVrqs
ChIJk5nEV8cHyIkRIhmxieR5ak8
ChIJs9INbrcfyIkRf0zLkA1NJEg
ChIJRycysg0cyIkRArqaCTwZ-E8
ChIJC8haxlUDyIkRfSfJOqwe698
ChIJxRVp80zpcEARAVmzvlCwA24
ChIJw8_LAaEEyIkR68nb8cpalSU
ChIJs35yqObit4kR05F4CXSHd_8
ChIJoRmgSdwGyIkRvLbhOE7xAHQ
ChIJaTtWBAWyVogRcpPDYK42-Nc
ChIJTUjGAqunVogR90Kc8hriW8c
ChIJN7P2NF8eVIgRwXdZeCjL5EQ
ChIJizGc0lsbVIgRDlIs85M5dBs
ChIJc8h6ZqccVIgR7u5aefJxjjc
ChIJ6YMOvOeYVogRjjCMCL6oQco
ChIJ54HcCsaeVogRIy9___RGZ6o
ChIJif92qn2YVogR87n0-9R5tLA
ChIJ0T5e1YaYVogRifrl7S_oeM8
ChIJwWGce4eYVogRcrfC5pvzNd4

Answer 1

有两种标准方法可以做到这一点:

用grep:

grep -vxFf removethese main

这用于:

• -v 反转比赛.
• -x匹配整行,以防止,例如,he匹配像hello或的行highway to hell.
• -F 使用固定字符串,以便按原样获取参数,而不是将其解释为正则表达式.
• -f从另一个文件中获取模式.在这种情况下,来自removethese.

用awk:

$ awk 'FNR==NR {a[$0];next} !($0 in a)' removethese main
1
5

像这样我们将每一行存储removethese在一个数组中a[].然后,我们读取main文件并打印那些不存在于数组中的行.

Answer 2

用grep:

grep -vxFf removethese.txt main.txt >output.txt

用fgrep:

fgrep -vxf removethese.txt main.txt >output.txt

fgrep已弃用.fgrep --help说:

作为'fgrep'的调用已被弃用; 请改用'grep -F'.

随着awk(来自@fedorqui):

awk 'FNR==NR {a[$0];next} !($0 in a)' removethese.txt main.txt >output.txt

用sed:

sed "s=^=/^=;s=$=$/d=" removethese.txt | sed -f- main.txt >output.txt

如果removethese.txt包含特殊字符,则会失败.为此你可以这样做:

sed 's/[^^]/[&]/g; s/\^/\\^/g' removethese.txt >newremovethese.txt

并在命令中使用newremovethese.txtsed.但这不值得努力,与其他方法相比,它太慢了.

对上述方法进行测试:

该sed方法需要花费太多时间而不值得测试.

使用的文件:

removethese.txt : Size: 15191908 (15MB)     Blocks: 29672   Lines: 100233
main.txt : Size: 27640864 (27.6MB)      Blocks: 53992   Lines: 180034

命令:
grep -vxFf| fgrep -vxf|awk

需要时间:
0m7.966s| 0m7.823s| 0m0.237s
0m7.877s| 0m7.889s| 0m0.241s
0m7.971s| 0m7.844s| 0m0.234s
0m7.864s| 0m7.840s| 0m0.251s
0m7.798s| 0m7.672s| 0m0.238s
0m7.793s| 0m8.013s|0m0.241s

AVG
0m7.8782s| 0m7.8468s|0m0.2403s

这个测试结果意味着比它fgrep快一点grep.

该awk方法(来自@fedorqui)通过测试用飞行颜色(0.2403 seconds仅!!!).

测试环境:

HP ProBook 440 G1 Laptop
8GB RAM
2.5GHz processor with turbo boost upto 3.1GHz
RAM being used: 2.1GB
Swap being used: 588MB
RAM being used when the grep/fgrep command is run: 3.5GB
RAM being used when the awk command is run: 2.2GB or less
Swap being used when the commands are run: 588MB (No change)

测试结果:

使用该awk方法.