Pandas如何过滤数据帧列中特定范围的数字

Question

我试图想出一种过滤数据帧的方法，以便它仅包含进一步处理所需的特定范围的数字。下面是一个示例数据框

data_sample = [['part1', 234], ['part2', 224], ['part3', 214],['part4', 114],['part5', 1111],
                ['part6',1067],['part7',1034],['part8',1457],['part9', 789],['part10',1367],
                ['part11',467],['part12',367]
        ]
data_df = pd.DataFrame(data_sample, columns = ['partname', 'sbin'])
data_df['sbin'] = pd.to_numeric(data_df['sbin'], errors='coerce', downcast='integer')

对于上面的数据帧，我想过滤，以便删除 sbin 在 [200-230] 和 [1000-1150] 以及 [350-370] 和 [100-130] 范围内的任何部分。

我有一个更大的数据框，需要删除更多范围，因此需要比使用以下命令更快的方法

data_df.loc[~( ((data_df.sbin >=200) & (data_df.sbin <= 230)) | ((data_df.sbin >=100) & (data_df.sbin <= 130)) | ((data_df.sbin >=350) & (data_df.sbin <= 370))| ((data_df.sbin >=1000) & (data_df.sbin <= 1150)))]

产生如下输出

    partname    sbin
0   part1       234
7   part8       1457
8   part9       789
9   part10      1367
10  part11      467

上面的方法需要很多条件并且需要很长时间，我想知道是否有更好的方法使用正则表达式或其他一些我不知道的python方法。

任何帮助都会很棒

Answer 1

pd.cut在这里工作得很好，特别是当你的时间间隔不重叠时：

intervals = pd.IntervalIndex.from_tuples([(200, 230), (1000, 1150), (350, 370), (100, 130)])

# if the values do not fall within the intervals, it is a null
# hence the isna check to keep only the null matches
# thanks to @corralien for the include_lowest=True suggestion

data_df.loc[pd.cut(data_df.sbin, intervals, include_lowest=True).isna()]

   partname  sbin
0     part1   234
7     part8  1457
8     part9   789
9    part10  1367
10   part11   467

Answer 2

新版本

\n

使用np.logical_and和any选择范围内的值并反转掩码以保留其他值。

\n

intervals = [(100, 130), (200, 230), (350, 370), (1000, 1150)]\nm = np.any([np.logical_and(data_df['sbin'] >= l, data_df['sbin'] <= u)\n                                    for l, u in intervals], axis=0)\nout = data_df.loc[~m]\n

\n

注释any可以替换为np.logical_or.reduce：

\n

intervals = [(100, 130), (200, 230), (350, 370), (1000, 1150)]\nm = np.logical_or.reduce([np.logical_and(data_df['sbin'] >= l, data_df['sbin'] <= u)\n                                    for l, u in intervals])\nout = data_df.loc[~m]\n

\n

输出结果：

\n

>>> out\n   partname  sbin\n0     part1   234\n7     part8  1457\n8     part9   789\n9    part10  1367\n10   part11   467\n

\n\n

旧版

\n

不能按原样使用浮点数

\n

使用np.where和in1d：

\n

intervals = [(100, 130), (200, 230), (350, 370), (1000, 1150)]\nm = np.hstack([np.arange(l, u+1) for l, u in intervals])\nout = data_df.loc[~np.in1d(data_df['sbin'], m)]\n

\n\n

性能：对于 100k 条记录：

\n

data_df = pd.DataFrame({'sbin': np.random.randint(0, 2000, 100000)})\n\ndef exclude_range_danimesejo():\n    intervals = sorted([(200, 230), (1000, 1150), (350, 370), (100, 130)])\n    intervals = np.array(intervals).flatten()\n    mask = (np.searchsorted(intervals, data_df['sbin']) % 2 == 0) & ~np.in1d(data_df['sbin'], intervals[::2])\n    return data_df.loc[mask]\n\ndef exclude_range_sammywemmy():\n    intervals = pd.IntervalIndex.from_tuples([(200, 230), (1000, 1150), (350, 370), (100, 130)])\n    return data_df.loc[pd.cut(data_df.sbin, intervals, include_lowest=True).isna()]\n\ndef exclude_range_corralien():\n    intervals = [(100, 130), (200, 230), (350, 370), (1000, 1150)]\n    m = np.hstack([np.arange(l, u+1) for l, u in intervals])\n    return data_df.loc[~np.in1d(data_df['sbin'], m)]\n\ndef exclude_range_corralien2():\n    intervals = [(100, 130), (200, 230), (350, 370), (1000, 1150)]\n    m = np.any([np.logical_and(data_df['sbin'] >= l, data_df['sbin'] <= u)\n                                        for l, u in intervals], axis=0)\n    return data_df.loc[~m]\n

\n

>>> %timeit exclude_range_danimesejo()\n2.66 ms \xc2\xb1 18.2 \xc2\xb5s per loop (mean \xc2\xb1 std. dev. of 7 runs, 100 loops each)\n\n>>> %timeit exclude_range_sammywemmy()\n63.6 ms \xc2\xb1 549 \xc2\xb5s per loop (mean \xc2\xb1 std. dev. of 7 runs, 10 loops each)\n\n>>> %timeit exclude_range_corralien()\n6.87 ms \xc2\xb1 58.8 \xc2\xb5s per loop (mean \xc2\xb1 std. dev. of 7 runs, 100 loops each)\n\n>>> %timeit exclude_range_corralien2()\n2.26 ms \xc2\xb1 8.9 \xc2\xb5s per loop (mean \xc2\xb1 std. dev. of 7 runs, 100 loops each)\n

\n