需要SQL优化(可能是DISTINCT ON的原因？)

Question

相关的前一个问题:
在按值(而不是列)分组后,从组中选择一个随机条目？

我当前的查询如下所示:

WITH
  points AS (
    SELECT unnest(array_of_points) AS p
  ),

 gtps AS (
   SELECT DISTINCT ON(points.p)
     points.p, m.groundtruth
   FROM measurement m, points
   WHERE st_distance(m.groundtruth, points.p) < distance
   ORDER BY points.p, RANDOM()
 )

SELECT DISTINCT ON(gtps.p, gtps.groundtruth, m.anchor_id)
  m.id, m.anchor_id, gtps.groundtruth, gtps.p
FROM measurement m, gtps
ORDER BY gtps.p, gtps.groundtruth, m.anchor_id, RANDOM()

语义:

1. 有两个输入值:

   • 第4行:点数组 array_of_points
   • 第12行:双精度数: distance

2. 第一段(第1-6行):

   • 从points数组创建一个表,用于...

3. 第二段(第8-14行):

   • 对于points表格内的每个点:从表格中获取一个距离< 的随机(!)groundtruth点measurementdistance
   • 将这些元组保存在gtps表中

4. 第三段(第16-19行):

   • 对于表中的每个groundtruth值gtps:获取所有anchor_id值和...
   • 如果anchor_id值不唯一:则选择随机值

5. 输出:id,anchor_id,groundtruth,p(从输入值array_of_points)

示例表:

id | anchor_id | groundtruth | data
-----------------------------------
1  | 1         | POINT(1 4)  | ...
2  | 3         | POINT(1 4)  | ...
3  | 8         | POINT(1 4)  | ...
4  | 6         | POINT(1 4)  | ...
-----------------------------------
5  | 2         | POINT(3 2)  | ...
6  | 4         | POINT(3 2)  | ...
-----------------------------------
7  | 1         | POINT(4 3)  | ...
8  | 1         | POINT(4 3)  | ...
9  | 6         | POINT(4 3)  | ...
10 | 7         | POINT(4 3)  | ...
11 | 3         | POINT(4 3)  | ...
-----------------------------------
12 | 1         | POINT(6 2)  | ...
13 | 5         | POINT(6 2)  | ...

示例结果:

id  | anchor_id | groundtruth | p
-----------------------------------------
1   | 1         | POINT(1 4)  | POINT(1 0)
2   | 3         | POINT(1 4)  | POINT(1 0)
4   | 6         | POINT(1 4)  | POINT(1 0)
3   | 8         | POINT(1 4)  | POINT(1 0)
5   | 2         | POINT(3 2)  | POINT(2 2)
6   | 4         | POINT(3 2)  | POINT(2 2)
1   | 1         | POINT(1 4)  | POINT(4 8)
2   | 3         | POINT(1 4)  | POINT(4 8)
4   | 6         | POINT(1 4)  | POINT(4 8)
3   | 8         | POINT(1 4)  | POINT(4 8)
12  | 1         | POINT(6 2)  | POINT(7 3)
13  | 5         | POINT(6 2)  | POINT(7 3)
1   | 1         | POINT(4 3)  | POINT(9 1)
11  | 3         | POINT(4 3)  | POINT(9 1)
9   | 6         | POINT(4 3)  | POINT(9 1)
10  | 7         | POINT(4 3)  | POINT(9 1)

如你看到的:

• 每个输入值可以具有多个相等的groundtruth值.
• 如果输入值具有多个groundtruth值,则它们必须全部相等.
• 每个groundtruth-inputPoint-tuple都与该groundtruth的每个可能性相连anchor_id.
• 两个不同的输入值可以具有相同的对应groundtruth值.
• 两个不同的groundtruth-inputPoint-tuples可以具有相同的结构 anchor_id
• 两个indentical groundtruth-inputPoint-tuples必须具有不同的anchor_ids

基准(两个输入值):

• 1-6行:16ms
• 8-14行:48ms
• 16-19行:600ms

EXPLAIN VERBOSE:

Unique  (cost=11119.32..11348.33 rows=18 width=72)
  Output: m.id, m.anchor_id, gtps.groundtruth, gtps.p, (random())
  CTE points
    ->  Result  (cost=0.00..0.01 rows=1 width=0)
          Output: unnest('{0101000000EE7C3F355EF24F4019390B7BDA011940:01010000003480B74082FA44402CD49AE61D173C40}'::geometry[])
  CTE gtps
    ->  Unique  (cost=7659.95..7698.12 rows=1 width=160)
          Output: points.p, m.groundtruth, (random())
          ->  Sort  (cost=7659.95..7679.04 rows=7634 width=160)
                Output: points.p, m.groundtruth, (random())
                Sort Key: points.p, (random())
                ->  Nested Loop  (cost=0.00..6565.63 rows=7634 width=160)
                      Output: points.p, m.groundtruth, random()
                      Join Filter: (st_distance(m.groundtruth, points.p) < m.distance)
                      ->  CTE Scan on points  (cost=0.00..0.02 rows=1 width=32)
                            Output: points.p
                      ->  Seq Scan on public.measurement m  (cost=0.00..535.01 rows=22901 width=132)
                            Output: m.id, m.anchor_id, m.tag_node_id, m.experiment_id, m.run_id, m.anchor_node_id, m.groundtruth, m.distance, m.distance_error, m.distance_truth, m."timestamp"
  ->  Sort  (cost=3421.18..3478.43 rows=22901 width=72)
        Output: m.id, m.anchor_id, gtps.groundtruth, gtps.p, (random())
        Sort Key: gtps.p, gtps.groundtruth, m.anchor_id, (random())
        ->  Nested Loop  (cost=0.00..821.29 rows=22901 width=72)
              Output: m.id, m.anchor_id, gtps.groundtruth, gtps.p, random()
              ->  CTE Scan on gtps  (cost=0.00..0.02 rows=1 width=64)
                    Output: gtps.p, gtps.groundtruth
              ->  Seq Scan on public.measurement m  (cost=0.00..535.01 rows=22901 width=8)
                    Output: m.id, m.anchor_id, m.tag_node_id, m.experiment_id, m.run_id, m.anchor_node_id, m.groundtruth, m.distance, m.distance_error, m.distance_truth, m."timestamp"

EXPLAIN ANALYZE:

Unique  (cost=11119.32..11348.33 rows=18 width=72) (actual time=548.991..657.992 rows=36 loops=1)
  CTE points
    ->  Result  (cost=0.00..0.01 rows=1 width=0) (actual time=0.004..0.011 rows=2 loops=1)
  CTE gtps
    ->  Unique  (cost=7659.95..7698.12 rows=1 width=160) (actual time=133.416..146.745 rows=2 loops=1)
          ->  Sort  (cost=7659.95..7679.04 rows=7634 width=160) (actual time=133.415..142.255 rows=15683 loops=1)
                Sort Key: points.p, (random())
                Sort Method: external merge  Disk: 1248kB
                ->  Nested Loop  (cost=0.00..6565.63 rows=7634 width=160) (actual time=0.045..46.670 rows=15683 loops=1)
                      Join Filter: (st_distance(m.groundtruth, points.p) < m.distance)
                      ->  CTE Scan on points  (cost=0.00..0.02 rows=1 width=32) (actual time=0.007..0.020 rows=2 loops=1)
                      ->  Seq Scan on measurement m  (cost=0.00..535.01 rows=22901 width=132) (actual time=0.013..3.902 rows=22901 loops=2)
  ->  Sort  (cost=3421.18..3478.43 rows=22901 width=72) (actual time=548.989..631.323 rows=45802 loops=1)
        Sort Key: gtps.p, gtps.groundtruth, m.anchor_id, (random())"
        Sort Method: external merge  Disk: 4008kB
        ->  Nested Loop  (cost=0.00..821.29 rows=22901 width=72) (actual time=133.449..166.294 rows=45802 loops=1)
              ->  CTE Scan on gtps  (cost=0.00..0.02 rows=1 width=64) (actual time=133.420..146.753 rows=2 loops=1)
              ->  Seq Scan on measurement m  (cost=0.00..535.01 rows=22901 width=8) (actual time=0.014..4.409 rows=22901 loops=2)
Total runtime: 834.626 ms

在运行时,这应该以大约100-1000个输入值运行.所以现在它需要35到350秒,这是非常多的.

我已经尝试删除这些RANDOM()功能.这将运行时间(对于2个输入值)从大约670ms减少到大约530ms.所以这不是目前的主要影响.

如果更容易/更快,也可以运行2或3个单独的查询并在软件中执行某些部分(它在Ruby on Rails服务器上运行).例如随机选择？!

工作正在进行中:

SELECT
  m.groundtruth, ps.p, ARRAY_AGG(m.anchor_id), ARRAY_AGG(m.id)
FROM
  measurement m
JOIN
  (SELECT unnest(point_array) AS p) AS ps
  ON ST_DWithin(ps.p, m.groundtruth, distance)
GROUP BY groundtruth, ps.p

使用此查询它非常快(15 毫秒),但缺少很多:

• 我只需要一个随机行 ps.p
• 这两个阵列彼此属于一个.意思是:里面的物品的顺序很重要!
• 这两个数组需要进行过滤(随机):
  对于anchor_id出现多次的数组中的每个数组:保留一个随机数并删除所有其他数组.这也意味着id从id每个删除的数组中删除相应的anchor_id

如果anchor_id并且id可以存储在元组数组中也很好.例如:( {[4,1],[6,3],[4,2],[8,5],[4,4]}约束:每个元组都是唯一的,每个id(在示例中= =第二个值)都是唯一的,anchor_ids不是唯一的).此示例显示的查询不包含仍必须应用的过滤器.应用过滤器后,它看起来像这样{[6,3],[4,4],[8,5]}.

正在进行的工作II:

SELECT DISTINCT ON (ps.p)
  m.groundtruth, ps.p, ARRAY_AGG(m.anchor_id), ARRAY_AGG(m.id)
FROM
  measurement m
JOIN
  (SELECT unnest(point_array) AS p) AS ps
  ON ST_DWithin(ps.p, m.groundtruth, distance)
GROUP BY ps.p, m.groundtruth
ORDER BY ps.p, RANDOM()

这现在给出了相当不错的结果并且仍然非常快:16ms
还有一件事要做:

• ARRAY_AGG(m.anchor_id) 已经随机化了,但是:
• 它包含许多重复的条目,因此:
• 我想在它上面使用类似DISTINCT的东西,但是:
• 它必须与...同步ARRAY_AGG(m.id).这意味着:
  如果DISTINCT命令保留anchor_id数组的索引1,4和7 ,那么它还要保留数组的索引1,4和7 id(当然还要删除所有其他索引)

Answer 1

如果anchor_id 和id 可以存储在元组数组中，那就太好了。

多维数组的聚合函数

我想你为此创建了一个二维数组。这比ARRAY of record. 标准array_agg()无法聚合多维数组。但是您可以为此轻松编写自己的聚合函数：

CREATE AGGREGATE array_agg_mult (anyarray)  (
    SFUNC     = array_cat
   ,STYPE     = anyarray
   ,INITCOND  = '{}'
);

阅读相关答案中的解释：
Selecting data into a Postgres array

对于数组中出现多次的每个anchor_id：保留一个随机的并删除所有其他的。这也意味着从 id 数组中删除每个删除的anchor_id 对应的 id

询问

SELECT DISTINCT ON (p)
       p, groundtruth, array_agg_mult(ARRAY[ARRAY[anchor_id, id]]) AS ids
FROM (
   SELECT DISTINCT ON (ps.p, m.groundtruth, m.anchor_id)
          ps.p, m.groundtruth, m.anchor_id, m.id
   FROM  (SELECT unnest(point_array) AS p) AS ps
   JOIN   measurement m ON ST_DWithin(ps.p, m.groundtruth, distance)
   ORDER  BY ps.p, m.groundtruth, m.anchor_id, random()
   ) x
GROUP  BY p, groundtruth
ORDER  BY p, random();

• 如果有多个对等点，子查询x将获得不同的anchor_id值并选择随机行。(p, groundtruth)这样连接就anchor_id - id可以保持完整。

• 外部查询聚合一个二维数组，如您所愿，按 排序anchor_id。如果您想anchor_id随机订购，请再次使用 random：

  array_agg_mult(ARRAY[ARRAY[anchor_id, id]] ORDER BY random())
  

• 最后，再次随机DISTINCT ON挑选 1groundtruth个p。