对于不区分大小写的数据，为什么不使用两列而不是排序规则？

Question

我需要在 Postgres 15 中存储电子邮件（如“ivan@email.com”），能够以不区分大小写的方式搜索它们（“iVaN@email.com”、“IVAN@email.com”等是相同的） ），并能够检索原始电子邮件以将其用于实际发送电子邮件。

处理不区分大小写的数据的建议方法是使用排序规则：

DROP TABLE IF EXISTS test_collation;
DROP COLLATION IF EXISTS case_insensitive;

CREATE COLLATION case_insensitive (PROVIDER = icu, LOCALE = '@colStrength=secondary', DETERMINISTIC = FALSE);

CREATE TABLE test_collation
(
    original_email TEXT COLLATE case_insensitive NOT NULL UNIQUE PRIMARY KEY
);

INSERT INTO test_collation (original_email)
VALUES ('IVAN@email.com');

-- This entry fails as expected as a duplicate:
INSERT INTO test_collation (original_email)
VALUES ('ivAn@email.com');

-- Getting the original email provided by the user regardless of case in which it is entered:
SELECT original_email
FROM test_collation
WHERE original_email = 'iVaN@EmaIl.com';

到目前为止一切顺利，但文档指出这种方法在性能方面有其缺点。这是有道理的，因为对非确定性数据建立索引具有挑战性。

这让我想到：如果我们使用两列，一列将存储原始电子邮件，第二列将存储小写电子邮件。为了确保万无一失，小写的电子邮件将是一个生成列（因此无法手动更改），同时也是一个主键以避免重复。在小写电子邮件列上搜索可能非常高效，因为它本质上是确定性的并且可以使用 B 树。例子：

DROP TABLE IF EXISTS test_two_columns;
CREATE TABLE test_two_columns
(
    original_email TEXT NOT NULL UNIQUE,
    lowered_email  TEXT NOT NULL UNIQUE PRIMARY KEY GENERATED ALWAYS AS ( LOWER(original_email) ) STORED
);

INSERT INTO test_two_columns (original_email)
VALUES ('IVAN@email.com');

-- This entry fails as expected as a duplicate:
INSERT INTO test_two_columns (original_email)
VALUES ('ivAn@email.com');

-- Getting the original email provided by the user regardless of case in which it is entered:
SELECT original_email
FROM test_two_columns
WHERE lowered_email = LOWER('iVaN@EmaIl.com');

除了明显浪费附加列的空间之外，这种解决方案还有哪些缺点？

Answer 1

第三个选项：

CREATE TABLE test
(
    original_email TEXT NOT NULL
);

CREATE UNIQUE INDEX test_email_uniq on test(lower(original_email));

INSERT INTO test (original_email)
VALUES ('IVAN@email.com');

-- This entry fails as expected as a duplicate:
INSERT INTO test (original_email)
VALUES ('ivAn@email.com');

-- Getting the original email provided by the user regardless of case in which it is entered:
SELECT original_email
FROM test
WHERE LOWER(original_email) = LOWER('iVaN@EmaIl.com');

在这种情况下，我们不在表中存储附加字段，仅存储唯一索引本身。该条件WHERE LOWER(original_email) = lower(?)将使用此唯一索引来加速此查询。

Answer 2

使用生成列的缺点是每次修改行时都会浪费空间和计算计算列所需的处理时间。另一方面，索引的修改和搜索速度会更快（特别是如果您使用C排序规则）。

两者都是可行的解决方案。如果良好的性能是您的主要目标，我建议您使用实际数据对这两种方法进行基准测试。

Answer 3

我使用所有方法以及区分大小写的方法作为基线进行了一些测试，通过向每个表插入 1,000,000 个条目，并使用相同长度的随机大小写的随机字符串，以下是 Macos 上 Podman 中 PostgreSQL 15 的结果：

I. 区分大小写（仅适用于基准测试基线） 插入：11 秒 698 毫秒内影响 1,000,000 行 选择：24 毫秒内检索 0 行（执行：3 毫秒，获取：21 毫秒）

二. 使用不区分大小写的排序规则插入：在 11 秒 880 毫秒内影响 1,000,000 行选择：在 27 毫秒内检索 0 行（执行：3 毫秒，获取：24 毫秒）

三．@melkij 的较低索引解决方案插入：1,000,000 行在 15 秒 418 毫秒内受影响选择：0 行在 24 毫秒内检索（执行：2 毫秒，获取：22 毫秒）

四．两列解决方案插入：15 ​​秒 627 毫秒内影响 1,000,000 行选择：23 毫秒内检索 0 行（执行：1 毫秒，获取：22 毫秒）

结论：我使用的两列解决方案速度很慢，代码中错误使用的空间更大，并且浪费空间。“小写索引”和不区分大小写的排序规则解决方案为我提供了可比的结果，但基于排序规则更易于使用并且速度更快，因此我将使用基于排序规则的解决方案。

这是完整的代码：

--
-- Case-SENSITIVE solution (just as a baseline for benchmarking)
--

DROP TABLE IF EXISTS test_case_sensitive;

CREATE TABLE test_case_sensitive
(
    original_email TEXT NOT NULL UNIQUE PRIMARY KEY
);

INSERT INTO test_case_sensitive (original_email)
VALUES ('IVAN@email.com');

-- This entry SUCCEEDS unlike case-insensitive solutions:
INSERT INTO test_case_sensitive (original_email)
VALUES ('ivAn@email.com');

-- Getting the original email provided by the user:
SELECT original_email
FROM test_case_sensitive
WHERE original_email = 'iVaN@EmaIl.com';

--
-- Case-insensitive solution with collation
--

DROP TABLE IF EXISTS test_collation;
DROP COLLATION IF EXISTS case_ins;

CREATE COLLATION case_ins (PROVIDER = icu, LOCALE = '@colStrength=secondary', DETERMINISTIC = FALSE);

CREATE TABLE test_collation
(
    original_email TEXT COLLATE case_ins NOT NULL UNIQUE PRIMARY KEY
);

INSERT INTO test_collation (original_email)
VALUES ('IVAN@email.com');

-- This entry fails as expected:
INSERT INTO test_collation (original_email)
VALUES ('ivAn@email.com');

-- Sending an email to exact email provided by the user:
SELECT original_email
FROM test_collation
WHERE original_email = 'iVaN@EmaIl.com';

--
-- Case-insensitive solution with an index on 'lowered' original email
--

DROP TABLE IF EXISTS test_lower_index;
DROP INDEX IF EXISTS test_email_uniq;

CREATE TABLE test_lower_index
(
    original_email TEXT NOT NULL UNIQUE PRIMARY KEY
);

CREATE UNIQUE INDEX test_email_uniq ON test_lower_index (LOWER(original_email));

INSERT INTO test_lower_index (original_email)
VALUES ('IVAN@email.com');

-- This entry fails as expected as a duplicate:
INSERT INTO test_lower_index (original_email)
VALUES ('ivAn@email.com');

-- Getting the original email provided by the user regardless of case in which it is entered:
SELECT original_email
FROM test_lower_index
WHERE LOWER(original_email) = LOWER('iVaN@EmaIl.com');

--
-- Case-insensitive solution with two columns
--

DROP TABLE IF EXISTS test_two_columns;
CREATE TABLE test_two_columns
(
    original_email TEXT NOT NULL UNIQUE,
    lowered_email  TEXT NOT NULL UNIQUE PRIMARY KEY GENERATED ALWAYS AS ( LOWER(original_email) ) STORED
);

INSERT INTO test_two_columns (original_email)
VALUES ('IVAN@email.com');

-- This entry fails as expected:
INSERT INTO test_two_columns (original_email)
VALUES ('ivAn@email.com');

-- Sending an email to exact email provided by the user:
SELECT original_email
FROM test_two_columns
WHERE lowered_email = LOWER('iVaN@EmaIl.com');

--
-- Testing
--

CREATE OR REPLACE FUNCTION random_string(length INTEGER) RETURNS TEXT AS
$$
DECLARE
    chars  TEXT[] := '{A,B,C,D,E,F,G,H,I,J,K,L,M,N,O,P,Q,R,S,T,U,V,W,X,Y,Z,a,b,c,d,e,f,g,h,i,j,k,l,m,n,o,p,q,r,s,t,u,v,w,x,y,z}';
    result TEXT   := '';
BEGIN
    IF length < 0 THEN
        RAISE EXCEPTION 'Given length cannot be less than 0';
    END IF;
    FOR _ IN 1..length
        LOOP
            result := result || chars[1 + RANDOM() * (ARRAY_LENGTH(chars, 1) - 1)];
        END LOOP;
    RETURN result;
END;
$$ LANGUAGE plpgsql;

TRUNCATE test_case_sensitive;

INSERT INTO test_case_sensitive(original_email)
SELECT random_string(30)
FROM GENERATE_SERIES(1, 1000000) id;

TRUNCATE test_collation;

INSERT INTO test_collation(original_email)
SELECT random_string(30)
FROM GENERATE_SERIES(1, 1000000) id;

TRUNCATE test_lower_index;

INSERT INTO test_lower_index(original_email)
SELECT random_string(30)
FROM GENERATE_SERIES(1, 1000000) id;

TRUNCATE test_two_columns;

INSERT INTO test_two_columns(original_email)
SELECT random_string(30)
FROM GENERATE_SERIES(1, 1000000) id;

SELECT original_email
FROM test_case_sensitive
WHERE original_email = 'iVaN@EmaIl.com';

SELECT original_email
FROM test_collation
WHERE original_email = 'iVaN@EmaIl.com';

SELECT original_email
FROM test_lower_index
WHERE LOWER(original_email) = LOWER('iVaN@EmaIl.com');

SELECT original_email
FROM test_two_columns
WHERE lowered_email = LOWER('iVaN@EmaIl.com');