那些遇到过的问题

Oracle - 当只对不同的匹配感兴趣时,优化CLOB列上所有正则表达式匹配的循环

5to*_*on3 2 regex oracle performance loops distinct

我正在创建一个循环遍历表(具有数千行)的存储过程,并且对于每一行,都有一个CLOB列,我想从中获取正则表达式上的所有匹配项(sa"FNR").此后,我想在新表中插入每个不同的匹配.单个CLOB列可能包含数千个匹配,但通常它是在CLOB中重复的相同"FNR" - 即不同的正则表达式匹配少得多,并且这些是我唯一感兴趣的.但是,我制作的程序花了很长时间,我怀疑在所有比赛中循环是最耗时的部分.

我的程序看起来像这样:

CREATE OR REPLACE PROCEDURE SP_MTV_FINN_FNR AS
BEGIN
DECLARE 
  v_n NUMBER;
  v_cnt NUMBER;
  v_mtrid NUMBER;
  v_regex_fnr VARCHAR2(54) := '(((0[1-9]|[12]\d|3[01])(0[1-9]|1[012])(\d{2}))(\d{5}))';
  v_doc CLOB;
  v_fnr VARCHAR2(11);

  BEGIN
    -- Get all rows from table --
    SELECT COUNT(*) INTO v_n FROM TABLE;
    IF v_n > 0 THEN
      -- Loop over all rows --
      FOR i IN 1..v_n LOOP
        SELECT doc, mtrid
          INTO v_doc, v_mtrid
          FROM (SELECT DOC doc, ID mtrid, ROWNUM rnum
            FROM TABLE
            WHERE ROWNUM <=i)
          WHERE rnum >= i;
        IF v_doc IS NOT NULL THEN
          SELECT REGEXP_COUNT(v_doc, v_regex_fnr) INTO v_cnt FROM DUAL;
          IF v_cnt >= 1 THEN
            -- For each regex match - time consuming, right? --
            FOR j IN 1..v_cnt LOOP
              SELECT REGEXP_SUBSTR(v_doc, v_regex_fnr, 1, j, 'm') INTO v_fnr FROM DUAL;
                IF CHECK_FNR(v_fnr) = 'TRUE' THEN
                  INSERT INTO TABLE2(MTR_ID, FNR)
                    SELECT v_mtrid, v_fnr FROM DUAL;
                END IF;
            END LOOP;
          END IF;
        END IF;
        COMMIT;
      END LOOP;
    END IF;
  END LOOP;
EXCEPTION WHEN OTHERS THEN
  DBMS_OUTPUT.PUT_LINE('Error - rollback');
  DBMS_OUTPUT.PUT_LINE('The error code is ' || SQLCODE || '- ' || SQLERRM);
  ROLLBACK;
END;
Run Code Online (Sandbox Code Playgroud)

/

有人知道如何优化这个程序吗？

我正在使用Oracle 11.2.0.3.0.(顺便说一句,我知道ctx_entity-package,但它在这个版本上被禁用.但是,我正在考虑启用它.)

UPDATE

在应用了nop77svk提供的非常有用的性能优化技术之后,我可以肯定地说regexp_substr()a over CLOB是瓶颈,因为遗憾的是没有性能提升.然而,我提出了一个"黑客/解决方法",我最大限度地减少了regexp_substr()通话量,并带来了巨大的性能提升.首先,我考虑制作一个逐步"训练"的正则表达式,不包括以前的匹配,但由于Oracle不支持负向前瞻,这不起作用.我最终保存了CLOB,并使用regexp_replace()删除匹配的所有出现.由于CLOB中发生了很多相同的事件,因此从许多regexp_substr()调用中保存了该过程,并且它同时处理了不同的要求.

根据nop77svk的贡献,我的结果如下.是的,我回到DUAL了MERGE-statement中使用,但这里有什么办法吗？

CREATE OR REPLACE PROCEDURE SP_MTV_FINN_FNR2 AS
BEGIN
DECLARE 
v_regex_fnr                 VARCHAR2(54) := '(((0[1-9]|[12]\d|3[01])(0[1-9]|1[012])(\d{2}))(\d{5}))';
v_fnr                       VARCHAR2(11);

v_doc CLOB;

type rec_table2             is record (
    mtr_id                      table2.mtr_id%type,
    fnr                         table2.fnr%type
);
type arr_table2             is table of rec_table2 index by simple_integer;
table2_bulk                 arr_table2;
table2_row                  rec_table2;
BEGIN
FOR rec IN (
    select doc, MTR_ID as mtrid
    from TABLE
    where DOC is not null
) LOOP
    v_doc := rec.doc;
    loop
        v_fnr := REGEXP_SUBSTR(v_doc, v_regex_fnr, 1, 1, 'm');
        exit when v_fnr is null;
        v_vedlegg := REGEXP_REPLACE(v_doc, v_fnr , '' , 1 , 0); -- Incrementally remove all occurences of match from doc --
        IF CHECK_FNR(v_fnr) = 'TRUE' THEN
            table2_row.mtr_id := rec.mtrid;
            table2_row.fnr := v_fnr;
            table2_bulk(table2_bulk.count+1) := table2_row;
        END IF;
    END LOOP;
END LOOP;
forall i in indices of table2_bulk
      MERGE INTO TABLE2 T
      USING (SELECT table2_bulk(i).mtr_id mtrid, table2_bulk(i).fnr fnr FROM DUAL) B
      ON (T.MTR_ID = B.mtrid AND T.FNR = B.fnr)
      WHEN NOT MATCHED THEN INSERT (T.MTR_ID, T.FNR)
      VALUES (B.mtrid, B.fnr);

COMMIT;
EXCEPTION
WHEN OTHERS THEN
    DBMS_OUTPUT.PUT_LINE('Error - rollback');
    DBMS_OUTPUT.PUT_LINE('The error code is ' || SQLCODE || '- ' || SQLERRM);
    ROLLBACK;
END;
END;
/
Run Code Online (Sandbox Code Playgroud)

nop*_*svk 5

迭代调整PL/SQL块...

迭代0:修复语法错误......

CREATE OR REPLACE PROCEDURE SP_MTV_FINN_FNR AS
  v_n NUMBER;
  v_cnt NUMBER;
  v_mtrid NUMBER;
  v_regex_fnr VARCHAR2(54) := '(((0[1-9]|[12]\d|3[01])(0[1-9]|1[012])(\d{2}))(\d{5}))';
  v_doc CLOB;
  v_fnr VARCHAR2(11);

  BEGIN
    -- Get all rows from table --
    SELECT COUNT(*) INTO v_n FROM TABLE;
    IF v_n > 0 THEN
      -- Loop over all rows --
      FOR i IN 1..v_n LOOP
        SELECT doc, mtrid
          INTO v_doc, v_mtrid
          FROM (SELECT DOC doc, ID mtrid, ROWNUM rnum
            FROM TABLE
            WHERE ROWNUM <=i)
          WHERE rnum >= i;
        IF v_doc IS NOT NULL THEN
          SELECT REGEXP_COUNT(v_doc, v_regex_fnr) INTO v_cnt FROM DUAL;
          IF v_cnt >= 1 THEN
            -- For each regex match - time consuming, right? --
            FOR j IN 1..v_cnt LOOP
              SELECT REGEXP_SUBSTR(v_doc, v_regex_fnr, 1, j, 'm') INTO v_fnr FROM DUAL;
                IF CHECK_FNR(v_fnr) = 'TRUE' THEN
                  INSERT INTO TABLE2(MTR_ID, FNR)
                    SELECT v_mtrid, v_fnr FROM DUAL;
                END IF;
            END LOOP;
          END IF;
        END IF;
        COMMIT;
      END LOOP;
    END IF;
EXCEPTION WHEN OTHERS THEN
  DBMS_OUTPUT.PUT_LINE('Error - rollback');
  DBMS_OUTPUT.PUT_LINE('The error code is ' || SQLCODE || '- ' || SQLERRM);
  ROLLBACK;
END;
Run Code Online (Sandbox Code Playgroud)

迭代1:删除不必要的上下文切换和无用的行计数......

DECLARE 
    v_cnt NUMBER;
    v_regex_fnr VARCHAR2(54) := '(((0[1-9]|[12]\d|3[01])(0[1-9]|1[012])(\d{2}))(\d{5}))';
    v_fnr VARCHAR2(11);
BEGIN
    FOR rec IN (
        select doc, id as mtrid
        from table
    ) LOOP
        IF rec.doc IS NOT NULL THEN
            v_cnt := REGEXP_COUNT(rec.doc, v_regex_fnr);

            IF v_cnt >= 1 THEN
                -- For each regex match - time consuming, right? --
                FOR j IN 1..v_cnt LOOP
                    v_fnr := REGEXP_SUBSTR(rec.doc, v_regex_fnr, 1, j, 'm');

                    IF CHECK_FNR(v_fnr) = 'TRUE' THEN
                        INSERT INTO TABLE2(MTR_ID, FNR) values (rec.mtrid, v_fnr);
                    END IF;
                END LOOP;
            END IF;
        END IF;
    END LOOP;
    COMMIT;
EXCEPTION
    WHEN OTHERS THEN
        DBMS_OUTPUT.PUT_LINE('Error - rollback');
        DBMS_OUTPUT.PUT_LINE('The error code is ' || SQLCODE || '- ' || SQLERRM);
        ROLLBACK;
END;
/
Run Code Online (Sandbox Code Playgroud)

迭代2:减少外环的数量......

DECLARE 
    v_cnt NUMBER;
    v_regex_fnr VARCHAR2(54) := '(((0[1-9]|[12]\d|3[01])(0[1-9]|1[012])(\d{2}))(\d{5}))';
    v_fnr VARCHAR2(11);
BEGIN
    FOR rec IN (
        select doc, id as mtrid
        from table
        where doc is not null
    ) LOOP
        v_cnt := REGEXP_COUNT(rec.doc, v_regex_fnr);

        IF v_cnt >= 1 THEN
            -- For each regex match - time consuming, right? --
            FOR j IN 1..v_cnt LOOP
                v_fnr := REGEXP_SUBSTR(rec.doc, v_regex_fnr, 1, j, 'm');

                IF CHECK_FNR(v_fnr) = 'TRUE' THEN
                    INSERT INTO TABLE2(MTR_ID, FNR) values (rec.mtrid, v_fnr);
                END IF;
            END LOOP;
        END IF;
    END LOOP;
    COMMIT;
EXCEPTION
    WHEN OTHERS THEN
        DBMS_OUTPUT.PUT_LINE('Error - rollback');
        DBMS_OUTPUT.PUT_LINE('The error code is ' || SQLCODE || '- ' || SQLERRM);
        ROLLBACK;
END;
/
Run Code Online (Sandbox Code Playgroud)

迭代3:缩短迭代代码2 ...

DECLARE 
    v_regex_fnr VARCHAR2(54) := '(((0[1-9]|[12]\d|3[01])(0[1-9]|1[012])(\d{2}))(\d{5}))';
    v_fnr VARCHAR2(11);
BEGIN
    FOR rec IN (
        select doc, id as mtrid, REGEXP_COUNT(rec.doc, v_regex_fnr) as regexp_cnt
        from table
        where doc is not null
            and regexp_like(doc, v_regex_fnt)
    ) LOOP
        FOR j IN 1..rec.regexp_cnt LOOP
            v_fnr := REGEXP_SUBSTR(rec.doc, v_regex_fnr, 1, j, 'm');

            IF CHECK_FNR(v_fnr) = 'TRUE' THEN
                INSERT INTO TABLE2(MTR_ID, FNR) values (rec.mtrid, v_fnr);
            END IF;
        END LOOP;
    END LOOP;
    COMMIT;
EXCEPTION
    WHEN OTHERS THEN
        DBMS_OUTPUT.PUT_LINE('Error - rollback');
        DBMS_OUTPUT.PUT_LINE('The error code is ' || SQLCODE || '- ' || SQLERRM);
        ROLLBACK;
END;
/
Run Code Online (Sandbox Code Playgroud)

迭代4:删除不必要的regexp_count()计数......

DECLARE 
    v_regex_fnr VARCHAR2(54) := '(((0[1-9]|[12]\d|3[01])(0[1-9]|1[012])(\d{2}))(\d{5}))';
    v_fnr VARCHAR2(11);
    j integer;
BEGIN
    FOR rec IN (
        select doc, id as mtrid
        from table
        where doc is not null
    ) LOOP
        j := 1;
        loop
            v_fnr := REGEXP_SUBSTR(rec.doc, v_regex_fnr, 1, j, 'm');
            exit when v_fnt is null;

            IF CHECK_FNR(v_fnr) = 'TRUE' THEN
                INSERT INTO TABLE2(MTR_ID, FNR) values (rec.mtrid, v_fnr);
            END IF;

            j := j + 1;
        END LOOP;
    END LOOP;
    COMMIT;
EXCEPTION
    WHEN OTHERS THEN
        DBMS_OUTPUT.PUT_LINE('Error - rollback');
        DBMS_OUTPUT.PUT_LINE('The error code is ' || SQLCODE || '- ' || SQLERRM);
        ROLLBACK;
END;
/
Run Code Online (Sandbox Code Playgroud)

迭代5:将结果保存到内存并立即将其刷新到DB(使用集合绑定),再加上处理不同的要求......

create or replace type obj_table2
as
object (
    mtr_id                      integer,
    fnr                         varchar2(4000)
);
/
create or replace type arr_table2
as
table of obj_table2;
/

DECLARE 
    v_regex_fnr                 VARCHAR2(54) := '(((0[1-9]|[12]\d|3[01])(0[1-9]|1[012])(\d{2}))(\d{5}))';
    v_fnr                       VARCHAR2(11);
    j                           integer;

    table2_bulk                 arr_table2 := arr_table2();
BEGIN
    FOR rec IN (
        select doc, id as mtrid
        from table
        where doc is not null
    ) LOOP
        j := 1;
        loop
            v_fnr := REGEXP_SUBSTR(rec.doc, v_regex_fnr, 1, j, 'm');
            exit when v_fnt is null;

            IF CHECK_FNR(v_fnr) = 'TRUE' THEN
                table2_bulk.extend();
                table2_bulk(table2_bulk.last) := new obj_table2(
                    mtr_id => rec.mtrid,
                    fnr => v_fnr
                );
            END IF;

            j := j + 1;
        END LOOP;
    END LOOP;

    insert into table2(mtr_id, fnr)
    select mtr_id, fnr
    from table(table2_bulk) X
    minus
    select mtr_id, fnr
    from table2;

    COMMIT;
EXCEPTION
    WHEN OTHERS THEN
        DBMS_OUTPUT.PUT_LINE('Error - rollback');
        DBMS_OUTPUT.PUT_LINE('The error code is ' || SQLCODE || '- ' || SQLERRM);
        ROLLBACK;
END;
/
Run Code Online (Sandbox Code Playgroud)

迭代6:把它扔掉,同时决定炫耀 ......

insert into table2 (mtr_id, fnr)
with xyz (doc, mtrid, fnr, j) as (
    select doc, id as mtrid, cast(null as varchar2(4000)) as fnr, 0 as j
    from table A
    where doc is not null
    --
    union all
    --
    select doc, mtrid,
        regexp_substr(doc, '(((0[1-9]|[12]\d|3[01])(0[1-9]|1[012])(\d{2}))(\d{5}))', 1, j+1, 'm') as fnr,
        j+1
    from xyz X
    where j = 0
        or j > 0 and X.fnr is not null
)
select distinct mtrid, fnr
from xyz
where j > 0
    and fnr is not null
    and CHECK_FNR(fnr) = 'TRUE'
;
commit;
Run Code Online (Sandbox Code Playgroud)

请注意,这些代码段甚至可能无法正常工作.由于您未向我们提供任何测试数据设置,因此我们只能以假设的方式调整您的代码.

请注意,最慢的部分仍然regexp_substr()超过一个CLOB值.您可能想要考虑使用position参数regexp_substr()而不是occurence参数来获取后续的正则表达式匹配.

请享用.

归档时间：

查看次数：

1732 次

最近记录：

11 年，11 月 前
相关归档
regex_match和regex_search之间的区别？ 27
ORM解决方案(JPA; Hibernate)与JDBC 18
将数字拆分为总和组件 15
在忽略大小写中将字符串与正则表达式匹 13
如何检查哪个存储过程在sql server中占用了最多时间 12
AVX/SSE版xorshift128 + 10
我可以通过在对象中定位属性来提高查找速度吗？ 8
降序排序索引 7
如何在不指定变量来保存其OUT参数的情况下调用PL/SQL过程？ 3
为什么不能在表达式中组合'continue'关键字？ 2
难疑归档
如何在Linux上找到包含特定文本的所有文件？ 4914
使用Git将特定文件重置或还原到特定版本？ 4255
在Python中手动引发(抛出)异常 2079
ssh"权限太开放"错误 1859
什么是移动语义？ 1614
使用jQuery将表单数据转换为JavaScript对象 1580
C中的函数指针如何工作？ 1170
如何列出使用ATTACH打开的SQLite数据库文件中的表？ 1151
使用Twitter Bootstrap 3将列居中 1109
如何在Java中创建通用数组？ 1045