bigquery url解码

Question

有没有一种简单的方法在BigQuery查询语言中进行URL解码？我正在使用一个包含某些值中包含URL编码字符串的列的表.例如:

http://xyz.com/example.php?url=http%3A%2F%2Fwww.example.com%2Fhello%3Fv%3D12345&foo=bar&abc=xyz

我像这样提取"url"参数:

SELECT REGEXP_EXTRACT(column_name, "url=([^&]+)") as url 
from [mydataset.mytable]

这给了我:

http%3A%2F%2Fwww.example.com%2Fhello%3Fv%3D12345

我想做的是:

SELECT URL_DECODE(REGEXP_EXTRACT(column_name, "url=([^&]+)")) as url 
from [mydataset.mytable]

从而返回:

http://www.example.com/hello?v=12345

如果可能的话,我想避免使用多个REGEXP_REPLACE()语句(替换%20,%3A等...).

想法？

Answer 1

下面是在@sigpwned答案之上构建的，但是使用SQL UDF进行了稍微的重构和包装（没有限制，JS UDF使用起来很安全）

#standardSQL
CREATE TEMP FUNCTION URLDECODE(url STRING) AS ((
  SELECT SAFE_CONVERT_BYTES_TO_STRING(
    ARRAY_TO_STRING(ARRAY_AGG(
        IF(STARTS_WITH(y, '%'), FROM_HEX(SUBSTR(y, 2)), CAST(y AS BYTES)) ORDER BY i
      ), b''))
  FROM UNNEST(REGEXP_EXTRACT_ALL(url, r"%[0-9a-fA-F]{2}|[^%]+")) AS y WITH OFFSET AS i 
));
SELECT 
  column_name, 
  URLDECODE(REGEXP_EXTRACT(column_name, "url=([^&]+)")) AS url
FROM `project.dataset.table`

可以用下面的问题示例进行测试

#standardSQL
CREATE TEMP FUNCTION URLDECODE(url STRING) AS ((
  SELECT SAFE_CONVERT_BYTES_TO_STRING(
    ARRAY_TO_STRING(ARRAY_AGG(
        IF(STARTS_WITH(y, '%'), FROM_HEX(SUBSTR(y, 2)), CAST(y AS BYTES)) ORDER BY i
      ), b''))
  FROM UNNEST(REGEXP_EXTRACT_ALL(url, r"%[0-9a-fA-F]{2}|[^%]+")) AS y WITH OFFSET AS i 
));
WITH `project.dataset.table` AS (
  SELECT 'http://example.com/example.php?url=http%3A%2F%2Fwww.example.com%2Fhello%3Fv%3D12345&foo=bar&abc=xyz' column_name
)
SELECT 
  URLDECODE(REGEXP_EXTRACT(column_name, "url=([^&]+)")) AS url,
  column_name
FROM `project.dataset.table`    

结果

Row url                                     column_name  
1   http://www.example.com/hello?v=12345    http://example.com/example.php?url=http%3A%2F%2Fwww.example.com%2Fhello%3Fv%3D12345&foo=bar&abc=xyz   

使用进一步优化的SQL UDF更新

CREATE TEMP FUNCTION URLDECODE(url STRING) AS ((
  SELECT STRING_AGG(
    IF(REGEXP_CONTAINS(y, r'^%[0-9a-fA-F]{2}'), 
      SAFE_CONVERT_BYTES_TO_STRING(FROM_HEX(REPLACE(y, '%', ''))), y), '' 
    ORDER BY i
    )
  FROM UNNEST(REGEXP_EXTRACT_ALL(url, r"%[0-9a-fA-F]{2}(?:%[0-9a-fA-F]{2})*|[^%]+")) y
  WITH OFFSET AS i 
));

Answer 2

这是一个很好的功能请求，但目前没有内置的 BigQuery 功能提供 URL 解码。