从 BigQuery 中删除停用词？

Question

我有一大堆来自 reddit 的评论。这些字符串被分割成单词，删除标点符号，并进行量化以显示特定 subreddit 上最常用的单词：

SELECT word, COUNT(*) as num_words
FROM(FLATTEN((
  SELECT SPLIT(LOWER(REGEXP_REPLACE(body, r'[\.\",*:()\[\]/|\n]', ' ')), ' ') word
  FROM [fh-bigquery:reddit_comments.2017_08]
  WHERE subreddit="The_Donald"
  ), word))
GROUP EACH BY word
HAVING num_words >= 1000
ORDER BY num_words DESC

我有一个要删除的停用词列表，我该如何将其添加到代码中？谢谢 ：）

Answer 1

下面的示例适用于 BigQuery Legacy SQL（正如您的问题中所示）

#legacydSQL
SELECT word, COUNT(*) AS num_words
FROM(FLATTEN((
  SELECT SPLIT(LOWER(REGEXP_REPLACE(body, r'[\.\",*:()\[\]/|\n]', ' ')), ' ') word
  FROM [fh-bigquery:reddit_comments.2017_08]
  WHERE subreddit="The_Donald"
  ), word))
WHERE NOT word IN (
  'the','to','a','and'
)
GROUP EACH BY word
HAVING num_words >= 1000
ORDER BY num_words DESC 

BigQuery 团队强烈建议使用标准 SQL
因此，如果您决定这样做migrate - 下面是标准 SQL 中的示例
它假设您在your_project.your_dataset.stop_words表中有停用词

#standardSQL
SELECT word, COUNT(*) AS num_words
FROM `fh-bigquery.reddit_comments.2017_08`,
UNNEST(SPLIT(LOWER(REGEXP_REPLACE(body, r'[\.\",*:()\[\]/|\n]', ' ')), ' ')) word
WHERE subreddit="The_Donald"
AND word NOT IN (SELECT stop_word FROM `your_project.your_dataset.stop_words`)
GROUP BY word
HAVING num_words >= 1000
AND word != ''
ORDER BY num_words DESC  

您可以在此处测试/使用以下虚拟数据

#standardSQL
WITH `your_project.your_dataset.stop_words` AS (
  SELECT stop_word 
  FROM UNNEST(['the','to','a','and']) stop_word
)
SELECT word, COUNT(*) AS num_words
FROM `fh-bigquery.reddit_comments.2017_08`,
UNNEST(SPLIT(LOWER(REGEXP_REPLACE(body, r'[\.\",*:()\[\]/|\n]', ' ')), ' ')) word
WHERE subreddit="The_Donald"
AND word NOT IN (SELECT stop_word FROM `your_project.your_dataset.stop_words`)
GROUP BY word
HAVING num_words >= 1000
AND word != ''
ORDER BY num_words DESC