如何将视频中的音频分段并转录成带时间戳的片段？

Question

我想根据每行语音的内容将视频脚本分割成章节。脚本将用于为每一章生成一系列开始和结束时间戳。这类似于 YouTube 现在“自动章节”视频的方式。

.srt 转录本示例：

...

70
00:02:53,640 --> 00:02:54,760
All right, coming in at number five,

71
00:02:54,760 --> 00:02:57,640
we have another habit that saves me around 15 minutes a day
...

我在使用 ChatGPT 时运气不佳，因为它发现很难按主题进行分段并准确地重新收集开始和结束时间戳。我现在正在探索是否还有其他选择可以做到这一点。

我知道一些 python 库可以实现基于时间序列的主题建模。我还阅读了有关文本平铺作为另一种选择的内容。有哪些选择可以实现这样的结果？

注意：上面的格式 (.srt) 不是必需的。这只是输入是带有开始和结束时间戳的文本内容列表的想法。

Answer 1

首先你可能需要安装这些

\n

! pip install -U mock pytube openai-whisper \n

\n

拉取 YouTube 视频并提取音频

\n

import re\nimport mock\n\nfrom pytube.cipher import get_throttling_function_code\n\ndef patched_throttling_plan(js: str):\n    """Patch throttling plan, from https://github.com/pytube/pytube/issues/1498"""\n    raw_code = get_throttling_function_code(js)\n\n    transform_start = r"try{"\n    plan_regex = re.compile(transform_start)\n    match = plan_regex.search(raw_code)\n\n    #transform_plan_raw = find_object_from_startpoint(raw_code, match.span()[1] - 1)\n    transform_plan_raw = js\n\n    # Steps are either c[x](c[y]) or c[x](c[y],c[z])\n    step_start = r"c\\[(\\d+)\\]\\(c\\[(\\d+)\\](,c(\\[(\\d+)\\]))?\\)"\n    step_regex = re.compile(step_start)\n    matches = step_regex.findall(transform_plan_raw)\n    transform_steps = []\n    for match in matches:\n        if match[4] != \'\':\n            transform_steps.append((match[0],match[1],match[4]))\n        else:\n            transform_steps.append((match[0],match[1]))\n\n    return transform_steps\n\n\nwith mock.patch(\'pytube.cipher.get_throttling_plan\', patched_throttling_plan):\n    from pytube import YouTube\n    url = \'https://www.youtube.com/watch?v=ZBVrPWwSlRM\'\n\n    video = YouTube(url)\n    audio = video.streams.filter(only_audio=True, file_extension=\'mp4\')[0]\n    audio.download(filename=\'team-rocket.mp4\')\n

\n

如果您想播放它以确保 Jupyter 中的音频正确：

\n

from IPython.display import Audio, display\n\ndisplay(Audio(\'team-rocket.mp4\', autoplay=True))\n

\n

将音频转录为文本

\n

import whisper\n\nmodel = whisper.load_model("base")\nresult = model.transcribe(\'team-rocket.mp4\')\nprint(result["text"])\n

\n

[出去]：

\n

Prepare for trouble! Make it double! Super tip the world from devastation! To unite all peoples within our nation! To denounce the evils of truth and love! To extend our reach to the stars above! Jesse! James! Team Rocket blast off at the speed of life! So then to now, we\'re prepared to fight! Be out! That\'s right! Let\'s...\n

\n

注意： ASR 的输出绝对不是完美的，但作为需要一些手动后期编辑的初稿非常有用。请不要将订阅者直接提交到您最喜欢的粉丝订阅者论坛，这肯定会让 QC 版主失败。

\n

但是时间戳呢？

\n

然后我们需要比粗略地提取音频然后进行自动语音识别（ASR）更复杂的技术

\n

这是一篇好文章：https://blog.searce.com/generate-srt-file-subtitles-using-google-clouds-speech-to-text-api-402b2f1da3bd

\n

而且我们在当今的人工智能时代也很幸运...... Tada： https: //github.com/linto-ai/whisper-timestamped

\n

! pip install -U git+https://github.com/linto-ai/whisper-timestamped\n

\n

然后在代码中：

\n

import whisper_timestamped as whisper\n\naudio = whisper.load_audio(\'team-rocket.mp4\')\nmodel = whisper.load_model("base")\n\nresult = whisper.transcribe(model, audio, language="en")\n\nimport json\nprint(json.dumps(result, indent = 2, ensure_ascii = False))\n

\n

[出去]：

\n

{\n  "text": " Prepare for trouble! Make it double! Super tip the world from devastation! To unite all peoples within our nation! To denounce the evils of truth and love! To extend our reach to the stars above! Jesse! James! Team Rocket blast off at the speed of life! So then to now, we\'re prepared to fight! Be out! That\'s right! Let\'s...",\n  "segments": [\n    {\n      "id": 0,\n      "seek": 0,\n      "start": 0.22,\n      "end": 1.66,\n      "text": " Prepare for trouble!",\n      "tokens": [\n        50364,\n        29689,\n        337,\n        5253,\n        0,\n        50464\n      ],\n      "temperature": 0.0,\n      "avg_logprob": -0.24259458803663067,\n      "compression_ratio": 1.5330188679245282,\n      "no_speech_prob": 0.7252825498580933,\n      "confidence": 0.873,\n      "words": [\n        {\n          "text": "Prepare",\n          "start": 0.22,\n          "end": 0.68,\n          "confidence": 0.726\n        },\n        {\n          "text": "for",\n          "start": 0.68,\n          "end": 1.1,\n          "confidence": 0.982\n        },\n        {\n          "text": "trouble!",\n          "start": 1.1,\n          "end": 1.66,\n          "confidence": 0.933\n        }\n      ]\n    },\n    {\n      "id": 1,\n      "seek": 0,\n      "start": 2.48,\n      "end": 3.72,\n      "text": " Make it double!",\n      "tokens": [\n        50464,\n        4387,\n        309,\n        3834,\n        0,\n        50564\n      ],\n      "temperature": 0.0,\n      "avg_logprob": -0.24259458803663067,\n      "compression_ratio": 1.5330188679245282,\n      "no_speech_prob": 0.7252825498580933,\n      "confidence": 0.878,\n      "words": [\n        {\n          "text": "Make",\n          "start": 2.48,\n          "end": 2.84,\n          "confidence": 0.979\n        },\n        {\n          "text": "it",\n          "start": 2.84,\n          "end": 3.12,\n          "confidence": 0.995\n        },\n        {\n          "text": "double!",\n          "start": 3.12,\n          "end": 3.72,\n          "confidence": 0.694\n        }\n      ]\n    },\n    {\n      "id": 2,\n      "seek": 0,\n      "start": 4.2,\n      "end": 6.06,\n      "text": " Super tip the world from devastation!",\n      "tokens": [\n        50564,\n        4548,\n        4125,\n        264,\n        1002,\n        490,\n        13959,\n        399,\n        0,\n        50664\n      ],\n      "temperature": 0.0,\n      "avg_logprob": -0.24259458803663067,\n      "compression_ratio": 1.5330188679245282,\n      "no_speech_prob": 0.7252825498580933,\n      "confidence": 0.709,\n      "words": [\n        {\n          "text": "Super",\n          "start": 4.2,\n          "end": 4.44,\n          "confidence": 0.728\n        },\n        {\n          "text": "tip",\n          "start": 4.44,\n          "end": 4.66,\n          "confidence": 0.195\n        },\n        {\n          "text": "the",\n          "start": 4.66,\n          "end": 4.86,\n          "confidence": 0.73\n        },\n        {\n          "text": "world",\n          "start": 4.86,\n          "end": 5.12,\n          "confidence": 0.913\n        },\n        {\n          "text": "from",\n          "start": 5.12,\n          "end": 5.36,\n          "confidence": 0.966\n        },\n        {\n          "text": "devastation!",\n          "start": 5.36,\n          "end": 6.06,\n          "confidence": 0.991\n        }\n      ]\n    },\n    {\n      "id": 3,\n      "seek": 0,\n      "start": 6.4,\n      "end": 8.78,\n      "text": " To unite all peoples within our nation!",\n      "tokens": [\n        50664,\n        1407,\n        29320,\n        439,\n        16915,\n        1951,\n        527,\n        4790,\n        0,\n        50814\n      ],\n      "temperature": 0.0,\n      "avg_logprob": -0.24259458803663067,\n      "compression_ratio": 1.5330188679245282,\n      "no_speech_prob": 0.7252825498580933,\n      "confidence": 0.943,\n      "words": [\n        {\n          "text": "To",\n          "start": 6.4,\n          "end": 6.52,\n          "confidence": 0.772\n        },\n        {\n          "text": "unite",\n          "start": 6.52,\n          "end": 6.88,\n          "confidence": 0.991\n        },\n        {\n          "text": "all",\n          "start": 6.88,\n          "end": 7.22,\n          "confidence": 0.992\n        },\n        {\n          "text": "peoples",\n          "start": 7.22,\n          "end": 7.64,\n          "confidence": 0.941\n        },\n        {\n          "text": "within",\n          "start": 7.64,\n          "end": 8.08,\n          "confidence": 0.97\n        },\n        {\n          "text": "our",\n          "start": 8.08,\n          "end": 8.28,\n          "confidence": 0.995\n        },\n        {\n          "text": "nation!",\n          "start": 8.28,\n          "end": 8.78,\n          "confidence": 0.96\n        }\n      ]\n    },\n    {\n      "id": 4,\n      "seek": 0,\n      "start": 9.26,\n      "end": 11.26,\n      "text": " To denounce the evils of truth and love!",\n      "tokens": [\n        50814,\n        1407,\n        1441,\n        7826,\n        264,\n        1073,\n        4174,\n        295,\n        3494,\n        293,\n        959,\n        0,\n        50914\n      ],\n      "temperature": 0.0,\n      "avg_logprob": -0.24259458803663067,\n      "compression_ratio": 1.5330188679245282,\n      "no_speech_prob": 0.7252825498580933,\n      "confidence": 0.883,\n      "words": [\n        {\n          "text": "To",\n          "start": 9.26,\n          "end": 9.44,\n          "confidence": 0.684\n        },\n        {\n          "text": "denounce",\n          "start": 9.44,\n          "end": 9.96,\n          "confidence": 0.739\n        },\n        {\n          "text": "the",\n          "start": 9.96,\n          "end": 10.14,\n          "confidence": 0.988\n        },\n        {\n          "text": "evils",\n          "start": 10.14,\n          "end": 10.44,\n          "confidence": 0.942\n        },\n        {\n          "text": "of",\n          "start": 10.44,\n          "end": 10.58,\n          "confidence": 0.995\n        },\n        {\n          "text": "truth",\n          "start": 10.58,\n          "end": 10.8,\n          "confidence": 0.906\n        },\n        {\n          "text": "and",\n          "start": 10.8,\n          "end": 11.02,\n          "confidence": 0.986\n        },\n        {\n          "text": "love!",\n          "start": 11.02,\n          "end": 11.26,\n          "confidence": 0.987\n        }\n      ]\n    },\n    {\n      "id": 5,\n      "seek": 0,\n      "start": 11.44,\n      "end": 13.22,\n      "text": " To extend our reach to the stars above!",\n      "tokens": [\n        50914,\n        1407,\n        10101,\n        527,\n        2524,\n        281,\n        264,\n        6105,\n        3673,\n        0,\n        51014\n      ],\n      "temperature": 0.0,\n      "avg_logprob": -0.24259458803663067,\n      "compression_ratio": 1.5330188679245282,\n      "no_speech_prob": 0.7252825498580933,\n      "confidence": 0.992,\n      "words": [\n        {\n          "text": "To",\n          "start": 11.44,\n          "end": 11.52,\n          "confidence": 0.993\n        },\n        {\n          "text": "extend",\n          "start": 11.52,\n          "end": 11.88,\n          "confidence": 0.996\n        },\n        {\n          "text": "our",\n          "start": 11.88,\n          "end": 12.1,\n          "confidence": 0.989\n        },\n        {\n          "text": "reach",\n          "start": 12.1,\n          "end": 12.32,\n          "confidence": 0.988\n        },\n        {\n          "text": "to",\n          "start": 12.32,\n          "end": 12.52,\n          "confidence": 0.997\n        },\n        {\n          "text": "the",\n          "start": 12.52,\n          "end": 12.62,\n          "confidence": 0.993\n        },\n        {\n          "text": "stars",\n          "start": 12.62,\n          "end": 12.86,\n          "confidence": 0.988\n        },\n        {\n          "text": "above!",\n          "start": 12.86,\n          "end": 13.22,\n          "confidence": 0.993\n        }\n      ]\n    },\n    {\n      "id": 6,\n      "seek": 0,\n      "start": 13.48,\n      "end": 14.16,\n      "text": " Jesse!",\n      "tokens": [\n        51014,\n        21895,\n        0,\n        51064\n      ],\n      "temperature": 0.0,\n      "avg_logprob": -0.24259458803663067,\n      "compression_ratio": 1.5330188679245282,\n      "no_speech_prob": 0.7252825498580933,\n      "confidence": 0.211,\n      "words": [\n        {\n          "text": "Jesse!",\n          "start": 13.48,\n          "end": 14.16,\n          "confidence": 0.211\n        }\n      ]\n    },\n    {\n      "id": 7,\n      "seek": 0,\n      "start": 14.48,\n      "end": 15.22,\n      "text": " James!",\n      "tokens": [\n        51064,\n        5678,\n        0,\n        51114\n      ],\n      "temperature": 0.0,\n      "avg_logprob": -0.24259458803663067,\n      "compression_ratio": 1.5330188679245282,\n      "no_speech_prob": 0.7252825498580933,\n      "confidence": 0.874,\n      "words": [\n        {\n          "text": "James!",\n          "start": 14.48,\n          "end": 15.22,\n          "confidence": 0.874\n        }\n      ]\n    },\n    {\n      "id": 8,\n      "seek": 0,\n      "start": 15.48,\n      "end": 18.42,\n      "text": " Team Rocket blast off at the speed of life!",\n      "tokens": [\n        51114,\n        7606,\n        29651,\n        12035,\n        766,\n        412,\n        264,\n        3073,\n        295,\n        993,\n        0,\n        51264\n      ],\n      "temperature": 0.0,\n      "avg_logprob": -0.24259458803663067,\n      "compression_ratio": 1.5330188679245282,\n      "no_speech_prob": 0.7252825498580933,\n      "confidence": 0.711,\n      "words": [\n        {\n          "text": "Team",\n          "start": 15.48,\n          "end": 16.22,\n          "confidence": 0.147\n        },\n        {\n          "text": "Rocket",\n          "start": 16.22,\n          "end": 16.66,\n          "confidence": 0.919\n        },\n        {\n          "text": "blast",\n          "start": 16.66,\n          "end": 17.06,\n          "confidence": 0.774\n        },\n        {\n          "text": "off",\n          "start": 17.06,\n          "end": 17.36,\n          "confidence": 0.722\n        },\n        {\n          "text": "at",\n          "start": 17.36,\n          "end": 17.5,\n          "confidence": 0.946\n        },\n        {\n          "text": "the",\n          "start": 17.5,\n          "end": 17.62,\n          "confidence": 0.993\n        },\n        {\n          "text": "speed",\n          "start": 17.62,\n          "end": 17.96,\n          "confidence": 0.983\n        },\n        {\n          "text": "of",\n          "start": 17.96,\n          "end": 18.2,\n          "confidence": 0.998\n        },\n        {\n          "text": "life!",\n          "start": 18.2,\n          "end": 18.42,\n          "confidence": 0.667\n        }\n      ]\n    },\n    {\n      "id": 9,\n      "seek": 0,\n      "start": 18.48,\n      "end": 20.96,\n      "text": " So then to now, we\'re prepared to fight!",\n      "tokens": [\n        51264,\n        407,\n        550,\n        281,\n        586,\n        11,\n        321,\n        434,\n        4927,\n        281,\n        2092,\n        0,\n        51414\n      ],\n      "temperature": 0.0,\n      "avg_logprob": -0.24259458803663067,\n      "compression_ratio": 1.5330188679245282,\n      "no_speech_prob": 0.7252825498580933,\n      "confidence": 0.79,\n      "words": [\n        {\n          "text": "So",\n          "start": 18.48,\n          "end": 19.0,\n          "confidence": 0.753\n        },\n        {\n          "text": "then",\n          "start": 19.0,\n          "end": 19.18,\n          "confidence": 0.564\n        },\n        {\n          "text": "to",\n          "start": 19.18,\n          "end": 19.34,\n          "confidence": 0.829\n        },\n        {\n          "text": "now,",\n          "start": 19.34,\n          "end": 19.66,\n          "confidence": 0.984\n        },\n        {\n          "text": "we\'re",\n          "start": 19.9,\n          "end": 20.14,\n          "confidence": 0.597\n        },\n        {\n          "text": "prepared",\n          "start": 20.14,\n          "end": 20.3,\n          "confidence": 0.973\n        },\n        {\n          "text": "to",\n          "start": 20.3,\n          "end": 20.56,\n          "confidence": 0.998\n        },\n        {\n          "text": "fight!",\n          "start": 20.56,\n          "end": 20.96,\n          "confidence": 0.996\n        }\n      ]\n    },\n    {\n      "id": 10,\n      "seek": 0,\n      "start": 21.1,\n      "end": 23.28,\n      "text": " Be out! That\'s right! Let\'s...",\n      "tokens": [\n        51414,\n        879,\n        484,\n        0,\n        663,\n        311,\n        558,\n        0,\n        961,\n        311,\n        485,\n        51514\n      ],\n      "temperature": 0.0,\n      "avg_logprob": -0.24259458803663067,\n      "compression_ratio": 1.5330188679245282,\n      "no_speech_prob": 0.7252825498580933,\n      "confidence": 0.795,\n      "words": [\n        {\n          "text": "Be",\n          "start": 21.1,\n          "end": 21.28,\n          "confidence": 0.462\n        },\n        {\n          "text": "out!",\n          "start": 21.28,\n          "end": 21.74,\n          "confidence": 0.807\n        },\n        {\n          "text": "That\'s",\n          "start": 22.02,\n          "end": 22.36,\n          "confidence": 0.958\n        },\n        {\n          "text": "right!",\n          "start": 22.36,\n          "end": 22.64,\n          "confidence": 0.992\n        },\n        {\n          "text": "Let\'s...",\n          "start": 22.86,\n          "end": 23.28,\n          "confidence": 0.769\n        }\n      ]\n    }\n  ],\n  "language": "en"\n}\n\n

\n

到目前为止，您已经了解了一些细粒度的 JSON 和时间戳，那么如何使用一些代码将 JSON 转换为 .srt 文件呢？

\n

for i, segment in enumerate(result[\'segments\']):\n  start, end = segment[\'start\'], segment[\'end\']\n  print(i)\n  print(f"00:00:{str(int(start)).replace(\'.\', \',\')} --> 00:00:{str(int(end)).replace(\'.\', \',\')}")\n  print(segment[\'text\'].strip())\n  print()\n\n

\n

[出去]：

\n

0\n00:00:0 --> 00:00:1\nPrepare for trouble!\n\n1\n00:00:2 --> 00:00:3\nMake it double!\n\n2\n00:00:4 --> 00:00:6\nSuper tip the world from devastation!\n\n3\n00:00:6 --> 00:00:8\nTo unite all peoples within our nation!\n\n4\n00:00:9 --> 00:00:11\nTo denounce the evils of truth and love!\n\n5\n00:00:11 --> 00:00:13\nTo extend our reach to the stars above!\n\n6\n00:00:13 --> 00:00:14\nJesse!\n\n7\n00:00:14 --> 00:00:15\nJames!\n\n8\n00:00:15 --> 00:00:18\nTeam Rocket blast off at the speed of life!\n\n9\n00:00:18 --> 00:00:20\nSo then to now, we\'re prepared to fight!\n\n10\n00:00:21 --> 00:00:23\nBe out! That\'s right! Let\'s...\n

\n

\xe5\xb0\x8f\xe7\xad\x89\xe5\x92\xa7！（等一下！），您已经对小时和分钟进行了硬编码...对于较长的视频我该怎么办？

\n

我确信通过 JSON 输出，whisper-timestamped您可以轻松计算出转换结果。暗示：from datetime import timedelta; str(timedelta(seconds=float(start)))

\n

将数据修改为您需要的格式，享受乐趣！

\n