如何使用 snscrape 抓取 Twitter 列表?

Ved*_*mle 2 python web-scraping

我必须使用此链接https://twitter.com/i/lists/777955586638053376抓取推文snscrape

到目前为止我已经尝试过使用这个:

from snscrape.modules.twitter import *
Run Code Online (Sandbox Code Playgroud)
search = TwitterListPostsScraper("Abgeordnetenhaus #agh").get_items()
for item in search:
    print(item)
Run Code Online (Sandbox Code Playgroud)
search = TwitterListPostsScraper("https://twitter.com/i/lists/777955586638053376").get_items()
for item in search:
    print(item)
Run Code Online (Sandbox Code Playgroud)
search = TwitterListPostsScraper("777955586638053376").get_items()
for item in search:
    print(item)
Run Code Online (Sandbox Code Playgroud)

每次,我都会遇到这个错误

Error retrieving https://api.twitter.com/2/search/adaptive.json?include_profile_interstitial_type=1&include_blocking=1&include_blocked_by=1&include_followed_by=1&include_want_retweets=1&include_mute_edge=1&include_can_dm=1&include_can_media_tag=1&include_ext_has_nft_avatar=1&skip_status=1&cards_platform=Web-12&include_cards=1&include_ext_alt_text=true&include_quote_count=true&include_reply_count=1&tweet_mode=extended&include_entities=true&include_user_entities=true&include_ext_media_color=true&include_ext_media_availability=true&include_ext_sensitive_media_warning=true&include_ext_trusted_friends_metadata=true&send_error_codes=true&simple_quoted_tweet=true&q=list%3Ahttps%3A%2F%2Ftwitter.com%2Fi%2Flists%2F777955586638053376&tweet_search_mode=live&count=20&query_source=spelling_expansion_revert_click&pc=1&spelling_corrections=1&ext=mediaStats%2ChighlightedLabel%2ChasNftAvatar%2CvoiceInfo%2Cenrichments%2CsuperFollowMetadata%2CunmentionInfo: 
non-200 status code
4 requests to https://api.twitter.com/2/search/adaptive.json?include_profile_interstitial_type=1&include_blocking=1&include_blocked_by=1&include_followed_by=1&include_want_retweets=1&include_mute_edge=1&include_can_dm=1&include_can_media_tag=1&include_ext_has_nft_avatar=1&skip_status=1&cards_platform=Web-12&include_cards=1&include_ext_alt_text=true&include_quote_count=true&include_reply_count=1&tweet_mode=extended&include_entities=true&include_user_entities=true&include_ext_media_color=true&include_ext_media_availability=true&include_ext_sensitive_media_warning=true&include_ext_trusted_friends_metadata=true&send_error_codes=true&simple_quoted_tweet=true&q=list%3Ahttps%3A%2F%2Ftwitter.com%2Fi%2Flists%2F777955586638053376&tweet_search_mode=live&count=20&query_source=spelling_expansion_revert_click&pc=1&spelling_corrections=1&ext=mediaStats%2ChighlightedLabel%2ChasNftAvatar%2CvoiceInfo%2Cenrichments%2CsuperFollowMetadata%2CunmentionInfo 
failed, giving up.
Run Code Online (Sandbox Code Playgroud)

我在这里还能做什么?

编辑:更新模块解决了问题

Fck*_*nda 5

我从项目的github中查看了代码,它应该与列表号一起使用:

\n
from snscrape.modules.twitter import *\n\nsearch = TwitterListPostsScraper("777955586638053376").get_items()\nfor item in search:\n    print({\n        "tweet": item.rawContent,\n        "url": item.url,\n    })\n
Run Code Online (Sandbox Code Playgroud)\n

它对我来说非常有效,结果如下:

\n
{\'tweet\': \'@hardysee \\U0001fae2\xe2\x98\xba\xef\xb8\x8f\', \'url\': \'https://twitter.com/kaddinsky/status/1620173624295718912\'}\n{\'tweet\': \'@N_Stansfield Nee geht ja schlie\xc3\x9flich um die direkte Zufahrt in die Tiefgarage ?\', \'url\': \'https://twitter.com/kaddinsky/status/1620173294589853702\'}\n{\'tweet\': \'@rainerschund Hab ich in den letzten Tagen viermal geschafft. Macht 60 Jahre mehr.\', \'url\': \'https://twitter.com/Dennis_Buchner/status/1620171154454646784\'}\n{\'tweet\': \'@LiffKiff Das ist sehr schade. Aber bitte schau dir genau an, was die Berliner LINKE vorhat & welche Positionen sie in der Frage bezieht. Abgesehen davon geht es bei der Berlin-Wahl ja um Berlin-Themen.\', \'url\': \'https://twitter.com/NiklasSchenker/status/1620170967132835843\'}\n...\n
Run Code Online (Sandbox Code Playgroud)\n

我还尝试使用命令行复制该行为,它也有效:

\n
snscrape twitter-list-posts 777955586638053376\nhttps://twitter.com/kaddinsky/status/1620173624295718912\nhttps://twitter.com/kaddinsky/status/1620173294589853702\nhttps://twitter.com/Dennis_Buchner/status/1620171154454646784\nhttps://twitter.com/NiklasSchenker/status/1620170967132835843\nhttps://twitter.com/kaddinsky/status/1620167590545956868\nhttps://twitter.com/Katina_Schubert/status/1620163364700962817\nhttps://twitter.com/kaddinsky/status/1620163262179610624\nhttps://twitter.com/klauslederer/status/1620162450489499649\nhttps://twitter.com/klauslederer/status/1620162398517891073\nhttps://twitter.com/klauslederer/status/1620162363470274560\nhttps://twitter.com/Tobias_Schulze/status/1620161108320608256\nhttps://twitter.com/Dennis_Buchner/status/1620159486278369280\nhttps://twitter.com/NiklasSchenker/status/1620159380875517952\nhttps://twitter.com/Dennis_Buchner/status/1620158807824543746\nhttps://twitter.com/FrankHansel/status/1620151106457473027\n
Run Code Online (Sandbox Code Playgroud)\n

但是,在代码中似乎您还可以使用用户名/列表来抓取列表:

\n
        subparser.add_argument(\'list\', type = snscrape.base.nonempty_string(\'list\'), help = \'A Twitter list ID or a string of the form "username/listname" (replace spaces with dashes)\')\n
Run Code Online (Sandbox Code Playgroud)\n

所以在这种情况下应该是snscrape twitter-list-posts wahl_beobachter/Abgeordnetenhaus-#agh,但是当我尝试这样做时,我收到了错误。

\n

如果您想测试其他抓取工具或扩展您的项目,您可以考虑使用Scrapy来爬取这些数据,并使用estela在云上运行爬虫

\n