如何用Python和lxml抓取这个网页？空列表返回

Question

出于教育目的,我试图用Python和lxml逐步抓取这个页面,从电影名称开始.

从我到目前为止读到的关于lxml上的Python文档和XPath上的W3Schools来看,这段代码应该会让我看到列表中的所有电影片段:

from lxml import html
import requests

page = requests.get('http://www.rottentomatoes.com/browse/dvd-top-rentals/')
tree = html.fromstring(page.text)

movies = tree.xpath('//h3[@class="movieTitle"]/text()')

print movies

基本上,它应该给我文件中任何class具有值为"movieTitle" 的属性的每个h3元素.在运行代码时,我只打印出一个空列表.

我无法弄清楚为什么.

我自己试过,所以我跑了:

movies = tree.xpath('//h3[@class]/text()')

print movies

那么这个应该返回任何具有属性类的H3,但它返回此列表:

['From RT Users Like You!', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ', ' ']

我尝试通过定位其类值("noSpacing center")来定位此列表中的第一个字符串,并成功返回此唯一字符串.所以我确信有一些我误解了lxml/XPath的工作原理.有人能指出我有用的方向吗？提前致谢!

Answer 1

http://www.rottentomatoes.com/browse/dvd-top-rentals/上的信息不会直接呈现到页面中，而是从XMLHttpRequests加载。

\n\n

您正在寻找的 API 似乎是：

\n\n

http://d3biamo577v4eu.cloudfront.net/api/private/v1.0/m/list/find?page=1&limit=30&type=dvd-top-rentals&services=amazon%3Bamazon_prime%3Bflixster%3Bhbo_go%3Bitunes%3Bnetflix_iw%3Bvudu&sortBy=人气

\n\n

查询字符串是根据所选过滤器准备的。

\n\n

因此，您必须向该端点（而不是您当前请求的 URL）发出请求并解析 JSON 以提取所需的数据。

\n\n

您应该使用“page”GET 变量来获取下一个。

\n\n

cURL + jq的示例：

\n\n

\xe2\x9e\x9c  ~  curl -s http://d3biamo577v4eu.cloudfront.net/api/private/v1.0/m/list/find\\?page\\=1\\&limit\\=30\\&type\\=dvd-top-rentals\\&services\\=amazon%3Bamazon_prime%3Bflixster%3Bhbo_go%3Bitunes%3Bnetflix_iw%3Bvudu\\&sortBy\\=popularity | jq \'.results[].title\'\n"Inside Out"\n"Vacation"\n"The End Of The Tour"\n"She\'s Funny That Way"\n"Best Of Enemies"\n"Before We Go"\n"Pixels"\n"The Gift"\n"Southpaw"\n"Max"\n"Jurassic World"\n"San Andreas"\n"Paper Towns"\n"Dragon Ball Z: Resurrection \'F\'"\n"Testament Of Youth"\n"The Wolfpack"\n"Z For Zachariah"\n"Tomorrowland"\n"Terminator Genisys"\n"Dope"\n"The Gallows"\n"Magic Mike XXL"\n"Insidious: Chapter 3"\n"Me and Earl and the Dying Girl"\n"Particle Fever"\n"Dark Places"\n"What We Did on Our Holiday"\n"Avengers: Age of Ultron"\n"Spy"\n"Poltergeist"\n

\n\n

Python + 请求示例：

\n\n

import json\nimport requests\n\nURL = "http://d3biamo577v4eu.cloudfront.net/api/private/v1.0/m/list/find?page=2&limit=30&type=dvd-top-rentals&services=amazon%3Bamazon_prime%3Bflixster%3Bhbo_go%3Bitunes%3Bnetflix_iw%3Bvudu&sortBy=popularity"\n\nresponse = requests.get(URL)\n\nfor movie in response.json()[\'results\']:\n    movie_id = movie.get(\'id\', None)\n    title = movie.get(\'title\', None)\n    synopsis = movie.get(\'synopsis\', None).encode(\'utf-8\')\n    actors = \', \'.join(movie.get(\'actors\', None)).encode(\'utf-8\')\n    tomato_score = movie.get(\'tomatoScore\', None)\n    popcorn_score = movie.get(\'popcornScore\', None)\n    mpaaRating = movie.get(\'mpaaRating\', None)\n    runtime = movie.get(\'runtime\', None)\n\n    print "Id: {}".format(movie_id)\n    print "Title: {}".format(title)\n    print "Synopsis: {}".format(synopsis)\n    print "Actors: {}".format(actors)\n    print "Rating: {}".format(mpaaRating)\n    print "Runtime: {}".format(runtime)\n    print "Scoring:"\n    print "Tomato score: {}".format(tomato_score)\n    print "PopCorn score: {}".format(popcorn_score)\n    print ""\n

\n\n

输出：

\n\n

Id: 771306118\nTitle: Inside Out\nSynopsis: Inventive, gorgeously animated, and powerfully moving, Inside Out is another outstanding addition to the Pixar library of modern animated classics.\nActors: Amy Poehler, Bill Hader, Lewis Black\nRating: PG\nRuntime: 1 hr. 34 min.\nScoring:\nTomato score: 98\nPopCorn score: 90\n\nId: 771312080\nTitle: Vacation\nSynopsis: Borrowing a basic storyline from the film that inspired it but forgetting the charm, wit, and heart, Vacation is yet another nostalgia-driven retread that misses the mark.\nActors: Ed Helms, Christina Applegate, Leslie Mann\nRating: R\nRuntime: 1 hr. 39 min.\nScoring:\nTomato score: 26\nPopCorn score: 53\n\nId: 771411936\nTitle: The End Of The Tour\nSynopsis: Brilliantly performed and smartly unconventional, The End of the Tour pays fitting tribute to a singular talent while offering profoundly poignant observations on the human condition.\nActors: Jesse Eisenberg, Jason Segel, Anna Chlumsky\nRating: R\nRuntime: 1 hr. 45 min.\nScoring:\nTomato score: 92\nPopCorn score: 88\n\nId: 771402562\nTitle: She\'s Funny That Way\nSynopsis: She\'s Funny That Way is an affectionate, talent-filled throwback to screwball comedies of old -- which makes it even more frustrating that the laughs are disappointingly few and far between.\nActors: Imogen Poots, Owen Wilson, Jennifer Aniston\nRating: R\nRuntime: 1 hr. 33 min.\nScoring:\nTomato score: 38\nPopCorn score: 38\n\nId: 771412027\nTitle: Best Of Enemies\nSynopsis: Smart, fascinating, and funny, Best of Enemies takes a penetrating -- and wildly entertaining -- look back at the dawn of pundit politics.\nActors: Dick Cavett, Gore Vidal, John Lithgow\nRating: R\nRuntime: 1 hr. 27 min.\nScoring:\nTomato score: 94\nPopCorn score: 94\n\nId: 771387770\nTitle: Before We Go\nSynopsis: BEFORE WE GO, the directorial debut of Chris Evans, follows the journey of two strangers stuck in New York City for the night. Starting as convenient acquaintances, the two soon grow into each other\'s most trusted confidants when a night of unexpected adventure forces them to confront their fears and take control of their lives. (C) Radius-TWC\nActors: Alice Eve, Chris Evans, Daniel Spink\nRating: PG-13\nRuntime: 1 hr. 29 min.\nScoring:\nTomato score: 25\nPopCorn score: 64\n\nId: 771263974\nTitle: Pixels\nSynopsis: Much like the worst arcade games from the era that inspired it, Pixels has little replay value and is hardly worth a quarter.\nActors: Adam Sandler, Peter Dinklage, Kevin James\nRating: PG-13\nRuntime: 1 hr. 46 min.\nScoring:\nTomato score: 17\nPopCorn score: 51\n\nId: 771415974\nTitle: The Gift\nSynopsis: The Gift is wickedly smart and playfully subversive, challenging the audience\'s expectations while leaving them leaning on the edges of their seats.\nActors: Jason Bateman, Rebecca Hall, Joel Edgerton\nRating: R\nRuntime: 1 hr. 48 min.\nScoring:\nTomato score: 93\nPopCorn score: 78\n\nId: 771258788\nTitle: Southpaw\nSynopsis: Jake Gyllenhaal delivers an impressively committed performance, but Southpaw beats it down with a dispiriting drama that pummels viewers with genre clich\xc3\xa9s.\nActors: Jake Gyllenhaal, Forest Whitaker, Rachel McAdams\nRating: R\nRuntime: 2 hr. 3 min.\nScoring:\nTomato score: 59\nPopCorn score: 79\n\nId: 771379965\nTitle: Max\nSynopsis: Max has good intentions and tries to hearken back to classic family-friendly features, but its disjointed, manipulative plot overwhelms the efforts of its talented human and canine stars.\nActors: Josh Wiggins, Lauren Graham, Thomas Haden Church\nRating: PG\nRuntime: 1 hr. 51 min.\nScoring:\nTomato score: 35\nPopCorn score: 73\n\nId: 771324839\nTitle: Jurassic World\nSynopsis: Jurassic World can\'t match the original for sheer inventiveness and impact, but it works in its own right as an entertaining -- and visually dazzling -- popcorn thriller.\nActors: Chris Pratt, Bryce Dallas Howard, Vincent D\'Onofrio\nRating: PG-13\nRuntime: 2 hr. 3 min.\nScoring:\nTomato score: 71\nPopCorn score: 80\n\nId: 771374432\nTitle: San Andreas\nSynopsis: San Andreas has a great cast and outstanding special effects, but amidst all the senses-shattering destruction, the movie\'s characters and plot prove less than structurally sound.\nActors: Dwayne "The Rock" Johnson, Carla Gugino, Alexandra Daddario\nRating: PG-13\nRuntime: 1 hr. 54 min.\nScoring:\nTomato score: 50\nPopCorn score: 56\n\nId: 771385882\nTitle: Paper Towns\nSynopsis: Paper Towns isn\'t as deep or moving as it wants to be, yet it\'s still earnest, well-acted, and thoughtful enough to earn a place in the hearts of teen filmgoers of all ages.\nActors: Nat Wolff, Cara Delevingne, Halston Sage\nRating: PG-13\nRuntime: 1 hr. 49 min.\nScoring:\nTomato score: 55\nPopCorn score: 53\n\nId: 771419940\nTitle: Dragon Ball Z: Resurrection \'F\'\nSynopsis: Even the complete obliteration of his physical form can\'t stop the galaxy\'s most evil overlord. After years in spiritual purgatory, Frieza has been resurrected and plans to take his revenge on the Z-Fighters of Earth. Facing off against Frieza\'s powerful new form, and his army of 1,000 soldiers, Goku and Vegeta must reach new levels of strength in order to protect Earth from their vengeful nemesis.\nActors: Koichi Yamadera, Todd Haberkorn, Sean Schemmel\nRating: Unrated\nRuntime: 1 hr. 33 min.\nScoring:\nTomato score: 75\nPopCorn score: 86\n\nId: 771385848\nTitle: Testament Of Youth\nSynopsis: Testament of Youth is well-acted and beautifully filmed, adding up to an enriching if not adventurous experience for fans of British period dramas.\nActors: Alicia Vikander, Kit Harington, Dominic West\nRating: PG-13\nRuntime: 2 hr. 9 min.\nScoring:\nTomato score: 81\nPopCorn score: 79\n\nId: 771412119\nTitle: The Wolfpack\nSynopsis: Offering a unique look at modern fears and our fascination with film, The Wolfpack is a fascinating -- and ultimately haunting -- urban fable.\nActors: Visnu Angulo, Susanne Angulo, Oscar Angulo\nRating: R\nRuntime: 1 hr. 20 min.\nScoring:\nTomato score: 84\nPopCorn score: 73\n\nId: 771408592\nTitle: Z For Zachariah\nSynopsis: Z for Zachariah wrings compelling drama out of its simplistic premise -- albeit at a pace that may test the patience of less contemplative viewers.\nActors: Chiwetel Ejiofor, Chris Pine, Margot Robbie\nRating: PG-13\nRuntime: 1 hr. 35 min.\nScoring:\nTomato score: 77\nPopCorn score: 47\n\nId: 771306778\nTitle: Tomorrowland\nSynopsis: Ambitious and visually stunning, Tomorrowland is unfortunately weighted down by uneven storytelling.\nActors: George Clooney, Hugh Laurie, Brittany Robertson\nRating: PG\nRuntime: 1 hr. 47 min.\nScoring:\nTomato score: 50\nPopCorn score: 52\n\nId: 771359910\nTitle: Terminator Genisys\nSynopsis: Mired in its muddled mythology, Terminator: Genisys is a lurching retread that lacks the thematic depth, conceptual intelligence, or visual thrills that launched this once-mighty franchise.\nActors: Emilia Clarke, Jason Clarke, Jai Courtney\nRating: PG-13\nRuntime: 1 hr. 59 min.\nScoring:\nTomato score: 26\nPopCorn score: 59\n\nId: 771412133\nTitle: Dope\nSynopsis: Featuring a starmaking performance from Shameik Moore and a refreshingly original point of view from writer-director Rick Famuyiwa, Dope is smart, insightful entertainment.\nActors: Shameik Moore, Kiersey Clemons, Tony Revolori\nRating: R\nRuntime: 1 hr. 55 min.\nScoring:\nTomato score: 87\nPopCorn score: 85\n\nId: 771417481\nTitle: The Gallows\nSynopsis: Narratively contrived and visually a mess, The Gallows sends viewers on a shaky tumble to the bottom of the found-footage horror barrel.\nActors: Cassidy Gifford, Pfeifer Brown, Ryan Shoos\nRating: R\nRuntime: 1 hr. 27 min.\nScoring:\nTomato score: 16\nPopCorn score: 25\n\nId: 771378808\nTitle: Magic Mike XXL\nSynopsis: Magic Mike XXL has enough narrative thrust and beefy charm to deliver another helping of well-oiled entertainment, even if this sequel isn\'t quite as pleasurable as its predecessor.\nActors: Channing Tatum, Matt Bomer, Joe Manganiello\nRating: R\nRuntime: 1 hr. 55 min.\nScoring:\nTomato score: 62\nPopCorn score: 61\n\nId: 771375494\nTitle: Insidious: Chapter 3\nSynopsis: Insidious: Chapter 3 isn\'t as terrifying as the original, although it boasts surprising thematic depth and is enlivened by another fine performance from Lin Shaye.\nActors: Dermot Mulroney, Stefanie Scott, Angus Sampson\nRating: PG-13\nRuntime: 1 hr. 37 min.\nScoring:\nTomato score: 60\nPopCorn score: 54\n\nId: 771412075\nTitle: Me and Earl and the Dying Girl\nSynopsis: Beautifully scripted and perfectly cast, Me & Earl & the Dying Girl is a coming-of-age movie with uncommon charm and insight.\nActors: Thomas Mann, R.J. Cyler, Olivia Cooke\nRating: PG-13\nRuntime: 1 hr. 44 min.\nScoring:\nTomato score: 81\nPopCorn score: 88\n\nId: 771364355\nTitle: Particle Fever\nSynopsis: The concepts behind its heady subject matter may fly over the heads of most viewers, but Particle Fever presents it in such a way that even the least science-inclined viewers will find themselves enraptured.\nActors: Savas Dimopoulos, Nima Arkani-Hamed, Fabiola Gianotti\nRating: Unrated\nRuntime: 1 hr. 39 min.\nScoring:\nTomato score: 95\nPopCorn score: 84\n\nId: 771362649\nTitle: Dark Places\nSynopsis: 25 years after testifying against her brother as the person responsible for massacring her entire family, a haunted woman (Charlize Theron) is approached by a secret society that specializes in complex, unsolved cases. Nicholas Hoult, Corey Stoll, and Chloe Moretz co-star in this Mandalay Pictures thriller directed by Gilles Paquet-Brenner, and based on the novel by Gillian Flynn. ~ Jason Buchanan, Rovi\nActors: Charlize Theron, Nicholas Hoult, Corey Stoll\nRating: R\nRuntime: 1 hr. 54 min.\nScoring:\nTomato score: 26\nPopCorn score: 35\n\nId: 771357112\nTitle: What We Did on Our Holiday\nSynopsis: Witty and well-cast, What We Did on Our Holiday injects unlikely laughs into a story dealing with dark, difficult themes.\nActors: Rosamund Pike, David Tennant, Billy Connolly\nRating: PG-13\nRuntime: 1 hr. 35 min.\nScoring:\nTomato score: 73\nPopCorn score: 73\n\nId: 771313962\nTitle: Avengers: Age of Ultron\nSynopsis: Exuberant and eye-popping, Avengers: Age of Ultron serves as an overstuffed but mostly satisfying sequel, reuniting its predecessor\'s unwieldy cast with a few new additions and a worthy foe.\nActors: Robert Downey Jr., Chris Evans, Mark Ruffalo\nRating: PG-13\nRuntime: 2 hr. 21 min.\nScoring:\nTomato score: 74\nPopCorn score: 85\n\nId: 771361497\nTitle: Spy\nSynopsis: Simultaneously broad and progressive, Spy offers further proof that Melissa McCarthy and writer-director Paul Feig bring out the best in one another -- and delivers scores of belly laughs along the way.\nActors: Melissa McCarthy, Jason Statham, Rose Byrne\nRating: R\nRuntime: 1 hr. 57 min.\nScoring:\nTomato score: 93\nPopCorn score: 81\n\nId: 770799339\nTitle: Poltergeist\nSynopsis: Paying competent homage without adding anything of real value to the original Poltergeist, this remake proves just as ephemeral (but half as haunting) as its titular spirit.\nActors: Sam Rockwell, Rosemarie DeWitt, Kyle Catlett\nRating: PG-13\nRuntime: 1 hr. 33 min.\nScoring:\nTomato score: 31\nPopCorn score: 23\n\nId: 771370467\nTitle: Entourage\nSynopsis: Entourage retains many elements of the HBO series, but feels less like a film than a particularly shallow, cameo-studded extended episode of the show.\nActors: Jeremy Piven, Adrian Grenier, Kevin Dillon\nRating: R\nRuntime: 1 hr. 45 min.\nScoring:\nTomato score: 32\nPopCorn score: 64\n\nId: 771412037\nTitle: Cop Car\nSynopsis: Cop Car boasts a terrific premise and a grimly gripping opening act -- and for some viewers, that will be enough to compensate for the movie\'s uneven denouement.\nActors: Kevin Bacon, Shea Whigham, Camryn Manheim\nRating: R\nRuntime: 1 hr. 26 min.\nScoring:\nTomato score: 79\nPopCorn score: 52\n\nId: 771412114\nTitle: Unexpected\nSynopsis: Unexpected proves a thoughtful and well-acted -- if somewhat mild -- look at worthy, thought-provoking themes.\nActors: Anders Holm, Cobie Smulders, Gail Bean\nRating: R\nRuntime: 1 hr. 30 min.\nScoring:\nTomato score: 67\nPopCorn score: 49\n\nId: 771356696\nTitle: Pitch Perfect 2\nSynopsis: Pitch Perfect 2 sings in sweet comedic harmony, even if it doesn\'t hit quite as many high notes as its predecessor.\nActors: Anna Kendrick, Rebel Wilson, Brittany Snow\nRating: PG-13\nRuntime: 1 hr. 54 min.\nScoring:\nTomato score: 66\nPopCorn score: 67\n\nId: 771354922\nTitle: Furious 7\nSynopsis: Serving up a fresh round of over-the-top thrills while adding unexpected dramatic heft, Furious 7 keeps the franchise moving in more ways than one.\nActors: Vin Diesel, Paul Walker, Jason Statham\nRating: PG-13\nRuntime: 2 hr. 20 min.\nScoring:\nTomato score: 81\nPopCorn score: 84\n\nId: 771270966\nTitle: Cinderella\nSynopsis: Refreshingly traditional in a revisionist era, Kenneth Branagh\'s Cinderella proves Disney hasn\'t lost any of its old-fashioned magic.\nActors: Lily James, Cate Blanchett, Richard Madden\nRating: PG\nRuntime: 1 hr. 45 min.\nScoring:\nTomato score: 85\nPopCorn score: 79\n\nId: 771359745\nTitle: Love & Mercy\nSynopsis: As unconventional and unwieldy as the life and legacy it honors, Love & Mercy should prove moving for Brian Wilson fans while still satisfying neophytes.\nActors: Paul Dano, Elizabeth Banks, Brett Davern\nRating: PG-13\nRuntime: 2 hr.\nScoring:\nTomato score: 89\nPopCorn score: 86\n\nId: 771378525\nTitle: Monkey Kingdom\nSynopsis: Monkey Kingdom\'s breathtaking footage of primates in the wild is likely to please animal lovers of all ages.\nActors: Tina Fey\nRating: G\nRuntime: 1 hr. 25 min.\nScoring:\nTomato score: 94\nPopCorn score: 77\n\nId: 771412081\nTitle: The Overnight\nSynopsis: Witty and unpredictable, The Overnight benefits from writer-director Patrick Brice\'s sure-handed touch and strong performances from a talented cast.\nActors: Adam Scott, Taylor Schilling, Jason Schwartzman\nRating: R\nRuntime: 1 hr. 20 min.\nScoring:\nTomato score: 82\nPopCorn score: 60\n\nId: 771374337\nTitle: Diplomacy\nSynopsis: For filmgoers who value character development and smart dialogue over plot, Diplomacy yields rich, powerfully acted rewards.\nActors: Andr\xc3\xa9 Dussollier, Niels Arestrup, Robert Stadlober\nRating: Unrated\nRuntime: 1 hr. 25 min.\nScoring:\nTomato score: 93\nPopCorn score: 80\n\nId: 771373149\nTitle: The Age of Adaline\nSynopsis: The Age of Adaline ruminates on mortality less compellingly than similarly themed films, but is set apart by memorable performances from Blake Lively and Harrison Ford.\nActors: Blake Lively, Michiel Huisman, Harrison Ford\nRating: PG-13\nRuntime: 1 hr. 49 min.\nScoring:\nTomato score: 54\nPopCorn score: 67\n\nId: 771028170\nTitle: Mad Max: Fury Road\nSynopsis: With exhilarating action and a surprising amount of narrative heft, Mad Max: Fury Road brings George Miller\'s post-apocalyptic franchise roaring vigorously back to life.\nActors: Tom Hardy, Charlize Theron, Nicholas Hoult\nRating: R\nRuntime: 2 hr.\nScoring:\nTomato score: 97\nPopCorn score: 87\n\nId: 770683518\nTitle: I\'ll See You in My Dreams\nSynopsis: I\'ll See You in My Dreams would be worth watching even if Blythe Danner\'s central performance was all it had going for it, but this thoughtful drama satisfies on multiple levels.\nActors: Blythe Danner, Martin Starr, Sam Elliott\nRating: PG-13\nRuntime: 1 hr. 35 min.\nScoring:\nTomato score: 94\nPopCorn score: 70\n\nId: 771387966\nTitle: Good Kill\nSynopsis: Thought-provoking, timely, and anchored by a strong performance from Ethan Hawke, Good Kill is a modern war movie with a troubled conscience.\nActors: Ethan Hawke, January Jones, Zo\xc3\xab Kravitz\nRating: R\nRuntime: 1 hr. 43 min.\nScoring:\nTomato score: 76\nPopCorn score: 49\n\nId: 771377895\nTitle: Dior and I\nSynopsis: Dior and I will obviously appeal to fashion fans, but this beautifully tailored documentary may draw in even the least sartorially inclined.\nActors: Omar Berrada, Marion Cotillard, Anna Wintour\nRating: R\nRuntime: 1 hr. 30 min.\nScoring:\nTomato score: 82\nPopCorn score: 80\n\nId: 771387025\nTitle: Glen Campbell: I\'ll Be Me\nSynopsis: The heartrendingly honest Glen Campbell: I\'ll Be Me offers a window into Alzheimer\'s that should prove powerful viewing for Campbell fans and novices alike.\nActors: Bruce Springsteen, Bill Clinton, Paul McCartney\nRating: PG\nRuntime: 1 hr. 56 min.\nScoring:\nTomato score: 100\nPopCorn score: 91\n\nId: 771378194\nTitle: Boulevard\nSynopsis: Boulevard features a richly layered performance from Robin Williams, but that may be this dour drama\'s sole distinctive feature.\nActors: Robin Williams, Kathy Baker, Roberto Aguire\nRating: R\nRuntime: 1 hr. 28 min.\nScoring:\nTomato score: 51\nPopCorn score: 33\n\nId: 770804151\nTitle: About Elly\nSynopsis: About Elly offers viewers performances as powerful as its thought-provoking ideas, and adds another strong entry to Asghar Farhadi\'s impressive filmography.\nActors: Golshif