从Pinterest网址获取电路板上的所有图像

Question

这个问题听起来很简单,但并不像听起来那么简单.

什么是错的简要总结

例如,使用此板; http://pinterest.com/dodo/web-designui-and-mobile/

检查页面顶部的板本身(在div类中GridItems)的HTML会产生:

<div class="variableHeightLayout padItems GridItems Module centeredWithinWrapper" style="..">
    <!-- First div with a displayed board image -->
    <div class="item" style="top: 0px; left: 0px; visibility: visible;">..</div>
    ...
    <!-- Last div with a displayed board image -->
    <div class="item" style="top: 3343px; left: 1000px; visibility: visible;">..</div>
</div>

然而在页面底部,在激活无限滚动几次之后,我们将其作为HTML:

<div class="variableHeightLayout padItems GridItems Module centeredWithinWrapper" style="..">
    <!-- First div with a displayed board image -->
    <div class="item" style="top: 12431px; left: 750px; visibility: visible;">..</div>
    ...
    <!-- Last div with a displayed board image -->
    <div class="item" style="top: 19944px; left: 750px; visibility: visible;">..</div>
</div>

正如您所看到的,页面上方图像的某些容器已消失,并且在首次加载页面时并非所有图像容器都会加载.

---

我想做的事

我希望能够创建一个C#脚本(或者目前的任何服务器端语言),可以下载页面的完整HTML(即检索页面上的每个图像),然后从他们的URL下载图像.下载网页并使用适当的XPath很容易,但真正的挑战是为每个图像下载完整的HTML.

有没有办法可以模拟滚动到页面底部,还是有更简单的方法可以检索每个图像？我想Pinterest使用AJAX来改变HTML,有没有一种方法可以通过编程方式触发事件来接收所有的HTML？提前感谢您的建议和解决方案,并且如果您没有任何建议和解决方案,甚至可以阅读这个非常长的问题!

伪代码

using System;
using System.Net;
using HtmlAgilityPack;

private void Main() {
    string pinterestURL = "http://www.pinterest.com/...";
    string XPath = ".../img";

    HtmlDocument doc = new HtmlDocument();

    // Currently only downloads the first 25 images.
    doc.Load(strPinterestUrl);

    foreach(HtmlNode link in doc.DocumentElement.SelectNodes(strXPath))
    {
         image_links[] = link["src"];
         // Use image links
    }
}

Answer 1

好的，所以我认为这可能是您所需要的（经过一些修改）。

注意事项：

1. 这是 PHP，而不是 C#（但您说过您对任何服务器端语言感兴趣）。
2. 此代码挂钩（非官方）Pinterest 搜索端点。您需要更改 $data 和 $search_res 以反映您的任务的适当端点（例如 BoardFeedResouce）。注意：至少对于搜索，Pinterest 目前使用两个端点，一个用于初始页面加载，另一个用于无限滚动操作。每个都有自己的预期参数结构。
3. Pinterest 没有官方的公共 API，预计每当他们更改任何内容时都会出现这种情况，并且不会发出警告。
4. 您可能会发现 pinterestapi.co.uk 更容易实施并且适合您正在做的事情。
5. 我在类下面有一些演示/调试代码，一旦您获得所需的数据，这些代码就不应该存在，并且您可能想要更改默认的页面获取限制。

兴趣点：

1. 下划线_参数采用 JavaScript 格式的时间戳，即。类似于 Unix 时间，但添加了毫秒。它实际上并不用于分页。
2. 分页使用该bookmarks属性，因此您向不需要它的“新”端点发出第一个请求，然后bookmarks从结果中获取并在请求中使用它来获取下一个“页面”结果，bookmarks从获取这些结果后获取下一页，依此类推，直到用完结果或达到预设限制（或者达到服务器脚本执行时间的最大值）。我很想知道到底是什么bookmarks字段到底编码什么。我想，除了 PIN ID 或其他页面标记之外，还有一些有趣的秘密武器。
3. 我跳过 html，而是处理 JSON，因为它（对我来说）比使用 DOM 操作解决方案或一堆正则表达式更容易。

<?php

if(!class_exists('Skrivener_Pins')) {

  class Skrivener_Pins {

    /**
     * Constructor
     */
    public function __construct() {
    }

    /**
     * Pinterest search function. Uses Pinterest's "internal" page APIs, so likely to break if they change.
     * @author [@skrivener] Philip Tillsley
     * @param $search_str     The string used to search for matching pins.
     * @param $limit          Max number of pages to get, defaults to 2 to avoid excessively large queries. Use care when passing in a value.
     * @param $bookmarks_str  Used internally for recursive fetches.
     * @param $pages          Used internally to limit recursion.
     * @return array()        int['id'], obj['image'], str['pin_link'], str['orig_link'], bool['video_flag']
     * 
     * TODO:
        * 
        * 
     */
    public function get_tagged_pins($search_str, $limit = 1, $bookmarks_str = null, $page = 1) {

      // limit depth of recursion, ie. number of pages of 25 returned, otherwise we can hang on huge queries
      if( $page > $limit ) return false;

      // are we getting a next page of pins or not
      $next_page = false;
      if( isset($bookmarks_str) ) $next_page = true;

      // build url components
      if( !$next_page ) {

        // 1st time
        $search_res = 'BaseSearchResource'; // end point
        $path = '&module_path=' . urlencode('SearchInfoBar(query=' . $search_str . ', scope=boards)');
        $data = preg_replace("'[\n\r\s\t]'","",'{
          "options":{
            "scope":"pins",
            "show_scope_selector":true,
            "query":"' . $search_str . '"
          },
          "context":{
            "app_version":"2f83a7e"
          },
          "module":{
            "name":"SearchPage",
            "options":{
              "scope":"pins",
              "query":"' . $search_str . '"
            }
          },
          "append":false,
          "error_strategy":0
          }');
      } else {

        // this is a fetch for 'scrolling', what changes is the bookmarks reference, 
        // so pass the previous bookmarks value to this function and it is included
        // in query
        $search_res = 'SearchResource'; // different end point from 1st time search
        $path = '';
        $data = preg_replace("'[\n\r\s\t]'","",'{
          "options":{
            "query":"' . $search_str . '",
            "bookmarks":["' . $bookmarks_str . '"],
            "show_scope_selector":null,
            "scope":"pins"
          },
          "context":{
            "app_version":"2f83a7e"
          },
            "module":{
              "name":"GridItems",
            "options":{
              "scrollable":true,
              "show_grid_footer":true,
              "centered":true,
              "reflow_all":true,
              "virtualize":true,
              "item_options":{
                "show_pinner":true,
                "show_pinned_from":false,
                "show_board":true
              },
              "layout":"variable_height"
            }
          },
          "append":true,
          "error_strategy":2
        }');
      }
      $data = urlencode($data);
      $timestamp = time() * 1000; // unix time but in JS format (ie. has ms vs normal server time in secs), * 1000 to add ms (ie. 0ms)

      // build url
      $url = 'http://pinterest.com/resource/' . $search_res . '/get/?source_url=/search/pins/?q=' . $search_str
          . '&data=' . $data
          . $path
          . '&_=' . $timestamp;//'1378150472669';

      // setup curl
      $ch = curl_init();
      curl_setopt($ch, CURLOPT_URL, $url);
      curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
      curl_setopt($ch, CURLOPT_HTTPHEADER, array("X-Requested-With: XMLHttpRequest"));

      // get result
      $curl_result = curl_exec ($ch); // this echoes the output
      $curl_result = json_decode($curl_result);
      curl_close ($ch);

      // clear html to make var_dumps easier to see when debugging
      // $curl_result->module->html = '';

      // isolate the pin data, different end points have different data structures
      if(!$next_page) $pin_array = $curl_result->module->tree->children[1]->children[0]->children[0]->children;
      else $pin_array = $curl_result->module->tree->children;

      // map the pin data into desired format
      $pin_data_array = array();
      $bookmarks = null;
      if(is_array($pin_array)) {
        if(count($pin_array)) {

          foreach ($pin_array as $pin) {

            //setup data
            $image_id = $pin->options->pin_id;
            $image_data = ( isset($pin->data->images->originals) ) ? $pin->data->images->originals : $pin->data->images->orig;
            $pin_url = 'http://pinterest.com/pin/' . $image_id . '/';
            $original_url = $pin->data->link;
            $video = $pin->data->is_video;

            array_push($pin_data_array, array(
              "id"          => $image_id,
              "image"       => $image_data,
              "pin_link"    => $pin_url,
              "orig_link"   => $original_url,
              "video_flag"  => $video,
              ));
          }
          $bookmarks = reset($curl_result->module->tree->resource->options->bookmarks);

        } else {
          $pin_data_array = false;
        }
      }

      // recurse until we're done
      if( !($pin_data_array === false) && !is_null($bookmarks) ) {

        // more pins to get
        $more_pins = $this->get_tagged_pins($search_str, $limit, $bookmarks, ++$page);
        if( !($more_pins === false) ) $pin_data_array = array_merge($pin_data_array, $more_pins);
        return $pin_data_array;
      }

      // end of recursion
      return false;
    }

  } // end class Skrivener_Pins
} // end if



/**
 * Debug/Demo Code
 * delete or comment this section for production
 */

// output headers to control how the content displays
// header("Content-Type: application/json");
header("Content-Type: text/plain");
// header("Content-Type: text/html");

// define search term
// $tag = "vader";
$tag = "haemolytic";
// $tag = "qjkjgjerbjjkrekhjk";

if(class_exists('Skrivener_Pins')) {

  // instantiate the class
  $pin_handler = new Skrivener_Pins();

  // get pins, pinterest returns 25 per batch, function pages through this recursively, pass in limit to 
  // override default limit on number of pages to retrieve, avoid high limits (eg. limit of 20 * 25 pins/page = 500 pins to pull 
  // and 20 separate calls to Pinterest)
  $pins1 = $pin_handler->get_tagged_pins($tag, 2);

  // display the pins for demo purposes
  echo '<h1>Images on Pinterest mentioning "' . $tag . '"</h1>' . "\n";
  if( $pins1 != false ) {
    echo '<p><em>' . count($pins1) . ' images found.</em></p>' . "\n";
    skrivener_dump_images($pins1, 5);
  } else {
    echo '<p><em>No images found.</em></p>' . "\n";
  }
}

// demo function, dumps images in array to html img tags, can pass limit to only display part of array
function skrivener_dump_images($pin_array, $limit = false) {
  if(is_array($pin_array)) {
    if($limit) $pin_array = array_slice($pin_array, -($limit));
    foreach ($pin_array as $pin) {
      echo '<img src="' . $pin['image']->url . '" width="' . $pin['image']->width . '" height="' . $pin['image']->height . '" >' . "\n";
    }
  }
}

?>

如果您在使其适应您的特定终点时遇到问题，请告诉我。Apols 对于代码中的任何草率之处，它最初并没有投入生产。