那些遇到过的问题

如何检测虚假用户(crawlers)和cURL

Ken*_* Le 39 php curl spam-prevention

其他一些网站使用cURL和假http引用来复制我的网站内容.我们有没有办法检测cURL或不是真正的网络浏览器？

Ala*_*blo 99

没有神奇的解决方案可以避免自动抓取.人类可以做到的一切,机器人也可以做到.只有解决方案才能使工作变得更加困难,只有强大的技术极客才能尝试通过它们.

几年前我也遇到了麻烦,我的第一个建议是,如果你有时间,自己成为一名爬行者(我假设"抓取者"是抓住你网站的人),这是这个主题最好的学校.通过抓取几个网站,我学会了不同类型的保护,通过关联它们,我一直很有效率.

我给你一些你可能会尝试的保护措施的例子.

每个IP会话

如果用户每分钟使用50个新会话,您可以认为该用户可能是不处理cookie的爬虫.当然,curl可以完美地管理cookie,但是如果你将它与每个会话的访问计数器(稍后解释)相结合,或者如果你的爬虫是一个有cookie的noobie,它可能是有效的.

很难想象同一个共享连接中的50个人会同时在您的网站上获得(当然这取决于您的流量,这取决于您).如果发生这种情况,您可以锁定网站的页面,直到填写验证码.

想法:

1)你创建2个表:1表示保存禁止的ips,1表示保存ip和会话

create table if not exists sessions_per_ip (
  ip int unsigned,
  session_id varchar(32),
  creation timestamp default current_timestamp,
  primary key(ip, session_id)
);

create table if not exists banned_ips (
  ip int unsigned,
  creation timestamp default current_timestamp,
  primary key(ip)
);
Run Code Online (Sandbox Code Playgroud)

2)在脚本的开头,删除两个表中过旧的条目

3)接下来你检查你的用户的ip是否被禁止(你设置一个标志为true)

4)如果没有,你可以计算他的ip会话数量

5)如果他有太多的会话,你将它插入你的禁止表并设置一个标志

6)如果尚未插入会话,则将ip插入每个ip表的会话

我写了一个代码示例,以更好的方式展示我的想法.

<?php

try
{

    // Some configuration (small values for demo)
    $max_sessions = 5; // 5 sessions/ip simultaneousely allowed
    $check_duration = 30; // 30 secs max lifetime of an ip on the sessions_per_ip table
    $lock_duration = 60; // time to lock your website for this ip if max_sessions is reached

    // Mysql connection
    require_once("config.php");
    $dbh = new PDO("mysql:host={$host};dbname={$base}", $user, $password);
    $dbh->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);

    // Delete old entries in tables
    $query = "delete from sessions_per_ip where timestampdiff(second, creation, now()) > {$check_duration}";
    $dbh->exec($query);

    $query = "delete from banned_ips where timestampdiff(second, creation, now()) > {$lock_duration}";
    $dbh->exec($query);

    // Get useful info attached to our user...
    session_start();
    $ip = ip2long($_SERVER['REMOTE_ADDR']);
    $session_id = session_id();

    // Check if IP is already banned
    $banned = false;
    $count = $dbh->query("select count(*) from banned_ips where ip = '{$ip}'")->fetchColumn();
    if ($count > 0)
    {
        $banned = true;
    }
    else
    {
        // Count entries in our db for this ip
        $query = "select count(*)  from sessions_per_ip where ip = '{$ip}'";
        $count = $dbh->query($query)->fetchColumn();
        if ($count >= $max_sessions)
        {
            // Lock website for this ip
            $query = "insert ignore into banned_ips ( ip ) values ( '{$ip}' )";
            $dbh->exec($query);
            $banned = true;
        }

        // Insert a new entry on our db if user's session is not already recorded
        $query = "insert ignore into sessions_per_ip ( ip, session_id ) values ('{$ip}', '{$session_id}')";
        $dbh->exec($query);
    }

    // At this point you have a $banned if your user is banned or not.
    // The following code will allow us to test it...

    // We do not display anything now because we'll play with sessions :
    // to make the demo more readable I prefer going step by step like
    // this.
    ob_start();

    // Displays your current sessions
    echo "Your current sessions keys are : <br/>";
    $query = "select session_id from sessions_per_ip where ip = '{$ip}'";
    foreach ($dbh->query($query) as $row) {
        echo "{$row['session_id']}<br/>";
    }

    // Display and handle a way to create new sessions
    echo str_repeat('<br/>', 2);
    echo '<a href="' . basename(__FILE__) . '?new=1">Create a new session / reload</a>';
    if (isset($_GET['new']))
    {
        session_regenerate_id();
        session_destroy();
        header("Location: " . basename(__FILE__));
        die();
    }

    // Display if you're banned or not
    echo str_repeat('<br/>', 2);
    if ($banned)
    {
        echo '<span style="color:red;">You are banned: wait 60secs to be unbanned... a captcha must be more friendly of course!</span>';
        echo '<br/>';
        echo '<img src="http://4.bp.blogspot.com/-PezlYVgEEvg/TadW2e4OyHI/AAAAAAAAAAg/QHZPVQcBNeg/s1600/feu-rouge.png" />';
    }
    else
    {
        echo '<span style="color:blue;">You are not banned!</span>';
        echo '<br/>';
        echo '<img src="http://identityspecialist.files.wordpress.com/2010/06/traffic_light_green.png" />';
    }
    ob_end_flush();
}
catch (PDOException $e)
{
    /*echo*/ $e->getMessage();
}

?>
Run Code Online (Sandbox Code Playgroud)

访问柜台

如果您的用户使用相同的Cookie来抓取您的网页,您将能够使用他的会话来阻止它.这个想法非常简单:您的用户可能在60秒内访问60页吗？

想法:

  1. 在用户会话中创建一个数组,它将包含访问时间()s.
  2. 删除此阵列中超过X秒的访问次数
  3. 为实际访问添加新条目
  4. 计算此数组中的条目
  5. 如果他访问过Y页,请禁用您的用户

示例代码:

<?php

$visit_counter_pages = 5; // maximum number of pages to load
$visit_counter_secs = 10; // maximum amount of time before cleaning visits

session_start();

// initialize an array for our visit counter
if (array_key_exists('visit_counter', $_SESSION) == false)
{
    $_SESSION['visit_counter'] = array();
}

// clean old visits
foreach ($_SESSION['visit_counter'] as $key => $time)
{
    if ((time() - $time) > $visit_counter_secs) {
        unset($_SESSION['visit_counter'][$key]);
    }
}

// we add the current visit into our array
$_SESSION['visit_counter'][] = time();

// check if user has reached limit of visited pages
$banned = false;
if (count($_SESSION['visit_counter']) > $visit_counter_pages)
{
    // puts ip of our user on the same "banned table" as earlier...
    $banned = true;
}

// At this point you have a $banned if your user is banned or not.
// The following code will allow us to test it...

echo '<script type="text/javascript" src="https://ajax.googleapis.com/ajax/libs/jquery/1.6.2/jquery.min.js"></script>';

// Display counter
$count = count($_SESSION['visit_counter']);
echo "You visited {$count} pages.";
echo str_repeat('<br/>', 2);

echo <<< EOT

<a id="reload" href="#">Reload</a>

<script type="text/javascript">

  $('#reload').click(function(e) {
    e.preventDefault();
    window.location.reload();
  });

</script>

EOT;

echo str_repeat('<br/>', 2);

// Display if you're banned or not
echo str_repeat('<br/>', 2);
if ($banned)
{
    echo '<span style="color:red;">You are banned! Wait for a short while (10 secs in this demo)...</span>';
    echo '<br/>';
    echo '<img src="http://4.bp.blogspot.com/-PezlYVgEEvg/TadW2e4OyHI/AAAAAAAAAAg/QHZPVQcBNeg/s1600/feu-rouge.png" />';
}
else
{
    echo '<span style="color:blue;">You are not banned!</span>';
    echo '<br/>';
    echo '<img src="http://identityspecialist.files.wordpress.com/2010/06/traffic_light_green.png" />';
}
?>
Run Code Online (Sandbox Code Playgroud)

要下载的图像

当一个爬虫需要做他的肮脏工作时,那就是大量的数据,并且在最短的时间内完成.这就是他们不在页面上下载图像的原因; 它需要太多带宽并使爬行速度变慢.

这个想法(我认为最优雅和最容易实现)使用mod_rewrite隐藏.jpg/.png/...图像文件中的代码.此图像应该在您要保护的每个页面上可用:它可能是您的徽标网站,但您将选择一个小尺寸的图像(因为此图像不得缓存).

想法:

1 /将这些行添加到.htaccess 

RewriteEngine On
RewriteBase /tests/anticrawl/
RewriteRule ^logo\.jpg$ logo.php
Run Code Online (Sandbox Code Playgroud)

2 /使用安全性创建logo.php

<?php

// start session and reset counter
session_start();
$_SESSION['no_logo_count'] = 0;

// forces image to reload next time
header("Cache-Control: no-store, no-cache, must-revalidate");

// displays image
header("Content-type: image/jpg");
readfile("logo.jpg");
die();
Run Code Online (Sandbox Code Playgroud)

3 /在您需要添加安全性的每个页面上增加no_logo_count,并检查它是否达到了您的限制.

示例代码:

<?php

$no_logo_limit = 5; // number of allowd pages without logo

// start session and initialize
session_start();
if (array_key_exists('no_logo_count', $_SESSION) == false)
{
    $_SESSION['no_logo_count'] = 0;
}
else
{
    $_SESSION['no_logo_count']++;
}

// check if user has reached limit of "undownloaded image"
$banned = false;
if ($_SESSION['no_logo_count'] >= $no_logo_limit)
{
    // puts ip of our user on the same "banned table" as earlier...
    $banned = true;
}

// At this point you have a $banned if your user is banned or not.
// The following code will allow us to test it...

echo '<script type="text/javascript" src="https://ajax.googleapis.com/ajax/libs/jquery/1.6.2/jquery.min.js"></script>';

// Display counter
echo "You did not loaded image {$_SESSION['no_logo_count']} times.";
echo str_repeat('<br/>', 2);

// Display "reload" link
echo <<< EOT

<a id="reload" href="#">Reload</a>

<script type="text/javascript">

  $('#reload').click(function(e) {
    e.preventDefault();
    window.location.reload();
  });

</script>

EOT;

echo str_repeat('<br/>', 2);

// Display "show image" link : note that we're using .jpg file
echo <<< EOT

<div id="image_container">
    <a id="image_load" href="#">Load image</a>
</div>
<br/>

<script type="text/javascript">

  // On your implementation, you'llO of course use <img src="logo.jpg" />
  $('#image_load').click(function(e) {
    e.preventDefault();
    $('#image_load').html('<img src="logo.jpg" />');
  });

</script>

EOT;

// Display if you're banned or not
echo str_repeat('<br/>', 2);
if ($banned)
{
    echo '<span style="color:red;">You are banned: click on "load image" and reload...</span>';
    echo '<br/>';
    echo '<img src="http://4.bp.blogspot.com/-PezlYVgEEvg/TadW2e4OyHI/AAAAAAAAAAg/QHZPVQcBNeg/s1600/feu-rouge.png" />';
}
else
{
    echo '<span style="color:blue;">You are not banned!</span>';
    echo '<br/>';
    echo '<img src="http://identityspecialist.files.wordpress.com/2010/06/traffic_light_green.png" />';
}
?>
Run Code Online (Sandbox Code Playgroud)

Cookie检查

您可以在javascript端创建cookie以检查您的用户是否确实解释了javascript(例如,使用Curl的爬虫).

这个想法非常简单:这与图像检查大致相同.

  1. 将$ _SESSION值设置为1并在每次访问中增加它
  2. 如果确实存在cookie(在JavaScript中设置),则将会话值设置为0
  3. 如果此值达到限制,请禁止您的用户

代码:

<?php

$no_cookie_limit = 5; // number of allowd pages without cookie set check

// Start session and reset counter
session_start();

if (array_key_exists('cookie_check_count', $_SESSION) == false)
{
    $_SESSION['cookie_check_count'] = 0;
}

// Initializes cookie (note: rename it to a more discrete name of course) or check cookie value
if ((array_key_exists('cookie_check', $_COOKIE) == false) || ($_COOKIE['cookie_check'] != 42))
{
    // Cookie does not exist or is incorrect...
    $_SESSION['cookie_check_count']++;
}
else
{
    // Cookie is properly set so we reset counter
    $_SESSION['cookie_check_count'] = 0;
}

// Check if user has reached limit of "cookie check"
$banned = false;
if ($_SESSION['cookie_check_count'] >= $no_cookie_limit)
{
    // puts ip of our user on the same "banned table" as earlier...
    $banned = true;
}

// At this point you have a $banned if your user is banned or not.
// The following code will allow us to test it...

echo '<script type="text/javascript" src="https://ajax.googleapis.com/ajax/libs/jquery/1.6.2/jquery.min.js"></script>';

// Display counter
echo "Cookie check failed {$_SESSION['cookie_check_count']} times.";
echo str_repeat('<br/>', 2);

// Display "reload" link
echo <<< EOT

<br/>
<a id="reload" href="#">Reload</a>
<br/>

<script type="text/javascript">

  $('#reload').click(function(e) {
    e.preventDefault();
    window.location.reload();
  });

</script>

EOT;

// Display "set cookie" link
echo <<< EOT

<br/>
<a id="cookie_link" href="#">Set cookie</a>
<br/>

<script type="text/javascript">

  // On your implementation, you'll of course put the cookie set on a $(document).ready()
  $('#cookie_link').click(function(e) {
    e.preventDefault();
    var expires = new Date();
    expires.setTime(new Date().getTime() + 3600000);
    document.cookie="cookie_check=42;expires=" + expires.toGMTString();
  });

</script>
EOT;


// Display "unset cookie" link
echo <<< EOT

<br/>
<a id="unset_cookie" href="#">Unset cookie</a>
<br/>

<script type="text/javascript">

  // On your implementation, you'll of course put the cookie set on a $(document).ready()
  $('#unset_cookie').click(function(e) {
    e.preventDefault();
    document.cookie="cookie_check=;expires=Thu, 01 Jan 1970 00:00:01 GMT";
  });

</script>
EOT;

// Display if you're banned or not
echo str_repeat('<br/>', 2);
if ($banned)
{
    echo '<span style="color:red;">You are banned: click on "Set cookie" and reload...</span>';
    echo '<br/>';
    echo '<img src="http://4.bp.blogspot.com/-PezlYVgEEvg/TadW2e4OyHI/AAAAAAAAAAg/QHZPVQcBNeg/s1600/feu-rouge.png" />';
}
else
{
    echo '<span style="color:blue;">You are not banned!</span>';
    echo '<br/>';
    echo '<img src="http://identityspecialist.files.wordpress.com/2010/06/traffic_light_green.png" />';
}
Run Code Online (Sandbox Code Playgroud)

防止代理

关于我们可能在网络上找到的不同类型代理的一些说法:

  • "普通"代理显示有关用户连接的信息(特别是他的IP)
  • 匿名代理不显示IP,但提供有关标头上代理使用情况的信息.
  • 高级匿名代理不显示用户IP,也不显示浏览器可能不发送的任何信息.

很容易找到连接任何网站的代理,但很难找到高匿名代理.

如果您的用户位于代理后面,某些$ _SERVER变量可能包含密钥(详细列表来自此问题):

  • CLIENT_IP
  • 转发
  • FORWARDED_FOR
  • FORWARDED_FOR_IP
  • HTTP_CLIENT_IP
  • HTTP_FORWARDED
  • HTTP_FORWARDED_FOR
  • HTTP_FORWARDED_FOR_IP
  • HTTP_PC_REMOTE_ADDR
  • HTTP_PROXY_CONNECTION"
  • HTTP_VIA
  • HTTP_X_FORWARDED
  • HTTP_X_FORWARDED_FOR
  • HTTP_X_FORWARDED_FOR_IP
  • HTTP_X_IMFORWARDS
  • HTTP_XROXY_CONNECTION
  • 通过
  • X_FORWARDED
  • X_FORWARDED_FOR

如果您在$_SERVER变量上检测到其中一个键,则可以为反爬行证券提供不同的行为(下限等).

结论

有很多方法可以检测您网站上的滥用行为,因此您肯定会找到解决方案.但您需要准确了解您的网站的使用方式,因此您的证券不会对您的"普通"用户产生侵略性.

归档时间：

查看次数：

21304 次

最近记录：

7 年，7 月 前
PHP获得真正的IP(代理检测) 7
如何检测服务器/脚本是否通过cURL/file_get_contents()访问其站点？(不包括用户代理和IP地址) 0
更多相关链接
相关归档
如何将数字四舍五入到最接近的10？ 87
在MySQL中将行值增加和减少1 39
Laravel快速入门指南路线不起作用 35
我们可以在PHP的任何函数中传递数组作为参数吗？ 30
jQuery AJAX请求中的相对与绝对URL 23
保存远程网页的cookie 9
更新贝宝计费计划（payment_definitions） 5
C++ cUrl将mulipart/form-data文件发送到webserver 5
如何从命令行在网络浏览器中打开站点？ 4
如何访问CouchDB数据库中的所有文档？ 3
难疑归档
如何在提交前撤消'git add'？ 8567
如何从当前的Git工作树中删除本地(未跟踪)文件？ 6561
当键盘出现时 - 如何开始编辑时,如何让UITextField向上移动？ 1674
如何使用Bash将stdout和stderr重定向并附加到文件中？ 1440
将文件从主机复制到Docker容器 1391
插入...值(SELECT ... FROM ...) 1340
使用Twitter Bootstrap 3将列居中 1109
"正确"的JSON日期格式 1071
如何删除选择框的所有选项,然后添加一个选项并使用jQuery选择它？ 1028
功能编程是否取代了GoF设计模式？ 1028