如何防止机器人尝试猜测我网站上的链接

Question

我最近安装的 logwatch 报告显示了这个：

--------------------- httpd Begin ------------------------
0.78 MB transferred in 5864 responses  (1xx 0, 2xx 4900, 3xx 0, 4xx 964, 5xx 0)
160 Images (0.16 MB),
857 Content pages (0.62 MB),
4847 Other (0.00 MB)

Requests with error response codes
404 Not Found
  /%E2%80%98planeat%E2%80%99-film-explores-l ... greenfudge-org/: 1 Time(s)
  /10-foods-to-add-to-the-brain-diet-to-help ... -function/feed/: 1 Time(s)
  /10-ways-to-reboot-your-body-with-healthy- ... s-and-exercise/: 1 Time(s)
  /bachmann-holds-her-ground-against-raising ... com-blogs/feed/: 1 Time(s)
  /behind-conan-the-barbarians-diet/: 1 Time(s)
  /tag/dietitian/: 1 Time(s)
  /tag/diets/page/10/: 1 Time(s)
  /tag/directory-products/feed/: 1 Time(s)
  /wp-content/uploads/2011/06/1309268736-49.jpg: 1 Time(s)
  /wp-content/uploads/2011/06/1309271430-30.jpg: 1 Time(s)
  /wp-content/uploads/2011/06/1309339847-35.jpg: 1 Time(s)

我在此注意：确实有很多像上面这样的请求，为了清楚起见，我只粘贴了一些。

  A total of 12 ROBOTS were logged
  Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) 2 Time(s)
  Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots) 5 Time(s)
  Twitterbot/1.0 1 Time(s)
  Mozilla/5.0 (compatible; AhrefsBot/2.0; +http://ahrefs.com/robot/) 4 Time(s)
  Sosospider+(+http://help.soso.com/webspider.htm) 3 Time(s)
  msnbot/2.0b (+http://search.msn.com/msnbot.htm)._ 1 Time(s)
  Mozilla/5.0 (compatible; MJ12bot/v1.4.2; http://www.majestic12.co.uk/bot.php?+) 1    Time(s)
  msnbot-media/1.1 (+http://search.msn.com/msnbot.htm) 77 Time(s)
  Mozilla/5.0 (compatible; Ezooms/1.0; ezooms.bot@gmail.com) 1 Time(s)
  Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) 17 Time(s)
  Baiduspider+(+http://www.baidu.com/search/spider.htm) 11 Time(s)
  Mozilla/5.0 (compatible; Butterfly/1.0; +http://labs.topsy.com/butterfly/)    Gecko/2009032608 Firefox/3.0.8 1 Time(s)
  ---------------------- httpd End -------------------------

所以，我认为这是某种机器人（并且可能是上面列出的机器人之一），所以请您指导我如何防止他们猜测链接以希望找到内容？

编辑：因为我拥有一台 VPS 服务器，所以上面有很多域。你能告诉我我怎么知道发生在哪个域上的特定 404？例如这一行：/tag/dietian/

Answer 1

真的，您再也无法阻止普通用户猜测链接了。正确保护您的内容，无论如何这都不会成为问题。

模糊链接不是隐藏事物的安全方式。

你可以确保你有一个正确配置的 robots.txt - 这将阻止大多数合法机器人。