在我的 asp.net mvc 网站中创建 robots.txt 文件的最佳实践

Question

我想为我的 asp.net mvc-5 网站创建一个 robots.txt，现在我找到了这个链接，它谈到了实现这个任务：-

http://rehansaeed.com/dynamically-generating-robots-txt-using-asp-net-mvc/

where in this link they are creating a separate Controller & Route rule to build the robots.txt ,,,so I am not sure why i can not just create the robot.txt file and add it to the root of my web site as follow:-:-

where if I navigate to the following URL http://www.mywebsite.com/robots.txt the text content will be shown ,without having to create separate controller for this?

so my question is if it is valid to add the robots.txt directly to my web site root without having to do so inside a controller and separate route rule , to keep things simpler ??

Answer 1

在我的网站 asp net core 3.1 上，我只使用了以下简单操作：

    [Route("/robots.txt")]
    public ContentResult RobotsTxt()
    {
        var sb = new StringBuilder();
        sb.AppendLine("User-agent: *")
            .AppendLine("Disallow:")
            .Append("sitemap: ")
            .Append(this.Request.Scheme)
            .Append("://")
            .Append(this.Request.Host)
            .AppendLine("/sitemap.xml");

        return this.Content(sb.ToString(), "text/plain", Encoding.UTF8);
    }

Answer 2

如果您需要动态生成内容，那么这是有意义的，因为它会给您这种能力。但除此之外，我在应用程序中使用 robots.txt 文件没有任何问题。