限制天蓝色 blob 上传速度的方法

Question

我有一个应用程序可以将数据库备份上传到 azure 上的 blob 容器。

我上传的文件大小约为 8GB，平均速率为 11Mbps

然而，一切正常，当 blob 被传输到容器时，它绝对会杀死该网络上的所有其他内容。

所以我的问题是，有什么方法可以将上传速度限制为 azure？

Answer 1

您的问题没有指定您希望使用哪种编程语言（或原始 REST）来实现上传速度限制。但是我找到了一种使用官方 azure 存储 .NET SDK (Microsoft.WindowsAzure.Storage) 来做到这一点的方法。

SDK 本身不提供明确的速度限制。然而，这些Upload*Async()功能，例如，UploadFromFileAsync()确实支持进度报告，通过采用一个IProgress<StorageProgress>实现，并Report()定期调用它的方法。并且它似乎是同步调用它的。

所以我们可以设置延迟Report()来限制上传速度。更好的是，它可以Report为您提供有关BytesTransferred. 因此，如果您的进度处理程序跟踪持续时间。然后你可以估计当前的上传速度，并用它来精确节流。

下面是这个想法的说明。请注意，这仅仅是该想法的演示。我在不到 3 分钟的时间内把它拼凑好了。速率估计和节流算法非常粗略，没有经过很好的测试。

这是你将如何使用它

await blob.UploadFromFileAsync(
    @"some_file.dat", 
    null, null, null, 
    new RateThrottleProgress(300 * 1024), // throttle at 300kb/s
    CancellationToken.None);

指数退避节流阀实现

class RateThrottleProgress : IProgress<StorageProgress>
{
    private readonly DateTime start = DateTime.Now;
    private readonly long maxbps;
    private long baseDelay, delay;

    public RateThrottleProgress(long maxbps)
    {
        this.maxbps = maxbps;
        baseDelay = 10;
        delay = baseDelay;
    }

    public void Report(StorageProgress value)
    {
        double duration = (DateTime.Now - start).TotalSeconds;
        double bps = value.BytesTransferred / duration;
        if (bps > maxbps) delay *= 2;
        else delay = Math.Max(baseDelay, delay/2);
        Console.WriteLine($"current estimated upload speed: {bps / 1024.0} KB/s. delay: {delay} ms");
        Thread.Sleep(TimeSpan.FromMilliseconds(delay));
    }
}

我还将下面的代码片段作为更好的版本控制和协作的要点。

节流效果：

current estimated upload speed: 287.486007463505 KB/s. delay: 10 ms
current estimated upload speed: 290.086402388889 KB/s. delay: 10 ms
current estimated upload speed: 292.685419108659 KB/s. delay: 10 ms
current estimated upload speed: 295.28201245662 KB/s. delay: 10 ms
current estimated upload speed: 297.876060423937 KB/s. delay: 10 ms
current estimated upload speed: 300.469027029562 KB/s. delay: 20 ms
current estimated upload speed: 302.927815243916 KB/s. delay: 40 ms
current estimated upload speed: 305.112558483135 KB/s. delay: 80 ms
current estimated upload speed: 306.778888691779 KB/s. delay: 160 ms
current estimated upload speed: 307.367196107083 KB/s. delay: 320 ms   <-- speed starts to drop from here...
current estimated upload speed: 305.910611140488 KB/s. delay: 640 ms
current estimated upload speed: 300.564767027164 KB/s. delay: 1280 ms
current estimated upload speed: 288.206861583389 KB/s. delay: 640 ms
current estimated upload speed: 283.672713628354 KB/s. delay: 320 ms
current estimated upload speed: 282.668039190231 KB/s. delay: 160 ms
current estimated upload speed: 283.351226090087 KB/s. delay: 80 ms
current estimated upload speed: 284.861107569046 KB/s. delay: 40 ms
current estimated upload speed: 286.781960850501 KB/s. delay: 20 ms
current estimated upload speed: 288.910675693183 KB/s. delay: 10 ms
current estimated upload speed: 291.140146046991 KB/s. delay: 10 ms
current estimated upload speed: 293.358817316007 KB/s. delay: 10 ms

Answer 2

Azure 存储本身不会提供特定的限制设置。但是：鉴于 Blob 上传本质上是一系列基于 REST 的块 BlobPUT操作，您可以通过执行自己的上传实现并选择将块上传到给定 Blob 的频率来进行自己的限制。