<?xml version="1.0" encoding="UTF-8" ?>
<rss version="2.0">
<channel>
<title><![CDATA[刘新修]]></title> 
<link>http://liuxinxiu.com:80/index.php</link> 
<description><![CDATA[刘新修的个人博客 (Liuxinxiu'S Blog)]]></description> 
<language>zh-cn</language> 
<copyright><![CDATA[刘新修]]></copyright>
<item>
<link>http://liuxinxiu.com:80/s//</link>
<title><![CDATA[Go实现多线程分片下载文件]]></title> 
<author>刘新修 &lt;admin@yourname.com&gt;</author>
<category><![CDATA[PHP/Java/Go]]></category>
<pubDate>Mon, 21 Oct 2024 08:39:22 +0000</pubDate> 
<guid>http://liuxinxiu.com:80/s//</guid> 
<description>
<![CDATA[ 
	<p>&nbsp;我们在下载大文件时，通常会使用多线程下载的方式来加快下载速度。例如常用的多线程下载工具（Gopeed、Aria2、XDM等等），都是通过多线程下载技术充分利用了网络带宽，以提高下载速度。</p><div><div><p>那么多线程下载是怎么实现的呢？多个线程发送网络请求，是怎么做到同时下载一个文件呢？事实上，借助HTTP协议中的一些机制就可以实现了！</p><p>今天我们就通过使用Go语言为例，从了解HTTP请求相关的一些机制开始，实现一个多线程下载的示例。</p><h2 data-id="heading-0">1，多线程下载原理</h2><p>事实上，多线程下载的原理很简单，主要的步骤如下：</p><ul><li>获取待下载文件大小</li><li>每个线程下载文件的一部分</li><li>全部下载完成后，拼接为完整文件</li></ul><p>实现这些步骤，就涉及到HTTP协议的下列相关机制。</p><h3 data-id="heading-1">(1) <code>HEAD</code>请求 - 只获取请求头</h3><p>我们通常发送HTTP请求大多数是<code>GET</code>或者<code>POST</code>类型，发送请求后我们会立即获取响应体，浏览器则会根据响应体的类型来处理内容，例如返回的是<code>text/html</code>就会作为网页显示，返回<code>image/png</code>就会解码为图片等等，响应体的类型由响应头<code>Content-Type</code>标识。当我们下载文件时，事实上也是发送HTTP请求，只不过<strong>服务器返回的响应体就是文件本身</strong>了！其类型则是<code>application/octet-stream</code>，浏览器也知道这是个文件需要下载。</p><p>当然，文件作为响应体通常比起网页、图片要大得多，在多线程下载时，<strong>我们就要先获取文件的大小，而不是立即获取文件本身</strong>，这时我们就可以向服务器发起<code>HEAD</code>请求而不是<code>GET</code>请求。</p><p>服务器收到<code>HEAD</code>请求后，就<strong>只会返回对应的响应头，而不会返回响应体</strong>，这样我们就可以在下载文件之前，读取响应头中的<code>Content-Length</code>来先获取待下载文件大小。</p><h3 data-id="heading-2">(2) <code>Range</code>请求头 - 只获取部分响应体</h3><p>知道了文件大小，我们就需要让每个线程只下载一部分文件，借助HTTP的<code>Range</code>请求头，就可以实现只让服务端返回响应体内容的一部分，而不是返回完整的响应体。</p><p>这里我们先来借助书籍《图解HTTP》中对<code>Range</code>请求头的讲解，来学习一下：</p></div><p><img src="/attachment/2024/10/image/2024-10-21_163241.png" width="610" height="946" alt="" /></p><div class="codeText"><div class="codeHead">XML/HTML代码</div><ol start="1" class="dp-xml"><li class="alt"><span><span>Range:&nbsp;</span><span class="attribute">bytes</span><span>=</span><span class="attribute-value">5001</span><span>-10000&nbsp;&nbsp;</span></span></li></ol></div><div><div><p>那么服务端就只会返回响应体的第<code>5001</code>到第<code>10000</code>字节的内容部分，包含第<code>5001</code>和第<code>10000</code>字节，<code>0</code>表示响应体的第一个字节。</p><p>这样，在多个线程同时下载文件时，我们在每个线程的请求中使用<code>Range</code>请求头，就可以实现一个线程只下载文件的一部分了！</p><h3 data-id="heading-3">(3) 为什么多线程下载可以提升速度？</h3><p>事实上，在我们客户端（下载文件的）和服务端双向网络通信情况都很好的情况下，使用单线程和多线程下载的速度是几乎没有差异的，也就是说能够跑满我们客户端的全部带宽，那么这种情况下我们使用单线程下载反而更能够节省硬件和网络资源。</p><p>但是在我们客户端和服务端之间网络波动较大的情况下，例如我们国内从Github下载文件的时候，就会发现多线程下载速度比单线程快得多，反之使用单线程完全无法充分利用我们的网络带宽。</p><p>这种现象事实上是因为TCP连接的<strong>慢启动机制</strong>导致的，众所周知HTTP是基于TCP的协议，每次我们建立HTTP连接时，包括下载文件，都是在传输层基于TCP协议进行传输。<strong>TCP慢启动机制</strong>是TCP 协议中一种拥塞控制的机制，目的是在开始数据传输时逐步探测网络的容量，避免瞬间发送大量数据而导致网络拥塞。慢启动不是字面意义上的&ldquo;慢&rdquo;，而是相对于立即使用最大带宽而言，它会逐渐增加传输速率。</p><p>慢启动机制的过程简要概括如下：</p><ul><li><strong>一开始建立连接</strong>：当一个新的TCP连接建立后，发送方并不知道当前网络的拥塞情况。因此，发送方不会马上发送大量数据，而是会使用<strong>慢启动机制</strong>来逐步增加数据传输的速率，在TCP中使用阻塞窗口<code>cwnd</code>来限制发送的数据量，也就是说一开始<code>cwnd</code>是非常小的</li><li><strong>拥塞窗口增长</strong>：在建立连接后，每当接收到一个确认<code>ACK</code>包时，<code>cwnd</code>会<strong>指数级增长</strong>，直到达到网络的带宽限制或者某个拥塞控制的阈值（称为慢启动阈值<code>ssthresh</code>），这个过程会一直持续，直到发送方探测到网络出现了拥塞（比如丢包或者确认延迟变长），或者<code>cwnd</code>达到了某个预定义的慢启动阈值<code>ssthresh</code></li><li><strong>慢启动的终止</strong>：慢启动机制会在以下情况终止：<ul><li><strong>达到慢启动阈值<code>ssthresh</code></strong>：当拥塞窗口<code>cwnd</code>增长到慢启动阈值<code>ssthresh</code>时，慢启动机制停止，此时TCP会进入另一种拥塞控制机制，称为<strong>拥塞避免</strong>，这时<code>cwnd</code>增长变为线性而非指数级</li><li><strong>发生拥塞（如丢包或超时）</strong>：如果发送方检测到数据包丢失（例如没有收到确认），它会认为网络已经出现拥塞，此时<code>ssthresh</code>会被调整为当前<code>cwnd</code>的一半，然后<code>cwnd</code>会重置为<code>1 MSS</code>，重新进入慢启动阶段</li></ul></li></ul><p>可见TCP连接使用<code>cwnd</code>限制两者发送的数据量的大小，并逐步&ldquo;试探&rdquo;两者传输数据速率的上限并增加传输的数据量。</p><p>在我们下载文件时，事实上是服务端在向我们发送文件，如果网络波动较大、不稳定，TCP连接机会一直将<code>cwnd</code>限制在一个较小的值，在单位时间内，服务端也无法向我们发送更大的数据量。</p><p>此时，如果我们使用多线程下载，和服务端建立多个TCP连接，这样即使每个TCP连接的<code>cwnd</code>较小，所有TCP连接加起来传输的数据量仍然可以占满我们的带宽。</p><h2 data-id="heading-4">2，Go代码实现</h2><p>知道了HTTP的上述几个机制，相信大家就知道如何实现一个简单的多线程下载了！我们可以总结主要步骤如下：</p><ul><li>发送<code>HEAD</code>类型请求，通过<code>Content-Length</code>请求头获取待下载文件大小</li><li>根据给定的线程数量，结合待下载文件大小，确定每个线程下载的范围部分，也就是每个线程的<code>Range</code>请求头字节范围</li><li>启动所有线程，使得每个线程下载它们对应的部分文件，并等待全部线程下载完成</li><li>合并每个线程下载的部分为最终文件</li><li>清理每个线程下载的文件部分</li></ul><p>这里分别设计下列类（结构体），用于存放多线程下载时的传入参数和状态量：</p></div><div><img src="/attachment/2024/10/image/2024-10-21_163453.png" alt="" /></div><div>&nbsp;</div><div><div><p>上述<code>ShardTask</code>类表示<strong>一个线程的下载任务</strong>，其中会完成一个分片（文件的一部分）的下载请求操作，它有如下作为参数的属性：</p><ul><li><code>Url</code> 下载的文件地址</li><li><code>Order</code> 分片序号</li><li><code>ShardFilePath</code> 这个分片文件的保存路径</li><li><code>RangeStart</code>和<code>RangeEnd</code> 下载的文件起始范围和结束范围，用于设定<code>Range</code>请求头</li></ul><p>此外，还有作为下载状态的属性：</p><ul><li><code>DownloadSize</code> 下载任务进行时，这个线程已下载的文件部分大小</li><li><code>TaskDone</code> 这个线程的下载任务是否完成</li></ul><p>该类的成员方法如下：</p><ul><li><code>DoShardGet</code> 执行分片下载任务，在其中会根据<code>RangeStart</code>和<code>RangeEnd</code>设定对应的HTTP请求头，发送请求并下载对应的文件部分</li></ul><p>然后就是<code>ParallelGetTask</code>类，表示<strong>一整个多线程下载任务</strong>，其中包含了一个多线程下载任务的参数和状态量，并且实现了多线程下载的每个步骤，它有如下作为参数的属性：</p><ul><li><code>Url</code> 文件的下载链接</li><li><code>FilePath</code> 文件下载完成后的保存位置</li><li><code>Concurrent</code> 下载并发数，即同时下载的分片数量</li><li><code>TempFolder</code> 临时分片文件的保存文件夹</li></ul><p>此外还有作为状态的属性：</p><ul><li><code>TotalSize</code> 待下载文件的总大小</li><li><code>ShardTaskList</code> 存储所有分片任务对象指针的列表</li></ul><p>该类中的方法主要是分片下载的一些步骤如下：</p><ul><li><code>getLength</code> 发送<code>HEAD</code>请求获取<code>Content-Length</code>以获取文件大小，获取后将其设定到<code>TotalSize</code>属性</li><li><code>allocateTask</code> 根据给定的线程数和获取到的文件大小，计算每个线程下载的文件内容范围，并创建对应的<code>ShardTask</code>结构体放入<code>ShardTaskList</code>中</li><li><code>downloadShard</code> 为每一个<code>ShardTask</code>对象创建一个线程（Goroutine）并在新的线程中调用<code>ShardTask</code>对象的下载分片方法，以启动所有线程的下载任务，并通过<code>sync.WaitGroup</code>来等待全部线程完成</li><li><code>mergeFile</code> 下载完成后，合并每个分片为最终文件</li><li><code>cleanShard</code> 合并完成后，清理下载的每个分片文件</li><li><code>printTotalProcess</code> 这是一个附加的辅助方法，用于实时输出下载进度</li><li><code>Run</code> 启动整个多线程下载任务，该函数是暴露的公开函数，其中对上述每个步骤函数进行了组织，按顺序调用执行</li></ul><p>下面，我们来看一下它们的代码实现。</p><h3 data-id="heading-5">(1) <code>ShardTask</code> - 一个线程的下载任务</h3></div><div class="codeText"><div class="codeHead">C#代码</div><ol start="1" class="dp-c"><li class="alt"><span><span>package&nbsp;model&nbsp;&nbsp;</span></span></li><li><span>&nbsp;&nbsp;</span></li><li class="alt"><span>import&nbsp;(&nbsp;&nbsp;</span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;<span class="string">&quot;bufio&quot;</span><span>&nbsp;&nbsp;</span></span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;<span class="string">&quot;fmt&quot;</span><span>&nbsp;&nbsp;</span></span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;<span class="string">&quot;github.com/fatih/color&quot;</span><span>&nbsp;&nbsp;</span></span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;<span class="string">&quot;io&quot;</span><span>&nbsp;&nbsp;</span></span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;<span class="string">&quot;net/http&quot;</span><span>&nbsp;&nbsp;</span></span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;<span class="string">&quot;os&quot;</span><span>&nbsp;&nbsp;</span></span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;<span class="string">&quot;sync&quot;</span><span>&nbsp;&nbsp;</span></span></li><li class="alt"><span>)&nbsp;&nbsp;</span></li><li><span>&nbsp;&nbsp;</span></li><li class="alt"><span><span class="comment">//&nbsp;全局HTTP客户端</span><span>&nbsp;&nbsp;</span></span></li><li><span>var&nbsp;httpClient&nbsp;=&nbsp;http.Client&#123;&nbsp;&nbsp;</span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;Transport:&nbsp;&amp;http.Transport&#123;&nbsp;&nbsp;</span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span class="comment">//&nbsp;关闭keep-alive确保一个线程就使用一个TCP连接</span><span>&nbsp;&nbsp;</span></span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;DisableKeepAlives:&nbsp;<span class="keyword">true</span><span>,&nbsp;&nbsp;</span></span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;&#125;,&nbsp;&nbsp;</span></li><li class="alt"><span>&#125;&nbsp;&nbsp;</span></li><li><span>&nbsp;&nbsp;</span></li><li class="alt"><span><span class="comment">//&nbsp;ShardTask&nbsp;单个分片下载任务的任务参数和状态量</span><span>&nbsp;&nbsp;</span></span></li><li><span>type&nbsp;ShardTask&nbsp;<span class="keyword">struct</span><span>&nbsp;&#123;&nbsp;&nbsp;</span></span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;<span class="comment">//&nbsp;下载链接</span><span>&nbsp;&nbsp;</span></span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;Url&nbsp;<span class="keyword">string</span><span>&nbsp;&nbsp;</span></span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;<span class="comment">//&nbsp;分片序号，从1开始</span><span>&nbsp;&nbsp;</span></span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;Order&nbsp;<span class="keyword">int</span><span>&nbsp;&nbsp;</span></span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;<span class="comment">//&nbsp;这个分片文件的路径</span><span>&nbsp;&nbsp;</span></span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;ShardFilePath&nbsp;<span class="keyword">string</span><span>&nbsp;&nbsp;</span></span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;<span class="comment">//&nbsp;分片的起始范围（字节，包含）</span><span>&nbsp;&nbsp;</span></span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;RangeStart&nbsp;int64&nbsp;&nbsp;</span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;<span class="comment">//&nbsp;分片的结束范围（字节，包含）</span><span>&nbsp;&nbsp;</span></span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;RangeEnd&nbsp;int64&nbsp;&nbsp;</span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;<span class="comment">//&nbsp;已下载的部分（字节）</span><span>&nbsp;&nbsp;</span></span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;DownloadSize&nbsp;int64&nbsp;&nbsp;</span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;<span class="comment">//&nbsp;该任务是否完成</span><span>&nbsp;&nbsp;</span></span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;TaskDone&nbsp;<span class="keyword">bool</span><span>&nbsp;&nbsp;</span></span></li><li class="alt"><span>&#125;&nbsp;&nbsp;</span></li><li><span>&nbsp;&nbsp;</span></li><li class="alt"><span><span class="comment">//&nbsp;NewShardTask&nbsp;构造函数</span><span>&nbsp;&nbsp;</span></span></li><li><span>func&nbsp;NewShardTask(url&nbsp;<span class="keyword">string</span><span>,&nbsp;order&nbsp;</span><span class="keyword">int</span><span>,&nbsp;shardFilePath&nbsp;</span><span class="keyword">string</span><span>,&nbsp;rangeStart&nbsp;int64,&nbsp;rangeEnd&nbsp;int64)&nbsp;*ShardTask&nbsp;&#123;&nbsp;&nbsp;</span></span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;<span class="keyword">return</span><span>&nbsp;&amp;ShardTask&#123;&nbsp;&nbsp;</span></span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span class="comment">//&nbsp;设定任务参数</span><span>&nbsp;&nbsp;</span></span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Url:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;url,&nbsp;&nbsp;</span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Order:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;order,&nbsp;&nbsp;</span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;ShardFilePath:&nbsp;shardFilePath,&nbsp;&nbsp;</span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;RangeStart:&nbsp;&nbsp;&nbsp;&nbsp;rangeStart,&nbsp;&nbsp;</span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;RangeEnd:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;rangeEnd,&nbsp;&nbsp;</span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span class="comment">//&nbsp;初始化状态量</span><span>&nbsp;&nbsp;</span></span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;DownloadSize:&nbsp;0,&nbsp;&nbsp;</span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;TaskDone:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span class="keyword">false</span><span>,&nbsp;&nbsp;</span></span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;&#125;&nbsp;&nbsp;</span></li><li><span>&#125;&nbsp;&nbsp;</span></li><li class="alt"><span>&nbsp;&nbsp;</span></li><li><span><span class="comment">//&nbsp;DoShardGet&nbsp;开始下载这个分片（该方法在goroutine中执行）</span><span>&nbsp;&nbsp;</span></span></li><li class="alt"><span>func&nbsp;(task&nbsp;*ShardTask)&nbsp;DoShardGet(waitGroup&nbsp;*sync.WaitGroup)&nbsp;&#123;&nbsp;&nbsp;</span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;<span class="comment">//&nbsp;创建文件</span><span>&nbsp;&nbsp;</span></span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;file,&nbsp;e&nbsp;:=&nbsp;os.OpenFile(task.ShardFilePath,&nbsp;os.O_CREATE&#124;os.O_WRONLY&#124;os.O_TRUNC,&nbsp;0755)&nbsp;&nbsp;</span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;<span class="keyword">if</span><span>&nbsp;e&nbsp;!=&nbsp;nil&nbsp;&#123;&nbsp;&nbsp;</span></span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;color.Red(<span class="string">&quot;任务%d创建文件失败！&quot;</span><span>,&nbsp;task.Order)&nbsp;&nbsp;</span></span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;color.HiRed(<span class="string">&quot;%s&quot;</span><span>,&nbsp;e)&nbsp;&nbsp;</span></span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span class="keyword">return</span><span>&nbsp;&nbsp;</span></span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;&#125;&nbsp;&nbsp;</span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;<span class="comment">//&nbsp;准备请求</span><span>&nbsp;&nbsp;</span></span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;request,&nbsp;e&nbsp;:=&nbsp;http.NewRequest(<span class="string">&quot;GET&quot;</span><span>,&nbsp;task.Url,&nbsp;nil)&nbsp;&nbsp;</span></span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;<span class="keyword">if</span><span>&nbsp;e&nbsp;!=&nbsp;nil&nbsp;&#123;&nbsp;&nbsp;</span></span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;color.Red(<span class="string">&quot;任务%d创建请求出错！&quot;</span><span>,&nbsp;task.Order)&nbsp;&nbsp;</span></span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;color.HiRed(<span class="string">&quot;%s&quot;</span><span>,&nbsp;e)&nbsp;&nbsp;</span></span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span class="keyword">return</span><span>&nbsp;&nbsp;</span></span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;&#125;&nbsp;&nbsp;</span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;<span class="comment">//&nbsp;设定请求头</span><span>&nbsp;&nbsp;</span></span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;request.Header.Set(<span class="string">&quot;Range&quot;</span><span>,&nbsp;fmt.Sprintf(</span><span class="string">&quot;bytes=%d-%d&quot;</span><span>,&nbsp;task.RangeStart,&nbsp;task.RangeEnd))&nbsp;&nbsp;</span></span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;<span class="comment">//&nbsp;发送请求</span><span>&nbsp;&nbsp;</span></span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;response,&nbsp;e&nbsp;:=&nbsp;httpClient.Do(request)&nbsp;&nbsp;</span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;<span class="keyword">if</span><span>&nbsp;e&nbsp;!=&nbsp;nil&nbsp;&#123;&nbsp;&nbsp;</span></span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;color.Red(<span class="string">&quot;任务%d发送下载请求出错！&quot;</span><span>,&nbsp;task.Order)&nbsp;&nbsp;</span></span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;color.HiRed(<span class="string">&quot;%s&quot;</span><span>,&nbsp;e)&nbsp;&nbsp;</span></span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span class="keyword">return</span><span>&nbsp;&nbsp;</span></span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;&#125;&nbsp;&nbsp;</span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;<span class="comment">//&nbsp;读取请求体</span><span>&nbsp;&nbsp;</span></span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;body&nbsp;:=&nbsp;response.Body&nbsp;&nbsp;</span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;<span class="comment">//&nbsp;读取缓冲区</span><span>&nbsp;&nbsp;</span></span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;buffer&nbsp;:=&nbsp;make([]<span class="keyword">byte</span><span>,&nbsp;8092)&nbsp;&nbsp;</span></span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;<span class="comment">//&nbsp;准备写入文件</span><span>&nbsp;&nbsp;</span></span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;writer&nbsp;:=&nbsp;bufio.NewWriter(file)&nbsp;&nbsp;</span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;<span class="keyword">for</span><span>&nbsp;&#123;&nbsp;&nbsp;</span></span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span class="comment">//&nbsp;读取一次内容至缓冲区</span><span>&nbsp;&nbsp;</span></span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;readSize,&nbsp;readError&nbsp;:=&nbsp;body.Read(buffer)&nbsp;&nbsp;</span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span class="keyword">if</span><span>&nbsp;readError&nbsp;!=&nbsp;nil&nbsp;&#123;&nbsp;&nbsp;</span></span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span class="comment">//&nbsp;如果读取完毕则退出循环</span><span>&nbsp;&nbsp;</span></span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span class="keyword">if</span><span>&nbsp;readError&nbsp;==&nbsp;io.EOF&nbsp;&#123;&nbsp;&nbsp;</span></span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span class="keyword">break</span><span>&nbsp;&nbsp;</span></span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&#125;&nbsp;<span class="keyword">else</span><span>&nbsp;&#123;&nbsp;&nbsp;</span></span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;color.Red(<span class="string">&quot;任务%d读取响应错误！&quot;</span><span>,&nbsp;task.Order)&nbsp;&nbsp;</span></span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;color.HiRed(<span class="string">&quot;%s&quot;</span><span>,&nbsp;readError)&nbsp;&nbsp;</span></span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span class="keyword">return</span><span>&nbsp;&nbsp;</span></span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&#125;&nbsp;&nbsp;</span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&#125;&nbsp;&nbsp;</span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span class="comment">//&nbsp;把缓冲区内容追加至文件</span><span>&nbsp;&nbsp;</span></span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;_,&nbsp;writeError&nbsp;:=&nbsp;writer.Write(buffer[0:readSize])&nbsp;&nbsp;</span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span class="keyword">if</span><span>&nbsp;writeError&nbsp;!=&nbsp;nil&nbsp;&#123;&nbsp;&nbsp;</span></span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;color.Red(<span class="string">&quot;任务%d写入文件时出现错误！&quot;</span><span>,&nbsp;task.Order)&nbsp;&nbsp;</span></span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;color.HiRed(<span class="string">&quot;%s&quot;</span><span>,&nbsp;writeError)&nbsp;&nbsp;</span></span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span class="keyword">return</span><span>&nbsp;&nbsp;</span></span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&#125;&nbsp;&nbsp;</span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;_&nbsp;=&nbsp;writer.Flush()&nbsp;&nbsp;</span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span class="comment">//&nbsp;记录下载进度</span><span>&nbsp;&nbsp;</span></span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;task.DownloadSize&nbsp;+=&nbsp;int64(readSize)&nbsp;&nbsp;</span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;&#125;&nbsp;&nbsp;</span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;<span class="comment">//&nbsp;关闭全部资源</span><span>&nbsp;&nbsp;</span></span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;_&nbsp;=&nbsp;body.Close()&nbsp;&nbsp;</span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;_&nbsp;=&nbsp;file.Close()&nbsp;&nbsp;</span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;<span class="comment">//&nbsp;标记任务完成</span><span>&nbsp;&nbsp;</span></span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;task.TaskDone&nbsp;=&nbsp;<span class="keyword">true</span><span>&nbsp;&nbsp;</span></span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;<span class="comment">//&nbsp;使线程组中计数器-1</span><span>&nbsp;&nbsp;</span></span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;waitGroup.Done()&nbsp;&nbsp;</span></li><li><span>&#125;&nbsp;&nbsp;</span></li></ol></div><div><div><p>构造函数<code>NewShardTask</code>负责完成<code>ShardTask</code>的参数传入和状态量初始化，而<code>DoShardGet</code>方法实现了下载一个文件分片的完整步骤，从创建文件准备写入，到设定请求头，发出请求，最后读取响应体保存到文件。</p><p>此外，可见这里的<code>http.Client</code>对象中，我们将其<code>DisableKeepAlives</code>设为了<code>true</code>即关闭<code>keep-alive</code>，这是因为<strong>默认情况下Go语言的HTTP客户端会复用TCP连接，即使你多个线程发起请求，也会使用一个TCP连接进行</strong>。</p><p>而多线程下载需要<strong>每个线程持有一个单独的TCP连接</strong>来达到突破<code>cwnd</code>的限制，因此这里关闭<code>keep-alive</code>实现每个线程发起请求时，使用单独的TCP连接。</p><h3 data-id="heading-6">(2) <code>ParallelGetTask</code> - 一整个多线程下载任务</h3></div><div class="codeText"><div class="codeHead">C#代码</div><ol start="1" class="dp-c"><li class="alt"><span><span>package&nbsp;model&nbsp;&nbsp;</span></span></li><li><span>&nbsp;&nbsp;</span></li><li class="alt"><span>import&nbsp;(&nbsp;&nbsp;</span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;<span class="string">&quot;bufio&quot;</span><span>&nbsp;&nbsp;</span></span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;<span class="string">&quot;fmt&quot;</span><span>&nbsp;&nbsp;</span></span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;<span class="string">&quot;gitee.com/swsk33/shard-download-demo/util&quot;</span><span>&nbsp;&nbsp;</span></span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;<span class="string">&quot;github.com/fatih/color&quot;</span><span>&nbsp;&nbsp;</span></span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;<span class="string">&quot;io&quot;</span><span>&nbsp;&nbsp;</span></span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;<span class="string">&quot;net/http&quot;</span><span>&nbsp;&nbsp;</span></span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;<span class="string">&quot;os&quot;</span><span>&nbsp;&nbsp;</span></span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;<span class="string">&quot;path/filepath&quot;</span><span>&nbsp;&nbsp;</span></span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;<span class="string">&quot;strconv&quot;</span><span>&nbsp;&nbsp;</span></span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;<span class="string">&quot;sync&quot;</span><span>&nbsp;&nbsp;</span></span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;<span class="string">&quot;time&quot;</span><span>&nbsp;&nbsp;</span></span></li><li class="alt"><span>)&nbsp;&nbsp;</span></li><li><span>&nbsp;&nbsp;</span></li><li class="alt"><span><span class="comment">//&nbsp;ParallelGetTask&nbsp;多线程下载任务类，存放一个多线程下载任务的参数和状态量</span><span>&nbsp;&nbsp;</span></span></li><li><span>type&nbsp;ParallelGetTask&nbsp;<span class="keyword">struct</span><span>&nbsp;&#123;&nbsp;&nbsp;</span></span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;<span class="comment">//&nbsp;文件的下载链接</span><span>&nbsp;&nbsp;</span></span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;Url&nbsp;<span class="keyword">string</span><span>&nbsp;&nbsp;</span></span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;<span class="comment">//&nbsp;文件的最终保存位置</span><span>&nbsp;&nbsp;</span></span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;FilePath&nbsp;<span class="keyword">string</span><span>&nbsp;&nbsp;</span></span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;<span class="comment">//&nbsp;下载并发数</span><span>&nbsp;&nbsp;</span></span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;Concurrent&nbsp;<span class="keyword">int</span><span>&nbsp;&nbsp;</span></span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;<span class="comment">//&nbsp;下载的分片临时文件保存文件夹</span><span>&nbsp;&nbsp;</span></span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;TempFolder&nbsp;<span class="keyword">string</span><span>&nbsp;&nbsp;</span></span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;<span class="comment">//&nbsp;下载文件的总大小</span><span>&nbsp;&nbsp;</span></span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;TotalSize&nbsp;int64&nbsp;&nbsp;</span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;<span class="comment">//&nbsp;全部的下载分片任务参数列表</span><span>&nbsp;&nbsp;</span></span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;ShardTaskList&nbsp;[]*ShardTask&nbsp;&nbsp;</span></li><li class="alt"><span>&#125;&nbsp;&nbsp;</span></li><li><span>&nbsp;&nbsp;</span></li><li class="alt"><span><span class="comment">//&nbsp;NewParallelGetTask&nbsp;构造函数</span><span>&nbsp;&nbsp;</span></span></li><li><span>func&nbsp;NewParallelGetTask(url&nbsp;<span class="keyword">string</span><span>,&nbsp;filePath&nbsp;</span><span class="keyword">string</span><span>,&nbsp;concurrent&nbsp;</span><span class="keyword">int</span><span>,&nbsp;tempFolder&nbsp;</span><span class="keyword">string</span><span>)&nbsp;*ParallelGetTask&nbsp;&#123;&nbsp;&nbsp;</span></span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;<span class="keyword">return</span><span>&nbsp;&amp;ParallelGetTask&#123;&nbsp;&nbsp;</span></span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span class="comment">//&nbsp;参数赋值</span><span>&nbsp;&nbsp;</span></span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Url:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;url,&nbsp;&nbsp;</span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;FilePath:&nbsp;&nbsp;&nbsp;filePath,&nbsp;&nbsp;</span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Concurrent:&nbsp;concurrent,&nbsp;&nbsp;</span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;TempFolder:&nbsp;tempFolder,&nbsp;&nbsp;</span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span class="comment">//&nbsp;初始化状态量</span><span>&nbsp;&nbsp;</span></span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;TotalSize:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;0,&nbsp;&nbsp;</span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;ShardTaskList:&nbsp;make([]*ShardTask,&nbsp;0),&nbsp;&nbsp;</span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;&#125;&nbsp;&nbsp;</span></li><li class="alt"><span>&#125;&nbsp;&nbsp;</span></li><li><span>&nbsp;&nbsp;</span></li><li class="alt"><span><span class="comment">//&nbsp;发送HEAD请求获取待下载文件的大小</span><span>&nbsp;&nbsp;</span></span></li><li><span>func&nbsp;(task&nbsp;*ParallelGetTask)&nbsp;getLength()&nbsp;error&nbsp;&#123;&nbsp;&nbsp;</span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;<span class="comment">//&nbsp;发送请求</span><span>&nbsp;&nbsp;</span></span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;response,&nbsp;e&nbsp;:=&nbsp;http.Head(task.Url)&nbsp;&nbsp;</span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;<span class="keyword">if</span><span>&nbsp;e&nbsp;!=&nbsp;nil&nbsp;&#123;&nbsp;&nbsp;</span></span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;color.Red(<span class="string">&quot;发送HEAD请求出错！&quot;</span><span>)&nbsp;&nbsp;</span></span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span class="keyword">return</span><span>&nbsp;e&nbsp;&nbsp;</span></span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;&#125;&nbsp;&nbsp;</span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;<span class="comment">//&nbsp;读取并设定长度</span><span>&nbsp;&nbsp;</span></span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;task.TotalSize&nbsp;=&nbsp;response.ContentLength&nbsp;&nbsp;</span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;<span class="keyword">return</span><span>&nbsp;nil&nbsp;&nbsp;</span></span></li><li><span>&#125;&nbsp;&nbsp;</span></li><li class="alt"><span>&nbsp;&nbsp;</span></li><li><span><span class="comment">//&nbsp;根据待下载文件的大小和设定的并发数，创建每个分片任务对象</span><span>&nbsp;&nbsp;</span></span></li><li class="alt"><span>func&nbsp;(task&nbsp;*ParallelGetTask)&nbsp;allocateTask()&nbsp;&#123;&nbsp;&nbsp;</span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;<span class="comment">//&nbsp;如果并发数大于总大小，则进行调整</span><span>&nbsp;&nbsp;</span></span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;<span class="keyword">if</span><span>&nbsp;int64(task.Concurrent)&nbsp;&gt;&nbsp;task.TotalSize&nbsp;&#123;&nbsp;&nbsp;</span></span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;task.Concurrent&nbsp;=&nbsp;<span class="keyword">int</span><span>(task.TotalSize)&nbsp;&nbsp;</span></span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;&#125;&nbsp;&nbsp;</span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;<span class="comment">//&nbsp;开始计算每个分片的下载范围</span><span>&nbsp;&nbsp;</span></span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;eachSize&nbsp;:=&nbsp;task.TotalSize&nbsp;/&nbsp;int64(task.Concurrent)&nbsp;&nbsp;</span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;<span class="comment">//&nbsp;创建任务对象</span><span>&nbsp;&nbsp;</span></span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;<span class="keyword">for</span><span>&nbsp;i&nbsp;:=&nbsp;0;&nbsp;i&nbsp;&lt;&nbsp;task.Concurrent;&nbsp;i++&nbsp;&#123;&nbsp;&nbsp;</span></span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;task.ShardTaskList&nbsp;=&nbsp;append(task.ShardTaskList,&nbsp;NewShardTask(task.Url,&nbsp;i+1,&nbsp;filepath.Join(task.TempFolder,&nbsp;strconv.Itoa(i+1)),&nbsp;int64(i)*eachSize,&nbsp;int64(i+1)*eachSize-1))&nbsp;&nbsp;</span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;&#125;&nbsp;&nbsp;</span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;<span class="comment">//&nbsp;处理末尾部分</span><span>&nbsp;&nbsp;</span></span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;<span class="keyword">if</span><span>&nbsp;task.TotalSize%int64(task.Concurrent)&nbsp;!=&nbsp;0&nbsp;&#123;&nbsp;&nbsp;</span></span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;task.ShardTaskList[task.Concurrent-1].RangeEnd&nbsp;=&nbsp;task.TotalSize&nbsp;-&nbsp;1&nbsp;&nbsp;</span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;&#125;&nbsp;&nbsp;</span></li><li><span>&#125;&nbsp;&nbsp;</span></li><li class="alt"><span>&nbsp;&nbsp;</span></li><li><span><span class="comment">//&nbsp;根据任务列表进行多线程分片下载操作</span><span>&nbsp;&nbsp;</span></span></li><li class="alt"><span>func&nbsp;(task&nbsp;*ParallelGetTask)&nbsp;downloadShard()&nbsp;&#123;&nbsp;&nbsp;</span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;<span class="comment">//&nbsp;创建线程组</span><span>&nbsp;&nbsp;</span></span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;waitGroup&nbsp;:=&nbsp;&amp;sync.WaitGroup&#123;&#125;&nbsp;&nbsp;</span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;<span class="comment">//&nbsp;开始执行全部分片下载线程</span><span>&nbsp;&nbsp;</span></span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;<span class="keyword">for</span><span>&nbsp;_,&nbsp;task&nbsp;:=&nbsp;range&nbsp;task.ShardTaskList&nbsp;&#123;&nbsp;&nbsp;</span></span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;go&nbsp;task.DoShardGet(waitGroup)&nbsp;&nbsp;</span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;waitGroup.Add(1)&nbsp;&nbsp;</span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;&#125;&nbsp;&nbsp;</span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;<span class="comment">//&nbsp;等待全部下载完成</span><span>&nbsp;&nbsp;</span></span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;waitGroup.Wait()&nbsp;&nbsp;</span></li><li class="alt"><span>&#125;&nbsp;&nbsp;</span></li><li><span>&nbsp;&nbsp;</span></li><li class="alt"><span><span class="comment">//&nbsp;下载完成后，合并分片文件</span><span>&nbsp;&nbsp;</span></span></li><li><span>func&nbsp;(task&nbsp;*ParallelGetTask)&nbsp;mergeFile()&nbsp;error&nbsp;&#123;&nbsp;&nbsp;</span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;<span class="comment">//&nbsp;创建目的文件</span><span>&nbsp;&nbsp;</span></span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;targetFile,&nbsp;e&nbsp;:=&nbsp;os.OpenFile(task.FilePath,&nbsp;os.O_CREATE&#124;os.O_WRONLY&#124;os.O_APPEND,&nbsp;0755)&nbsp;&nbsp;</span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;<span class="keyword">if</span><span>&nbsp;e&nbsp;!=&nbsp;nil&nbsp;&#123;&nbsp;&nbsp;</span></span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;color.Red(<span class="string">&quot;创建目标文件出错！&quot;</span><span>)&nbsp;&nbsp;</span></span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span class="keyword">return</span><span>&nbsp;e&nbsp;&nbsp;</span></span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;&#125;&nbsp;&nbsp;</span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;<span class="comment">//&nbsp;创建写入器</span><span>&nbsp;&nbsp;</span></span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;writer&nbsp;:=&nbsp;bufio.NewWriter(targetFile)&nbsp;&nbsp;</span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;<span class="comment">//&nbsp;准备读取每个分片文件</span><span>&nbsp;&nbsp;</span></span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;<span class="keyword">for</span><span>&nbsp;_,&nbsp;shard&nbsp;:=&nbsp;range&nbsp;task.ShardTaskList&nbsp;&#123;&nbsp;&nbsp;</span></span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;shardFile,&nbsp;e&nbsp;:=&nbsp;os.OpenFile(shard.ShardFilePath,&nbsp;os.O_RDONLY,&nbsp;0755)&nbsp;&nbsp;</span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span class="keyword">if</span><span>&nbsp;e&nbsp;!=&nbsp;nil&nbsp;&#123;&nbsp;&nbsp;</span></span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;color.Red(<span class="string">&quot;读取分片文件出错！&quot;</span><span>)&nbsp;&nbsp;</span></span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span class="keyword">return</span><span>&nbsp;e&nbsp;&nbsp;</span></span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&#125;&nbsp;&nbsp;</span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;reader&nbsp;:=&nbsp;bufio.NewReader(shardFile)&nbsp;&nbsp;</span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;readBuffer&nbsp;:=&nbsp;make([]<span class="keyword">byte</span><span>,&nbsp;1024*1024)&nbsp;&nbsp;</span></span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span class="keyword">for</span><span>&nbsp;&#123;&nbsp;&nbsp;</span></span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span class="comment">//&nbsp;读取每个分片文件，一次读取1KB</span><span>&nbsp;&nbsp;</span></span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;readSize,&nbsp;readError&nbsp;:=&nbsp;reader.Read(readBuffer)&nbsp;&nbsp;</span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span class="comment">//&nbsp;处理结束或错误</span><span>&nbsp;&nbsp;</span></span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span class="keyword">if</span><span>&nbsp;readError&nbsp;!=&nbsp;nil&nbsp;&#123;&nbsp;&nbsp;</span></span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span class="keyword">if</span><span>&nbsp;readError&nbsp;==&nbsp;io.EOF&nbsp;&#123;&nbsp;&nbsp;</span></span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span class="keyword">break</span><span>&nbsp;&nbsp;</span></span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&#125;&nbsp;<span class="keyword">else</span><span>&nbsp;&#123;&nbsp;&nbsp;</span></span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;color.Red(<span class="string">&quot;读取分片文件出错！&quot;</span><span>)&nbsp;&nbsp;</span></span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span class="keyword">return</span><span>&nbsp;readError&nbsp;&nbsp;</span></span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&#125;&nbsp;&nbsp;</span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&#125;&nbsp;&nbsp;</span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span class="comment">//&nbsp;写入到最终合并的文件</span><span>&nbsp;&nbsp;</span></span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;_,&nbsp;writeError&nbsp;:=&nbsp;writer.Write(readBuffer[0:readSize])&nbsp;&nbsp;</span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span class="keyword">if</span><span>&nbsp;writeError&nbsp;!=&nbsp;nil&nbsp;&#123;&nbsp;&nbsp;</span></span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;color.Red(<span class="string">&quot;写入合并文件出错！&quot;</span><span>)&nbsp;&nbsp;</span></span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span class="keyword">return</span><span>&nbsp;writeError&nbsp;&nbsp;</span></span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&#125;&nbsp;&nbsp;</span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;_&nbsp;=&nbsp;writer.Flush()&nbsp;&nbsp;</span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&#125;&nbsp;&nbsp;</span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span class="comment">//&nbsp;关闭分片文件资源</span><span>&nbsp;&nbsp;</span></span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;_&nbsp;=&nbsp;shardFile.Close()&nbsp;&nbsp;</span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;&#125;&nbsp;&nbsp;</span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;<span class="comment">//&nbsp;关闭目的文件资源</span><span>&nbsp;&nbsp;</span></span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;_&nbsp;=&nbsp;targetFile.Close()&nbsp;&nbsp;</span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;<span class="keyword">return</span><span>&nbsp;nil&nbsp;&nbsp;</span></span></li><li><span>&#125;&nbsp;&nbsp;</span></li><li class="alt"><span>&nbsp;&nbsp;</span></li><li><span><span class="comment">//&nbsp;删除分片临时文件</span><span>&nbsp;&nbsp;</span></span></li><li class="alt"><span>func&nbsp;(task&nbsp;*ParallelGetTask)&nbsp;cleanShard()&nbsp;error&nbsp;&#123;&nbsp;&nbsp;</span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;<span class="keyword">for</span><span>&nbsp;_,&nbsp;shard&nbsp;:=&nbsp;range&nbsp;task.ShardTaskList&nbsp;&#123;&nbsp;&nbsp;</span></span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;e&nbsp;:=&nbsp;os.Remove(shard.ShardFilePath)&nbsp;&nbsp;</span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span class="keyword">if</span><span>&nbsp;e&nbsp;!=&nbsp;nil&nbsp;&#123;&nbsp;&nbsp;</span></span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;color.Red(<span class="string">&quot;删除分片临时文件%s出错！&quot;</span><span>,&nbsp;shard.ShardFilePath)&nbsp;&nbsp;</span></span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span class="keyword">return</span><span>&nbsp;e&nbsp;&nbsp;</span></span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&#125;&nbsp;&nbsp;</span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;&#125;&nbsp;&nbsp;</span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;<span class="keyword">return</span><span>&nbsp;nil&nbsp;&nbsp;</span></span></li><li><span>&#125;&nbsp;&nbsp;</span></li><li class="alt"><span>&nbsp;&nbsp;</span></li><li><span><span class="comment">//&nbsp;在一个新线程中，实时输出每个分片的下载进度和总进度</span><span>&nbsp;&nbsp;</span></span></li><li class="alt"><span>func&nbsp;(task&nbsp;*ParallelGetTask)&nbsp;printTotalProcess()&nbsp;&#123;&nbsp;&nbsp;</span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;go&nbsp;func()&nbsp;&#123;&nbsp;&nbsp;</span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span class="comment">//&nbsp;上一次统计时的已下载大小，用于计算速度</span><span>&nbsp;&nbsp;</span></span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;var&nbsp;lastDownloadSize&nbsp;int64&nbsp;=&nbsp;0&nbsp;&nbsp;</span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span class="keyword">for</span><span>&nbsp;&#123;&nbsp;&nbsp;</span></span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span class="comment">//&nbsp;如果全部任务完成则结束输出，并统计并发数</span><span>&nbsp;&nbsp;</span></span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;allDone&nbsp;:=&nbsp;<span class="keyword">true</span><span>&nbsp;&nbsp;</span></span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span class="comment">//&nbsp;当前并发数</span><span>&nbsp;&nbsp;</span></span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;currentTaskCount&nbsp;:=&nbsp;0&nbsp;&nbsp;</span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span class="keyword">for</span><span>&nbsp;_,&nbsp;shardTask&nbsp;:=&nbsp;range&nbsp;task.ShardTaskList&nbsp;&#123;&nbsp;&nbsp;</span></span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span class="keyword">if</span><span>&nbsp;!shardTask.TaskDone&nbsp;&#123;&nbsp;&nbsp;</span></span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;allDone&nbsp;=&nbsp;<span class="keyword">false</span><span>&nbsp;&nbsp;</span></span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;currentTaskCount&nbsp;+=&nbsp;1&nbsp;&nbsp;</span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&#125;&nbsp;&nbsp;</span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&#125;&nbsp;&nbsp;</span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span class="keyword">if</span><span>&nbsp;allDone&nbsp;&#123;&nbsp;&nbsp;</span></span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span class="keyword">break</span><span>&nbsp;&nbsp;</span></span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&#125;&nbsp;&nbsp;</span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span class="comment">//&nbsp;统计所有分片已下载大小之和</span><span>&nbsp;&nbsp;</span></span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;var&nbsp;totalDownloadSize&nbsp;int64&nbsp;=&nbsp;0&nbsp;&nbsp;</span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span class="keyword">for</span><span>&nbsp;_,&nbsp;shardTask&nbsp;:=&nbsp;range&nbsp;task.ShardTaskList&nbsp;&#123;&nbsp;&nbsp;</span></span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;totalDownloadSize&nbsp;+=&nbsp;shardTask.DownloadSize&nbsp;&nbsp;</span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&#125;&nbsp;&nbsp;</span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span class="comment">//&nbsp;计算速度</span><span>&nbsp;&nbsp;</span></span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;currentDownload&nbsp;:=&nbsp;totalDownloadSize&nbsp;-&nbsp;lastDownloadSize&nbsp;&nbsp;</span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;lastDownloadSize&nbsp;=&nbsp;totalDownloadSize&nbsp;&nbsp;</span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;speedString&nbsp;:=&nbsp;util.ComputeSpeed(currentDownload,&nbsp;300)&nbsp;&nbsp;</span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span class="comment">//&nbsp;输出到控制台</span><span>&nbsp;&nbsp;</span></span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;fmt.Printf(<span class="string">&quot;&#92;r当前并发数：%3d&nbsp;速度：%s&nbsp;总进度：%3.2f%%&quot;</span><span>,&nbsp;currentTaskCount,&nbsp;speedString,&nbsp;float32(totalDownloadSize)/float32(task.TotalSize)*100)&nbsp;&nbsp;</span></span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span class="comment">//&nbsp;等待300ms</span><span>&nbsp;&nbsp;</span></span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;time.Sleep(300&nbsp;*&nbsp;time.Millisecond)&nbsp;&nbsp;</span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&#125;&nbsp;&nbsp;</span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;&#125;()&nbsp;&nbsp;</span></li><li><span>&#125;&nbsp;&nbsp;</span></li><li class="alt"><span>&nbsp;&nbsp;</span></li><li><span><span class="comment">//&nbsp;Run&nbsp;开始执行整个分片多线程下载任务</span><span>&nbsp;&nbsp;</span></span></li><li class="alt"><span>func&nbsp;(task&nbsp;*ParallelGetTask)&nbsp;Run()&nbsp;error&nbsp;&#123;&nbsp;&nbsp;</span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;<span class="comment">//&nbsp;获取文件大小</span><span>&nbsp;&nbsp;</span></span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;e&nbsp;:=&nbsp;task.getLength()&nbsp;&nbsp;</span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;<span class="keyword">if</span><span>&nbsp;e&nbsp;!=&nbsp;nil&nbsp;&#123;&nbsp;&nbsp;</span></span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;color.Red(<span class="string">&quot;%s&quot;</span><span>,&nbsp;e)&nbsp;&nbsp;</span></span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span class="keyword">return</span><span>&nbsp;e&nbsp;&nbsp;</span></span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;&#125;&nbsp;&nbsp;</span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;color.HiYellow(<span class="string">&quot;已获取到下载文件大小：%d字节&quot;</span><span>,&nbsp;task.TotalSize)&nbsp;&nbsp;</span></span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;<span class="comment">//&nbsp;分配任务</span><span>&nbsp;&nbsp;</span></span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;task.allocateTask()&nbsp;&nbsp;</span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;color.HiYellow(<span class="string">&quot;已完成分片任务分配，共计%d个任务&quot;</span><span>,&nbsp;len(task.ShardTaskList))&nbsp;&nbsp;</span></span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;<span class="comment">//&nbsp;开启进度输出</span><span>&nbsp;&nbsp;</span></span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;task.printTotalProcess()&nbsp;&nbsp;</span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;<span class="comment">//&nbsp;开始下载分片</span><span>&nbsp;&nbsp;</span></span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;task.downloadShard()&nbsp;&nbsp;</span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;color.HiYellow(<span class="string">&quot;&#92;n所有分片已下载完成！&quot;</span><span>)&nbsp;&nbsp;</span></span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;<span class="comment">//&nbsp;开始合并文件</span><span>&nbsp;&nbsp;</span></span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;e&nbsp;=&nbsp;task.mergeFile()&nbsp;&nbsp;</span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;<span class="keyword">if</span><span>&nbsp;e&nbsp;!=&nbsp;nil&nbsp;&#123;&nbsp;&nbsp;</span></span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;color.Red(<span class="string">&quot;%s&quot;</span><span>,&nbsp;e)&nbsp;&nbsp;</span></span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span class="keyword">return</span><span>&nbsp;e&nbsp;&nbsp;</span></span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;&#125;&nbsp;&nbsp;</span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;color.HiYellow(<span class="string">&quot;合并分片完成！&quot;</span><span>)&nbsp;&nbsp;</span></span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;<span class="comment">//&nbsp;清理临时分片文件</span><span>&nbsp;&nbsp;</span></span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;e&nbsp;=&nbsp;task.cleanShard()&nbsp;&nbsp;</span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;<span class="keyword">if</span><span>&nbsp;e&nbsp;!=&nbsp;nil&nbsp;&#123;&nbsp;&nbsp;</span></span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;color.Red(<span class="string">&quot;%s&quot;</span><span>,&nbsp;e)&nbsp;&nbsp;</span></span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span class="keyword">return</span><span>&nbsp;e&nbsp;&nbsp;</span></span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;&#125;&nbsp;&nbsp;</span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;color.HiYellow(<span class="string">&quot;清理分片临时文件完成！&quot;</span><span>)&nbsp;&nbsp;</span></span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;color.Green(<span class="string">&quot;分片下载任务完成！&quot;</span><span>)&nbsp;&nbsp;</span></span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;<span class="keyword">return</span><span>&nbsp;nil&nbsp;&nbsp;</span></span></li><li class="alt"><span>&#125;&nbsp;&nbsp;</span></li></ol></div><p><span style="color: rgb(37, 41, 51); font-family: -apple-system, system-ui, &quot;Segoe UI&quot;, Roboto, Ubuntu, Cantarell, &quot;Noto Sans&quot;, sans-serif, BlinkMacSystemFont, &quot;Helvetica Neue&quot;, &quot;PingFang SC&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei&quot;, Arial; font-size: 16px;">上述</span><code style="font-family: Menlo, Monaco, Consolas, &quot;Courier New&quot;, monospace; font-size: 0.87em; word-break: break-word; border-radius: 2px; overflow-x: auto; background-color: rgb(255, 245, 245); color: rgb(255, 80, 44); padding: 0.065em 0.4em;">printTotalProcess</code><span style="color: rgb(37, 41, 51); font-family: -apple-system, system-ui, &quot;Segoe UI&quot;, Roboto, Ubuntu, Cantarell, &quot;Noto Sans&quot;, sans-serif, BlinkMacSystemFont, &quot;Helvetica Neue&quot;, &quot;PingFang SC&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei&quot;, Arial; font-size: 16px;">函数中，</span><code style="font-family: Menlo, Monaco, Consolas, &quot;Courier New&quot;, monospace; font-size: 0.87em; word-break: break-word; border-radius: 2px; overflow-x: auto; background-color: rgb(255, 245, 245); color: rgb(255, 80, 44); padding: 0.065em 0.4em;">util.ComputeSpeed</code><span style="color: rgb(37, 41, 51); font-family: -apple-system, system-ui, &quot;Segoe UI&quot;, Roboto, Ubuntu, Cantarell, &quot;Noto Sans&quot;, sans-serif, BlinkMacSystemFont, &quot;Helvetica Neue&quot;, &quot;PingFang SC&quot;, &quot;Hiragino Sans GB&quot;, &quot;Microsoft YaHei&quot;, Arial; font-size: 16px;">函数用于计算下载速度并自动转换为可读单位，代码如下：</span></p><div class="codeText"><div class="codeHead">C#代码</div><ol start="1" class="dp-c"><li class="alt"><span><span>package&nbsp;util&nbsp;&nbsp;</span></span></li><li><span>&nbsp;&nbsp;</span></li><li class="alt"><span>import&nbsp;(&nbsp;&nbsp;</span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;<span class="string">&quot;fmt&quot;</span><span>&nbsp;&nbsp;</span></span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;<span class="string">&quot;math&quot;</span><span>&nbsp;&nbsp;</span></span></li><li><span>)&nbsp;&nbsp;</span></li><li class="alt"><span>&nbsp;&nbsp;</span></li><li><span><span class="comment">//&nbsp;关于单位的实用工具函数</span><span>&nbsp;&nbsp;</span></span></li><li class="alt"><span>&nbsp;&nbsp;</span></li><li><span><span class="comment">//&nbsp;ComputeSpeed&nbsp;计算网络速度</span><span>&nbsp;&nbsp;</span></span></li><li class="alt"><span><span class="comment">//&nbsp;size&nbsp;一段时间内下载的数据大小，单位字节</span><span>&nbsp;&nbsp;</span></span></li><li><span><span class="comment">//&nbsp;timeElapsed&nbsp;经过的时间长度，单位毫秒</span><span>&nbsp;&nbsp;</span></span></li><li class="alt"><span><span class="comment">//&nbsp;返回计算得到的网速，会自动换算单位</span><span>&nbsp;&nbsp;</span></span></li><li><span>func&nbsp;ComputeSpeed(size&nbsp;int64,&nbsp;timeElapsed&nbsp;<span class="keyword">int</span><span>)&nbsp;</span><span class="keyword">string</span><span>&nbsp;&#123;&nbsp;&nbsp;</span></span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;bytePerSecond&nbsp;:=&nbsp;size&nbsp;/&nbsp;int64(timeElapsed)&nbsp;*&nbsp;1000&nbsp;&nbsp;</span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;<span class="keyword">if</span><span>&nbsp;0&nbsp;&lt;=&nbsp;bytePerSecond&nbsp;&amp;&amp;&nbsp;bytePerSecond&nbsp;&lt;=&nbsp;1024&nbsp;&#123;&nbsp;&nbsp;</span></span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span class="keyword">return</span><span>&nbsp;fmt.Sprintf(</span><span class="string">&quot;%4d&nbsp;Byte/s&quot;</span><span>,&nbsp;bytePerSecond)&nbsp;&nbsp;</span></span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;&#125;&nbsp;&nbsp;</span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;<span class="keyword">if</span><span>&nbsp;bytePerSecond&nbsp;&gt;&nbsp;1024&nbsp;&amp;&amp;&nbsp;bytePerSecond&nbsp;&lt;=&nbsp;int64(math.Pow(1024,&nbsp;2))&nbsp;&#123;&nbsp;&nbsp;</span></span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span class="keyword">return</span><span>&nbsp;fmt.Sprintf(</span><span class="string">&quot;%6.2f&nbsp;KB/s&quot;</span><span>,&nbsp;float64(bytePerSecond)/1024)&nbsp;&nbsp;</span></span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;&#125;&nbsp;&nbsp;</span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;<span class="keyword">if</span><span>&nbsp;bytePerSecond&nbsp;&gt;&nbsp;1024*1024&nbsp;&amp;&amp;&nbsp;bytePerSecond&nbsp;&lt;=&nbsp;int64(math.Pow(1024,&nbsp;3))&nbsp;&#123;&nbsp;&nbsp;</span></span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span class="keyword">return</span><span>&nbsp;fmt.Sprintf(</span><span class="string">&quot;%6.2f&nbsp;MB/s&quot;</span><span>,&nbsp;float64(bytePerSecond)/math.Pow(1024,&nbsp;2))&nbsp;&nbsp;</span></span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;&#125;&nbsp;&nbsp;</span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;<span class="keyword">return</span><span>&nbsp;fmt.Sprintf(</span><span class="string">&quot;%6.2f&nbsp;GB/s&quot;</span><span>,&nbsp;float64(bytePerSecond)/math.Pow(1024,&nbsp;3))&nbsp;&nbsp;</span></span></li><li><span>&#125;&nbsp;&nbsp;</span></li></ol></div><p>&nbsp;</p><div><div><p>可见通过构造函数<code>NewParallelGetTask</code>完成参数传递和状态量设定后，其它每个私有函数都对应我们多线程下载中的一个步骤，最后由公开函数<code>Run</code>统筹组织起所有的步骤，完成整个多线程下载任务。</p><h2 data-id="heading-7">3，实现效果</h2><p>现在我们在<code>main</code>函数中创建一个<code>ParallelGetTask</code>对象，设定好参数后调用其<code>Run</code>方法即可开始多线程下载文件的任务：</p></div><div class="codeText"><div class="codeHead">C#代码</div><ol start="1" class="dp-c"><li class="alt"><span><span>package&nbsp;main&nbsp;&nbsp;</span></span></li><li><span>&nbsp;&nbsp;</span></li><li class="alt"><span>import&nbsp;(&nbsp;&nbsp;</span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;<span class="string">&quot;gitee.com/swsk33/shard-download-demo/model&quot;</span><span>&nbsp;&nbsp;</span></span></li><li class="alt"><span>)&nbsp;&nbsp;</span></li><li><span>&nbsp;&nbsp;</span></li><li class="alt"><span>func&nbsp;main()&nbsp;&#123;&nbsp;&nbsp;</span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;<span class="comment">//&nbsp;创建任务</span><span>&nbsp;&nbsp;</span></span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;task&nbsp;:=&nbsp;model.NewParallelGetTask(&nbsp;&nbsp;</span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span class="string">&quot;https://github.com/jgraph/drawio-desktop/releases/download/v24.7.17/draw.io-24.7.17-windows-installer.exe&quot;</span><span>,&nbsp;&nbsp;</span></span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span class="string">&quot;C:&#92;&#92;Users&#92;&#92;swsk33&#92;&#92;Downloads&#92;&#92;draw.io.exe&quot;</span><span>,&nbsp;&nbsp;</span></span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;64,&nbsp;&nbsp;</span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<span class="string">&quot;C:&#92;&#92;Users&#92;&#92;swsk33&#92;&#92;Downloads&#92;&#92;temp&quot;</span><span>,&nbsp;&nbsp;</span></span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;)&nbsp;&nbsp;</span></li><li class="alt"><span>&nbsp;&nbsp;&nbsp;&nbsp;<span class="comment">//&nbsp;执行任务</span><span>&nbsp;&nbsp;</span></span></li><li><span>&nbsp;&nbsp;&nbsp;&nbsp;_&nbsp;=&nbsp;task.Run()&nbsp;&nbsp;</span></li><li class="alt"><span>&#125;&nbsp;&nbsp;</span></li></ol></div><div>&nbsp;</div></div></div></div></div></div>
]]>
</description>
</item><item>
<link>http://liuxinxiu.com:80/s//#blogcomment</link>
<title><![CDATA[[评论] Go实现多线程分片下载文件]]></title> 
<author> &lt;user@domain.com&gt;</author>
<category><![CDATA[评论]]></category>
<pubDate>Thu, 01 Jan 1970 00:00:00 +0000</pubDate> 
<guid>http://liuxinxiu.com:80/s//#blogcomment</guid> 
<description>
<![CDATA[ 
	
]]>
</description>
</item>
</channel>
</rss>