Patching the Diablo 3 Game Client
Every Beta Patch, I noticed that some websites were able to obtain the Patch MPQs before my Beta game client was able to download them. Some folks on various forums posted as much, and Bashiok even commented on this on Twitter:
@crypto2000 @Angryrobotics Nah, they just know where to look. The patch is technically public, just not enabled in the game launcher.
@crypto2000 @angryrobotics Our patch clients are always available before the patch is officially released, if you know where to look.
I found those tweets quite annoying because rather than informing the public exactly where the publicly available patches were, Bashiok confirmed that they did exist, and implied that you were just not cool enough to be in the club with those in-the-know.
A friend from the great d3db.com website gave me the links to the Patch MPQ downloads which I really appreciated. But what I really wanted was to understand more about the patching process so that I would not have to rely on others should the process change down the road.
Tools of the Trade
One way to understand how the game client is communicating with the Blizzard back-end servers is to use a packet sniffer to collect all the network packets that go back and forth from your computer out to the Internet, and then filter those results to examine the ones that are Blizzard specific. It goes without saying that while you are doing this, it is best to be doing nothing else (no background downloads, Skype, IM, IRC, etc) to limit the amount of data captured and hence the amount of data you will need to sift through.
I decided to use Wireshark because it is free (for non-commercial use), it is open source, it is multi-platform, it is excellent and has been for a very long time. It's also the world's foremost network protocol analyzer. There's pretty much no reason to use anything else. For Windows environments, Wireshark comes with WinPcap which is the service that performs that actual packet capture for Wireshark. Note that if you don't want to start WinPcap on every startup, you can later start it with "net start NPF".
Analyzing packet data requires, at a minimum, a basic understanding of how networks work, and how IP and TCP work in particular.
When examining the data captured by Wireshark, you may want to filter out packets that aren't relevant to what you are trying to do. In this case, it means ignoring things like ARP, DHCP, ICMP, IGMP, MDNS, and NBNS packets. Also, when looking at a given packet, you should always ignore the Frame and Ethernet layers since those do not contain any information that is sent to or received from an internet connection.
Patching Process
First, let me start by saying that this is not a complete picture of how the entire patching process works. It is an attempt to describe enough to allow folks to download the Patch MPQs before they are available from the Launcher.
The "Diablo III Beta Launcher.exe" process makes use of the Battle.net "Agent" and "Client" executables (located under the "C:\ProgramData\Battle.net" directory by default in Windows 7). These programs will establish a number of connections with Blizzard back-end servers to retrieve version information.
The first such request gets the trackers from "us.tracker.worldofwarcraft.com". Note that the D3 Beta currently uses the trackers from a "worldofwarcraft.com" host. This request returns a list of peers (includes IP and port information). Note that one of the GET params is "event=started". Here is a sample of this request and response:
GET /announce?info_hash=DOTRYWORLDOFWARCRAFT&peer_id=E231E159953A6B51720F&key=58748130&port=3724&uploaded=0&downloaded=0&left=1&event=started&stats=app=AGNT,ver=0,lang=enUS,stg=1000,sta=0 HTTP/1.1
User-Agent: Blizzard Web Client
Host: us.tracker.worldofwarcraft.com:3724
Cache-Control: no-cache
Cookie: __utma=155754724.1758108837.1326246656.1326246656.1326333957.2; __utmz=155754724.1326246656.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none)
HTTP/1.1 200 OK
Server: Battle.net HTTP
d8:intervali1800e2:ip11:12.14.156.25:peersld2:ip14:114.234.243.687:peer id20:61505F56C0699CFC7DBA4:porti3724eed2:ip12:112.3.191.757:peer id20:B2C0C779620BC37351DF4:porti3724eed2:ip12:58.141.138.77:peer id20:3108179E181D3E7B82AC4:porti3724eed2:ip15:190.113.143.1957:peer id20:209E9FA44784C12AD82D4:porti3724eed2:ip13:113.13.22.1707:peer id20:497D8E7F059C76FDB4F24:porti3724eed2:ip12:27.192.60.127:peer (...)
This is followed by a GET to the same host and service, but with a GET param of "event=stopped". The response is a simple acknowledgment that closes the connection. Here is a sample of this request and response:
GET /announce?info_hash=DOTRYWORLDOFWARCRAFT&peer_id=E231E159953A6B51720F&key=58748130&port=3724&uploaded=0&downloaded=1&left=0&event=stopped HTTP/1.1
User-Agent: Blizzard Web Client
Host: us.tracker.worldofwarcraft.com:3724
Cache-Control: no-cache
Cookie: __utma=155754724.1758108837.1326246656.1326246656.1326333957.2; __utmz=155754724.1326246656.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none)
HTTP/1.1 200 OK
Server: Battle.net HTTP
Connection: close
This is followed by a POST to "enUS.patch.battle.net:1119" to obtain patch information for "Bnet" and "Agnt" on behalf of the "Agnt" (Agent). This happens before the game patch requests since the Agent and Bnet files may need to be patched before the new client files can be downloaded and patched. Note that the version "516" is the current version of the Agent that you will find in the "C:\ProgramData\Battle.net\Agent\Agent.516" directory. Here is a sample of this request and response:
POST /patch HTTP/1.1
Content-Type: text/html
Content-Length: 142
User-Agent: Blizzard Web Client
Host: enUS.patch.battle.net:1119
Cache-Control: no-cache
<version program="Agnt">
<record program="Bnet" component="Win" version="1" />
<record program="Agnt" component="Win" version="516" />
</version>
HTTP/1.1 200 OK
Server: Protocol HTTP
Content-Length: 259
Connection: close
Content-Type: application/xml
<patch>
<record program="Bnet" component="Win">
http://attdist.blizzard.com/sc2patch/update/B16117-5C52FCF1/sc2-1-22280-x86-Win-enUS-bnet-bin-component-dl;sc2-1-22280-x86-Win-enUS-bnet-bin-component-dl;69B17A0F007E49F5B37E2876C87A17DA;22280
</record>
</patch>
In this particular case, the Agent did not have a new version and hence the "attdist.blizzard.com" server was not contacted.
This request was followed by a very similar POST to the same host to obtain patch information for "Bnet" and "Clnt" on behalf of the "Clnt" (Client). Note that the version "812" is the current version of the Agent which you will find in the "C:\ProgramData\Battle.net\Client\Blizzard Launcher.812" directory. Here is a sample of this request and response:
POST /patch HTTP/1.1
Content-Type: text/html
Content-Length: 142
User-Agent: Blizzard Web Client
Host: enUS.patch.battle.net:1119
Cache-Control: no-cache
<version program="Clnt">
<record program="Bnet" component="Win" version="1" />
<record program="Clnt" component="Win" version="812" />
</version>
HTTP/1.1 200 OK
Server: Protocol HTTP
Content-Length: 259
Connection: close
Content-Type: application/xml
<patch>
<record program="Bnet" component="Win">
http://attdist.blizzard.com/sc2patch/update/B16117-5C52FCF1/sc2-1-22280-x86-Win-enUS-bnet-bin-component-dl;sc2-1-22280-x86-Win-enUS-bnet-bin-component-dl;69B17A0F007E49F5B37E2876C87A17DA;22280
</record>
</patch>
Note that in this particular case, the above response is identical to the one from the prior POST.
Finally, things get a little more interesting with a POST to "public-test.patch.battle.net:1119" to obtain PTR (and Beta) patch information for "Bnet" and "D3B" on behalf of the "D3B". Note that the version for "D3B" (Diablo3 Beta) is sent as "3" even though the currently installed version is higher. Also, sending the program as "D3" rather than "D3B" currently yield the same results. Here is a sample of this request and response:
POST /patch HTTP/1.1
Content-Type: text/html
Content-Length: 139
User-Agent: Blizzard Web Client
Host: public-test.patch.battle.net:1119
Cache-Control: no-cache
<version program="D3B">
<record program="Bnet" component="Win" version="1" />
<record program="D3B" component="enUS" version="3" />
</version>
HTTP/1.1 200 OK
Server: Protocol HTTP
Content-Length: 246
Connection: close
Content-Type: application/xml
<patch>
<record program="D3B" component="enUS">
http://ak.worldofwarcraft.com.edgesuite.net/d3-pod/20FB5BE9/NA/d3b-8296-D06777069202ACA0C804BC5BCC99909F.xml;
75071FE7E728942C01CEABD8B5F0BA77;941A7BD6BAF437A7A7773644DA9B795B;8296
</record>
</patch>
One thing to note is that the response points to an "edgesuite.net" host. This domain is owned by Akamai Technologies. Akamai is the largest edge content provider in the world. In essence, they host content for other companies and distribute that content throughout the world so that when someone requests it, they will obtain it from a nearby (relatively) provider. This greatly reduces Internet congestion (especially for large files and high quality video content) and improves the end-user experience.
The response above contains 4 pieces of information in the text of the XML record element. These are semi-colon delimited:
- http://ak.worldofwarcraft.com.edgesuite.net/d3-pod/20FB5BE9/NA/d3b-8296-D06777069202ACA0C804BC5BCC99909F.xml
This is a URL to the XML file that will contain additional download information. - 75071FE7E728942C01CEABD8B5F0BA77
This is the name of the "torrent" file that will be retrieved later if client game files need to be downloaded. - 941A7BD6BAF437A7A7773644DA9B795B
This is the name of the "mfil" file that will be retrieved later. - 8296
This is the version of the most recent (current) patch.
One very important note about these filenames is the the names of the files (excluding prefixes and suffixes) are the MD5s of the data. The game client actually validates that the file contents have the same MD5 value as the file names. If not, the game will abort with an MD5 validation error.
The next request is a GET for "ak.worldofwarcraft.com.edgesuite.net" to obtain the XML file from the previous request. Here is a sample of this request and response:
GET /d3-pod/20FB5BE9/NA/d3b-8296-D06777069202ACA0C804BC5BCC99909F.xml HTTP/1.1
User-Agent: Blizzard Web Client
Host: ak.worldofwarcraft.com.edgesuite.net
Cache-Control: no-cache
HTTP/1.1 200 OK
Server: Apache
ETag: "d06777069202aca0c804bc5bcc99909f:1327352410"
Last-Modified: Mon, 23 Jan 2012 21:00:10 GMT
Accept-Ranges: bytes
Content-Length: 1184
Content-Type: application/xml
Date: Thu, 26 Jan 2012 07:18:22 GMT
Connection: keep-alive
<config>
<versioninfo type="pod">
<version product="D3B">
<servers>
<server id="akamai" url="http://ak.worldofwarcraft.com.edgesuite.net/d3-pod/20FB5BE9/NA/7162.direct/"/>
</servers>
<thresholds>
<threshold speed="307200" red="11" yellow="11" />
<threshold speed="716800" red="3" yellow="11" />
<threshold speed="1280000" red="2" yellow="11" />
<threshold speed="3379200" red="1" yellow="11" />
<threshold speed="3379201" red="0" yellow="11" />
</thresholds>
<settings>
<setting name="patchapplicationstage" value="Recommended"/>
<setting name="json_blob_overrides" value="{"supported_locales" : ["deDE", "enGB", "esES", "frFR", "itIT", "plPL", "ptPT", "ruRU", "trTR", "enSG", "enUS", "esMX", "ptBR", "koKR", "zhTW", "zhCN"], "display_locales":["enSG", "enUS", "esMX", "ptBR"]}"/>
</settings>
</version>
</versioninfo>
</config>
The important piece of information in the response above is the server. The URL of this server will be the root URL for the direct Patch MPQ downloads:
http://ak.worldofwarcraft.com.edgesuite.net/d3-pod/20FB5BE9/NA/7162.direct/
If there are game client files that need to be downloaded, a new GET for the torrent file will be issued against the same server on the same TCP stream (which is left open by the server in this case). Here is a sample of this request and response. Note that I added spaces in the response to make the file easier to visually parse. Please refer to The BitTorrent Protocol Specification for more information.
GET /d3-pod/20FB5BE9/NA/7162.direct/d3b-8296-75071FE7E728942C01CEABD8B5F0BA77.torrent HTTP/1.1
User-Agent: Blizzard Web Client
Host: ak.worldofwarcraft.com.edgesuite.net
Cache-Control: no-cache
HTTP/1.1 200 OK
Server: Apache
ETag: "75071fe7e728942c01ceabd8b5f0ba77:1327352410"
Last-Modified: Mon, 23 Jan 2012 21:00:10 GMT
Accept-Ranges: bytes
Content-Length: 288685
Content-Type: text/plain
Date: Fri, 27 Jan 2012 03:39:27 GMT
Connection: keep-alive
d[
8:announce
51:http://us.tracker.worldofwarcraft.com:3724/announce
10:autolaunch
i0e
11:choose path
i0e
13:creation date
i1327117390e
15:direct download
74:http://ak.worldofwarcraft.com.edgesuite.net/d3-pod/20FB5BE9/NA/7162.direct14:download label
37:d3beta-1.0.0.8101-to-1.0.0.8296-patch
13:download type
i2e
4:info
d[
5:files
l[
d6:length
i25558366e
4:path
l[
7:Data_D3
2:PC
4:MPQs
(...)
The following request is a GET for "ak.worldofwarcraft.com.edgesuite.net" (just like the previous request) to retrieve the ".mfil" file. Here is a sample of this request and response:
GET /d3-pod/20FB5BE9/NA/7162.direct/d3b-8296-941A7BD6BAF437A7A7773644DA9B795B.mfil HTTP/1.1
User-Agent: Blizzard Web Client
Host: ak.worldofwarcraft.com.edgesuite.net
Cache-Control: no-cache
HTTP/1.1 200 OK
Server: Apache
ETag: "941a7bd6baf437a7a7773644da9b795b:1327352410"
Last-Modified: Mon, 23 Jan 2012 21:00:10 GMT
Accept-Ranges: bytes
Content-Length: 9456
Content-Type: text/plain
Date: Thu, 26 Jan 2012 07:18:23 GMT
Connection: keep-alive
version=2
serverpath=base
.path=
serverpath=locale_enUS
.path=
file=Data_D3/PC/MPQs
.name=Data_D3/PC/MPQs
.size=0
.fileversion=7162
file=Data_D3/PC/Misc
.name=Data_D3/PC/Misc
.size=0
.fileversion=7162
file=Data_D3/PC
.name=Data_D3/PC
.size=0
.fileversion=7162
file=Data_D3
.name=Data_D3
.size=0
.fileversion=7162
file=Data_D3/PC/MPQs/base-OSX.mpq
.name=Data_D3/PC/MPQs/base-OSX.mpq
.size=25558366
.fileversion=7162
.flags=0
.path=base
file=Data_D3/PC/MPQs/base-Win.mpq
.name=Data_D3/PC/MPQs/base-Win.mpq
.size=14968942
.fileversion=7162
.flags=0
.path=base
file=Data_D3/PC/MPQs/ClientData.mpq
.name=Data_D3/PC/MPQs/ClientData.mpq
.size=552838479
.fileversion=7162
.flags=0
.path=base
file=Data_D3/PC/MPQs/CoreData.mpq
.name=Data_D3/PC/MPQs/CoreData.mpq
.size=70056175
.fileversion=7162
.flags=0
.path=base
file=Data_D3/PC/MPQs/HLSLShaders.mpq
.name=Data_D3/PC/MPQs/HLSLShaders.mpq
.size=4791287
.fileversion=7162
.flags=0
.path=base
file=Data_D3/PC/MPQs/OpenGLShaders.mpq
.name=Data_D3/PC/MPQs/OpenGLShaders.mpq
.size=41690788
.fileversion=7162
.flags=0
.path=base
file=Data_D3/PC/MPQs/Sound.mpq
.name=Data_D3/PC/MPQs/Sound.mpq
.size=554763862
.fileversion=7162
.flags=0
.path=base
file=Data_D3/PC/MPQs/Texture.mpq
.name=Data_D3/PC/MPQs/Texture.mpq
.size=559782089
.fileversion=7162
.flags=0
.path=base
file=Data_D3/PC/MPQs/enUS_Audio.mpq
.name=Data_D3/PC/MPQs/enUS_Audio.mpq
.size=265035801
.fileversion=7162
.flags=0
.path=locale_enUS
file=Data_D3/PC/MPQs/enUS_Cutscene.mpq
.name=Data_D3/PC/MPQs/enUS_Cutscene.mpq
.size=41023397
.fileversion=7162
.flags=0
.path=locale_enUS
file=Data_D3/PC/MPQs/enUS_Text.mpq
.name=Data_D3/PC/MPQs/enUS_Text.mpq
.size=1079182
.fileversion=7162
.flags=0
.path=locale_enUS
(...)
file=Data_D3/PC/MPQs/enUS/d3-update-enUS-8296.MPQ
.name=Data_D3/PC/MPQs/enUS/d3-update-enUS-8296.MPQ
.size=767349
.fileversion=8296
.flags=0
.path=base
file=Data_D3/PC/MPQs/base/d3-update-base-8296.MPQ
.name=Data_D3/PC/MPQs/base/d3-update-base-8296.MPQ
.size=29886365
.fileversion=8296
.flags=0
.path=base
file=Data_D3/PC/MPQs/Win/d3-update-Win-8296.MPQ
.name=Data_D3/PC/MPQs/Win/d3-update-Win-8296.MPQ
.size=1610603
.fileversion=8296
.flags=0
.path=base
file=Updates/d3-0-8296-OSX-final.MPQ
.name=Updates/d3-0-8296-OSX-final.MPQ
.size=25677267
.fileversion=8296
.flags=0
.path=base
file=Updates/d3-0-8296-Win-final.MPQ
.name=Updates/d3-0-8296-Win-final.MPQ
.size=19571550
.fileversion=8296
.flags=0
.path=base
file=Data_D3/PC/MPQs/OSX/d3-update-OSX-8296.MPQ
.name=Data_D3/PC/MPQs/OSX/d3-update-OSX-8296.MPQ
.size=4802
.fileversion=8296
.flags=0
.path=base
When you combine the server URL from earlier with the "file" names from the previous response, you get the full URL for the Patch MPQ downloads. For example:
http://ak.worldofwarcraft.com.edgesuite.net/d3-pod/20FB5BE9/NA/7162.direct/Data_D3/PC/MPQs/base/d3-update-base-8296.MPQ
If you do this for all the MPQ files, you will be able to download all the files. Note that patching the root installation directory requires care and it is recommended that you let the launcher perform this task. The other MPQs are not manipulated by the launcher and will function correctly if they are placed in the appropriate directories.
One thing that I am not sure about and will need to verify next patch is whether the "public-test.patch.battle.net" server is updated prior to Blizzard re-opening the game for playing. In other words, I do not know if this server is updated with the new patch info ahead of time. If not, the new Patch MPQs can still be obtained, but only if you already know the correct Akamai URL. This URL could presumeably change in between patches although it does not appear to have done so recently.
