Wargaming speeds up game build releases with Resilio Connect

Introduction

Have you played World of Tanks? Over 180 million players have played the team-based massive multiplayer online action game, or one of the other titles in the series, World of Warplanes or World of Warships. This places Wargaming as one of the leaders in the free-to-play MMO market. Maintaining a leadership position in the hyper-competitive gaming market isn’t easy, and Wargaming has to keep players engaged through fast-paced game update cycles.

Challenge

It takes a large army to create such complex games, and Wargaming has over 4500 employees, located in 20+ offices and development centers across the world, working on titles for mobile, PC, MAC and various gaming consoles. Each title has its own producers, designers, developers and publishers, some of whom are third party studios who need to plug in - securely - to the Wargaming infrastructure.

A key step before publishing a game update to fans, is the game build, a combination of compiled code and media assets, like textures, audio and video content. Multiple teams from many offices contribute to each game build, and the build itself needs to be transferred between the different teams as fast as possible, but also in a reliable and manageable manner.  

It may not sound like a big challenge until you dive into all of Wargaming’s requirements. Each build is large, with 50GB and up of code, audio and video. Those big files have to be transferred over the available network channels, which can be great, or a not so great 10Mbps with high latency and packet loss. Regardless of the network channel limitations, the build has to go through fast.

Wargaming were certain they needed to solve this with a peer-to-peer solution that could overcome network channel limitations by using multiple nodes simultaneously. At the same time, they required a very robust set of centralized management features from this business-critical network component. They must be able to track progress and manually stop and restart transfers from a web interface, and also control bandwidth consumption. They work with 3rd parties and must be able to secure access with granular controls. Cross platform support and simple integration with the rest of their stack and into their workflow were also important.

Solution

The engineering team evaluated 14 different synchronization solutions, taking a deep dive into 4 solutions. When they took Resilio for a spin they were pleasantly surprised. “Our engineers deployed a successful test version in a hour - without reading a ton of manuals - that delivered the best result of all the solutions we tried,” says Roman Sakno, Enterprise Architect at Wargaming.

All in all, Resilio was the clear winner because of its combination of very high performance and across the board simplicity. The pure peer-to-peer architecture yielded significant speed improvements. “Resilio met all our technical requirements, but was also very easy to deploy and integrate into our workflow. We also liked the fact that the simplicity carried over to transparent pricing and clear documentation” explained Sanko.

Wargaming has successfully implemented Resilio Connect in 10 geographical locations. On the one hand it is used to synchronize BitBucket builds using the REST API. It is also tightly integrated with TeamCity and Jenkins, and is used to synchronize builds to DevOps teams located on different continents.

Benefits

  • Robust performance over limited bandwidth network channels with latency and packet loss by incorporating WAN optimization

  • Speed up transfers of very large files with multiple types of content - binary, audio, video

  • Fine-grained user access control, including management of 3rd parties

  • Web-based tracking of synchronization progress with manual interrupt & resume and exception handling (e.g. lack of disk space)

  • Cross-platform support (Linux, Windows, Ubuntu servers and CentOS), as well as support for all package formats

  • Simple integration with CI/CD tools Jenkins and TeamCity

  • Flexible API for integration with existing workflow

Technical Description

The main collaborative development tool is BitBucket, and it’s integrated with Resilio using commit hook and its REST API. When someone changes the file in Git LFS hosted on BitBucket, it notifies Resilio Connect to synchronize file from Git LFS in the current location with another location.

Primarily, the integration is used to transfer audio/video/textures from our designers. On other side, Resilio is tightly integrated with TeamCity and Jenkins. When a build is completed then CI just calls Resilio via API to start transfer and send file to the DevOps team, which is located on the different continent.

Industry

Tech, Gaming

“We here at Turner Sports like to try new things. If there's something new, I'm always willing to jump into the pool and try it out. Some people are stuck in their old ways and workflows. Well, there might be a better way. For us, that's Resilio.” — Brady Small, Sr. Production Engineer Sports Central Systems.
Read Customer Story
"We were able to turn around graphic proofs quicker based on our staff’s availability, versus forcing it through a bottleneck at a specific office. Connect has made us more efficient" — Ken Brubacher, Project Manager.
Read Customer Story
“Our engineers deployed a successful test version in an hour - without reading a ton of manuals - that delivered the best result of all the solutions we tried” — Roman Sakno, Enterprise Architect at Wargaming.
Read Customer Story
"Resilio Connect enables us to reliably distribute our code, specifically new language models in a fraction of time. These copy jobs now take an hour, down from eight."— Christian Bryndum, VoiceBase’s Director of Operations
Read Customer Story
“Resilio would keep Ross Maritime fully compliant and would be very easy to use for those onboard the vessels.”— Brent Nevison, IT consultant at Ross Maritime.
Read Customer Story
"If we bought about 40 additional servers it would have been pretty expensive. Resilio is a much cheaper solution for us right now." — Mario Jost, Systems Engineer @Mercedes-Benz
Read Customer Story
“Our entire situation is tremendously better, great cost savings and our data is now delivered on time.” — Jason McCune , Director of Information Technology.
Read Customer Story
“I see Resilio Connect as being absolutely pivotal to the way they work now. The media and entertainment industry is really an ideal opportunity for this technology.” — Abraham Treadwell, Tangelo CEO. (Blockhead's IT Consultants)
Read Customer Story
“For us, the main thing was fast transfer of data and obviously maintaining the integrity of all the data that we sync to our offices. And definitely the cost is very important. We were happy to find a solution where we could use our in-place architecture to keep down the cost.” — Sebastiaan Sprengers, Lead Tools Programmer @Larian Studios.
Read Customer Story
“With Resilio, you know your job has run and you can initiate scripting triggers to get feedback from clients to know if a patch is installed correctly or not” — Senior ICT Engineer (name withheld for the privacy of our customer)
Read Customer Story
"The results we get about the file transfer being completed and having no issues with the files on the other end has been fantastic. These are things I couldn’t do before using Connect.." — Jacques Vermeulen, Project Manager.
Read Customer Story
“Resilio Connect has been a game changer. It’s proven to be reliable in file transfer, It’s proven to be reliable in database replication. Overall, Resilio Connect jobs are very easy to set up and they just work!”— IT Developer at Lindblad Expeditions
Read Customer Story