Wargaming speeds up game build releases with Resilio Connect
Have you played World of Tanks? Over 180 million players have played the team-based massive multiplayer online action game, or one of the other titles in the series, World of Warplanes or World of Warships. This places Wargaming as one of the leaders in the free-to-play MMO market. Maintaining a leadership position in the hyper-competitive gaming market isn’t easy, and Wargaming has to keep players engaged through fast-paced game update cycles.
It takes a large army to create such complex games, and Wargaming has over 4500 employees, located in 20+ offices and development centers across the world, working on titles for mobile, PC, MAC and various gaming consoles. Each title has its own producers, designers, developers and publishers, some of whom are third party studios who need to plug in - securely - to the Wargaming infrastructure.
A key step before publishing a game update to fans, is the game build, a combination of compiled code and media assets, like textures, audio and video content. Multiple teams from many offices contribute to each game build, and the build itself needs to be transferred between the different teams as fast as possible, but also in a reliable and manageable manner.
It may not sound like a big challenge until you dive into all of Wargaming’s requirements. Each build is large, with 50GB and up of code, audio and video. Those big files have to be transferred over the available network channels, which can be great, or a not so great 10Mbps with high latency and packet loss. Regardless of the network channel limitations, the build has to go through fast.
Wargaming were certain they needed to solve this with a peer-to-peer solution that could overcome network channel limitations by using multiple nodes simultaneously. At the same time, they required a very robust set of centralized management features from this business-critical network component. They must be able to track progress and manually stop and restart transfers from a web interface, and also control bandwidth consumption. They work with 3rd parties and must be able to secure access with granular controls. Cross platform support and simple integration with the rest of their stack and into their workflow were also important.
The engineering team evaluated 14 different synchronization solutions, taking a deep dive into 4 solutions. When they took Resilio for a spin they were pleasantly surprised. “Our engineers deployed a successful test version in a hour - without reading a ton of manuals - that delivered the best result of all the solutions we tried,” says Roman Sakno, Enterprise Architect at Wargaming.
All in all, Resilio was the clear winner because of its combination of very high performance and across the board simplicity. The pure peer-to-peer architecture yielded significant speed improvements. “Resilio met all our technical requirements, but was also very easy to deploy and integrate into our workflow. We also liked the fact that the simplicity carried over to transparent pricing and clear documentation” explained Sanko.
Wargaming has successfully implemented Resilio Connect in 10 geographical locations. On the one hand it is used to synchronize BitBucket builds using the REST API. It is also tightly integrated with TeamCity and Jenkins, and is used to synchronize builds to DevOps teams located on different continents.
Robust performance over limited bandwidth network channels with latency and packet loss by incorporating WAN optimization
Speed up transfers of very large files with multiple types of content - binary, audio, video
Fine-grained user access control, including management of 3rd parties
Web-based tracking of synchronization progress with manual interrupt & resume and exception handling (e.g. lack of disk space)
Cross-platform support (Linux, Windows, Ubuntu servers and CentOS), as well as support for all package formats
Simple integration with CI/CD tools Jenkins and TeamCity
Flexible API for integration with existing workflow
The main collaborative development tool is BitBucket, and it’s integrated with Resilio using commit hook and its REST API. When someone changes the file in Git LFS hosted on BitBucket, it notifies Resilio Connect to synchronize file from Git LFS in the current location with another location.
Primarily, the integration is used to transfer audio/video/textures from our designers. On other side, Resilio is tightly integrated with TeamCity and Jenkins. When a build is completed then CI just calls Resilio via API to start transfer and send file to the DevOps team, which is located on the different continent.
Resilio chosen after evaluation of 14 solutions. Syncs international offices with a managed and secure P2P solution that is simple to deploy.
Synchronize game builds between international offices fast, regardless of network infrastructure. Need a secure, manageable solution that will easily integrate with existing stack and workflow.
Resilio Connect was selected for its high speed performance, simplicity of operation and its full complement of enterprise features.
"We were able to turn around graphic proofs quicker based on our staff’s availability, versus forcing it through a bottleneck at a specific office. Connect has made us more efficient" — Ken Brubacher, Project Manager.Read Customer Story
“Our engineers deployed a successful test version in an hour - without reading a ton of manuals - that delivered the best result of all the solutions we tried” — Roman Sakno, Enterprise Architect at Wargaming.Read Customer Story
"If we bought about 40 additional servers it would have been pretty expensive. Resilio is a much cheaper solution for us right now." — Mario Jost, Systems Engineer @Mercedes-BenzRead Customer Story
"Resilio Connect enables us to reliably distribute our code, specifically new language models in a fraction of time. These copy jobs now take an hour, down from eight."— Christian Bryndum, VoiceBase’s Director of OperationsRead Customer Story