GDC is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them. Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

View, browse and sort the ever-growing list of sessions by pass type, topic, and format. All registered attendees will be able to build their personal schedule in the mobile event app, once live early 2022.

GDC Masterclass courses require a paid pass to attend. Class sizes are kept intentionally small and seats are limited to ensure every student receives one-on-one assistance from the instructor. Pass pricing per course varies, please review the passes and pricing page for more details and secure your spot today!

Tools Summit: Efficiently Shuffling Loads of Data from Place to Place

Adrian Astley  (Tools & Infrastructure Engineer, Activision)

Pass Type: All Access Pass, Summits Pass - Get your pass now!

Topic: Programming

Format: Session

Vault Recording: TBD

Audience Level: No

Games and tools are big and only getting bigger. Games are expanding to be developed by multiple studios in potentially distant physical locations. In addition, the game data and assets are also growing. Thus, using a network file server to transfer data for both Continuous Testing servers as well as individual developers is no longer viable. This talk is a postmortem of a tool we created to manage data transfers in order to optimize for speed, space, and bandwidth. The talk will discuss the reasons for needing this tool, the architecture we ended up with, an analysis of how well it's working, and finally a discussion of all the issues / pitfalls we ran into while developing the tool and running it at scale.


Attendees will learn an approach for highly scalable and performant data transfer. Attendees will also learn some of the pitfalls of large scale file I/O as well as the curse of scale.

Intended Audience

Programmers, producers, build engineers, and IT interested in data transfer and the pitfalls that you can run into for large-scale data data transfer / shared services.