The impact of gaming updates and why your streaming may also be affected

There’s nothing worse than sitting down to enjoy some nice down time, only for the Netflix show you are currently bingeing to show you the dreaded ‘buffering circle’.

It’s closely followed by the disappointment of messaging your friends to squad up and play some Warzone, only to find that there is another 30GB update.

*Holds back tears*

Wow, that was tougher to type out than it should have been.

So, why is your Netflix buffering or your COD update downloading at slower speeds?

Games and their updates are slowly getting bigger and bigger. Currently, Call of Duty takes up 172.7GB of my PS4 storage, that’s over a third of my storage. For those of you that don’t game, that’s big, real big.

To add further misery, next-gen consoles are right around the corner and games/updates are going to get a whole lot bigger. Modern Warfare size files and updates are going to be the new norm. Some people have projected games to reach 400-500GB within the next several years, which is absurd to think about.

With bigger games and more people resorting to gaming during lockdown (because, well, ugh, what else are you going to do?), it’s meant that there is a greater strain placed on game developers’ servers and the links that bring Australia the data.

It also means, for those of you who don’t game, that on particular nights when massive updates are released, you might feel the effect of hundreds of thousands of gamers downloading the same update at the same time.

While there’s no easy answer or explanation, let’s go through some of the things we’ve seen.

Increased usage

Each time we see a Call of Duty update, it has a huge network effect. There’s usually around a 30% uplift in traffic.

Our network technicians cited that at 4pm on the day updates are released, we may have 200G to 300G more usage than normal, that’s a lot. And because of the cost of CVC, we don’t have that just sitting around either. It means that during peak times, there is a lot of manual intervention from our network technicians in upgrading CVCs, so that people still have reasonable speeds.

However, on the night when COD released its season 5 Warzone update, Fortnite also released an update, the two combined resulting in the highest traffic count we’ve ever seen.

Our Managing Director Phil summed up the night perfectly.

“[On that night] we saw a large number of our CVCs flatline (capacity maxed and speeds start to slow) as a result,” said Phil.

“We already have nbn’s 40% boost provisioned across the network. But because two popular games released at the same time, combined with the already higher usage due to stay at home restrictions, saw bandwidth well exceed this and created the highest bandwidth event we have ever seen on the network.”

On those particular nights, even those of you who are not downloading a COD update are feeling the effects of the sheer amount of people downloading it.

Whilst we always over-provision CVC and ensure there is enough capacity, on update nights you might see a slight dip in the speeds you are used to.

For gamers: Server throttling

As mentioned, more people are gaming during lockdown. This means that as more people access their platform’s servers to download games/updates, the server may only allow certain speeds to ensure that it can keep up with the amount of people accessing the download.

When servers have limits on their output, there’s not much ISPs can do to improve the situation; the speed has been set before it reaches the ISP.

For gamers: Lack of update warning

Often games release their patches without much warning, if any warning. Without a heads up, ISPs can’t provision ahead and ensure there is enough bandwidth available for the increased demand.

For gamers: Pathing

The issues with trying to prepare and account for an uptick in gaming downloads are, firstly, if game developers don’t offer alternative methods to receive the data, everyone is funnelled to the same place. This leads to the other problem, if game providers don’t provision enough capacity for the amount of people accessing their data servers, it will cause the capacity and connection to flatline.

In the case of COD, the content is primarily delivered by their content delivery partner, CenturyLink, which Aussie Broadband reaches primarily via their 100G, now 200G, link into IX AUstralia. With the amount of people trying to download the update, CenturyLink also deliver content via various other networks in Australia and overseas, however, sometimes a particular path within Aussie Broadband or CenturyLink’s (and other content delivery partners) networks can become full.

What does that mean?

Think of it like the CenturyLink 200G line is a ‘shared’ freeway that has 4 lanes, it’s suitable for ‘normal’ traffic.

When a COD update is released, almost the entire player base jumps on to download the update at essentially the same time. Everyone packs onto the freeway. When the freeway reaches capacity, people try their luck on back roads, these are either overflow caches or other transit links within the Aussie Broadband network. But in time, as more people enter the freeway and use back roads, all avenues slow.

The problem is that if game developers don’t give us (and all ISPs) better ways to receive and distribute the data (i.e. another freeway), then all ISPs are at the mercy of how busy that one freeway is.

Moreover, if the gaming developers don’t account for the number of users by including more lanes on the freeway, the freeway will never be able to comfortably handle the number of users at the speeds expected when on a freeway.

What’s happening when these updates are released is a combination of both problems, there is no other cost effective avenues for Aussie Broadband, bearing in mind these large game updates generally only occur 2-3 times a month at most, to receive/distribute the data. The current capacity, on the available avenues, are being maxed, meaning people may experience slower speeds.

However, in better news regarding future COD updates, Activision-Blizzard appear to be trialling delivery of patches via their own content delivery network in conjunction with their existing arrangements. They have also agreed to having a private network interface directly with us. This means that we will have a private freeway just for our customers. This won’t completely bypass CenturyLink’s 200G ‘shared’ freeway, they will be used in conjunction to mutually benefit each other.

When our customers connect to our network to download the new update, they will be automatically sent to the dedicated freeway and CenturyLink’s freeway simultaneously, expanding the available capacity significantly and hopefully improving the overall patching experience with increased download speeds.

For a more technical take on pathing

When capacity on the CenturyLink port and the akamai caches is full, traffic will come in from just about every path that it can. There is very little a provider can do to limit it or confine it to a particular path.

This is problematic because some of the other avenues are expensive.

“What happens next is Akamai then overflows to other caches and in our case, we then saw a lot of it come through our Telstra transit links which then also flat-lined, in the case of the season 5 and Fortnite update. This is the most expensive way to get into our network,” said Phil.

Whilst we have akamai cashes in the IX, what we have seen is that most of the connectivity from customer’s devices (e.g. PS4) to the download servers are all via CenturyLink, with a small amount of overflow onto Telstra, Akamai or other caches.

Even though we can provision a lot of capacity anywhere in the network, if the providers we peer with do not (e.g. needing more than the CenturyLink’s capacity), we cannot control that. And if the game developer cannot distribute data over other peering networks to us, we will have no other cost effective way to get to it. We will be limited by the capacity of CenturyLink peering, caches and our own transit connectivity (the most expensive option).

So, for the COD update, a majority of the Aussie Broadband customer traffic is trying to come in via CenturyLink and their capacity/ports are not solely designed to provide only Aussie Broadband connectivity when a COD update is released. Given the number of people accessing the download, it may result in available capacity being full, via that path, during peak download periods.

Due to the unprecedented number of downloads, CenturyLink, to their credit, made an upgrade to get another 100G port (200G capacity), so gamers should see an improvement with their download speeds.

Moreover, we’ve also seen Activision-Blizzard reach out to trial delivering content via their own content delivery network (CDN) simultaneously with CenturyLink’s 200G link.

We’re arranging a 2 x 100G private network interface (PNI) directly to Activision-Blizzard. Our customers will be directed to the dedicated link for us, which will improve the speeds of everyone’s download, whether your connected to the PNI or CenturyLink’s 200G link.

So, there you have it. These are the main reasons why your Netflix may, on certain nights, experience some buffering, or why your game patch is downloading speeds are slower than usual.

As always we are working behind the scenes to make sure that, when certain games release patches, we are giving our customers the best speeds possible. Shout-out to our network technicians, we see you!