While looking at Romain's community post yesterday, I was elated to see that they're using metadata to make balance-driven decisions. I think it's definitely the right direction, especially if they can use this data to confirm the opinions of top players who are giving them feedback.
However, there is some data that I think is missing, or at least not being shown that can help us a little more when it comes to balance. I'm not sure what the telemetry is like over at Tindalos, but part of what I do professionally is helping people smarter than me make better decisions using data. Since I love the game and I want it to succeed, I figure I can just iterate on some things to be considered.
When looking at the win percentages like what I see above, I'm not exactly how to interrupt this data. Most of the time when we're looking at data, we're asking questions first. It starts with questions, then the data is collected, then we analyze the data, to make decisions based on that data, and it eventually (and hopefully) turns into actions such as balance changes.
Some questions I would ask based on the data displayed above:
- Is this the global average of all skill levels and all matchups?
- Does this combine 1v1 and 2v2 as well?
- Are we focusing balance on the majority or by the top players?
- Are we targeting balance around 2v2 or should 1v1 be the priority?
- If we're using data from top players, what does that data look like compared to the average player?
- Is Legendary rank being used or is overall ELO or possibly a mixture?
Just based on what we see so far, I would interrupt this as the global average of all skills vs. all matchups in both 1v1 and 2v2.
I think this is a good start, and as I said before, I'm not sure if more data is being collected in the background so this is going off the assumption that it is. Most telemetry data being collected these days in modern game companies capture quite a lot, so I think we have some wiggle room for me to mass interrupt and make assumptions.
I think what data would be really useful would be, especially in the realm of balance would be to answer the following questions:
- Who are the top players by ELO and what factions are they using?
- Within these top factions, what fleet composition are they using?
- Within these fleets, what upgrades/skills are they using?
This data can be used to confirm whether or not something like Tyranids, for example, are as dominant as everyone says they are. More importantly, it can also determine whether or not it's just Tyranids, or are other factions displaying the same outliers and patterns. I'll elaborate below..
Let's just say that the data we got back the data above and it looks like this. All the data is hypothetical and high-level:
- We find out that there are 4 players in the Top 10 ELO that are using Nids as a faction, however, there are also other factions like CW Eldar, Drukhari, Chaos, Orks..etc.
- OK, so we think that Nids might be over-performing a bit, so let's dig a little deeper and see what fleet comp they're using. It turns out that 4 of the 4 are using heavy Carrier fleets hmm..
- Digging again, we look at the Nid Admiral Skills and Upgrades are the exact same from all 4 Nid players in the top 10.
Data, when aggregated correctly, can tell us a crap ton. Just based off of the high-level hypothetical data that I presented above, we can tell:
- That Nids are performing well in top-level play, making up of 40% of the 10 players in 1v1. We can expand this to show top 25, I bet that will tell even a greater story.
- That most common fleet type being taken is Carriers for those Nid players, and identifying which ships are the most competitive choice for these players.
- That all the Nids and Admiral skills are the exact same, identifying that not only is there very little competitive variance, but perhaps maybe these upgrades to increase the effectiveness of Carriers specifically, or if they're just incredibly good that they're auto-include (Psychic Scream, I'm looking at you).
However, what we can deduce what this looks like for everyone else on the top tables. The reason why this analysis is important is that it allows us to answer questions like: Are Nid Carriers too powerful specifically? or are Carriers too powerful specifically? If you look at the Top25 and all 25 of every faction is packing Carriers, that might be saying something. If you look at the data and see convincingly that it's just Nids, the balance decision can be to just target balance fixes at just Nid Carriers.
Looking at this larger subset of data can also tell us more subtle things like:
- Proportionally, there is a smaller number of CW Eldar players in the top 25, but whether or not this means design expectations is another story.
- The follow-up here is that the 4 or whatever CW Eldar players are all packing the same fleet comp and the same skills/upgrades. Do we want that or should we promote more diverse play?
- Likewise, we see that there is only 1 Corsair player in the top 50, and go hmm.. we need to look at evening the odds here somehow.
- This can also be used to tell outliers, like this HERO guy asking for 270 arcs to be nerfed and a nerf to Eldar torps. Is he the only CW Eldar saying this, or are there more players of similar skill levels all using the same fleet comp/skills/upgrades?
I'm sure I can go on, but I feel strongly that when it comes to balancing and interrupting data, the data itself should be used to confirm the opinions of top players giving feedback. This is healthy and can only benefit the game in the long run.
