Community update, looking at metadata and interpreting it

0_1547927081085_gsEmSNM.jpg

While looking at Romain's community post yesterday, I was elated to see that they're using metadata to make balance-driven decisions. I think it's definitely the right direction, especially if they can use this data to confirm the opinions of top players who are giving them feedback.

However, there is some data that I think is missing, or at least not being shown that can help us a little more when it comes to balance. I'm not sure what the telemetry is like over at Tindalos, but part of what I do professionally is helping people smarter than me make better decisions using data. Since I love the game and I want it to succeed, I figure I can just iterate on some things to be considered.

When looking at the win percentages like what I see above, I'm not exactly how to interrupt this data. Most of the time when we're looking at data, we're asking questions first. It starts with questions, then the data is collected, then we analyze the data, to make decisions based on that data, and it eventually (and hopefully) turns into actions such as balance changes.

Some questions I would ask based on the data displayed above:

  • Is this the global average of all skill levels and all matchups?
  • Does this combine 1v1 and 2v2 as well?
  • Are we focusing balance on the majority or by the top players?
  • Are we targeting balance around 2v2 or should 1v1 be the priority?
  • If we're using data from top players, what does that data look like compared to the average player?
  • Is Legendary rank being used or is overall ELO or possibly a mixture?

Just based on what we see so far, I would interrupt this as the global average of all skills vs. all matchups in both 1v1 and 2v2.

I think this is a good start, and as I said before, I'm not sure if more data is being collected in the background so this is going off the assumption that it is. Most telemetry data being collected these days in modern game companies capture quite a lot, so I think we have some wiggle room for me to mass interrupt and make assumptions.

I think what data would be really useful would be, especially in the realm of balance would be to answer the following questions:

  • Who are the top players by ELO and what factions are they using?
  • Within these top factions, what fleet composition are they using?
  • Within these fleets, what upgrades/skills are they using?

This data can be used to confirm whether or not something like Tyranids, for example, are as dominant as everyone says they are. More importantly, it can also determine whether or not it's just Tyranids, or are other factions displaying the same outliers and patterns. I'll elaborate below..

Let's just say that the data we got back the data above and it looks like this. All the data is hypothetical and high-level:

  • We find out that there are 4 players in the Top 10 ELO that are using Nids as a faction, however, there are also other factions like CW Eldar, Drukhari, Chaos, Orks..etc.
  • OK, so we think that Nids might be over-performing a bit, so let's dig a little deeper and see what fleet comp they're using. It turns out that 4 of the 4 are using heavy Carrier fleets hmm..
  • Digging again, we look at the Nid Admiral Skills and Upgrades are the exact same from all 4 Nid players in the top 10.

Data, when aggregated correctly, can tell us a crap ton. Just based off of the high-level hypothetical data that I presented above, we can tell:

  • That Nids are performing well in top-level play, making up of 40% of the 10 players in 1v1. We can expand this to show top 25, I bet that will tell even a greater story.
  • That most common fleet type being taken is Carriers for those Nid players, and identifying which ships are the most competitive choice for these players.
  • That all the Nids and Admiral skills are the exact same, identifying that not only is there very little competitive variance, but perhaps maybe these upgrades to increase the effectiveness of Carriers specifically, or if they're just incredibly good that they're auto-include (Psychic Scream, I'm looking at you).

However, what we can deduce what this looks like for everyone else on the top tables. The reason why this analysis is important is that it allows us to answer questions like: Are Nid Carriers too powerful specifically? or are Carriers too powerful specifically? If you look at the Top25 and all 25 of every faction is packing Carriers, that might be saying something. If you look at the data and see convincingly that it's just Nids, the balance decision can be to just target balance fixes at just Nid Carriers.

Looking at this larger subset of data can also tell us more subtle things like:

  • Proportionally, there is a smaller number of CW Eldar players in the top 25, but whether or not this means design expectations is another story.
  • The follow-up here is that the 4 or whatever CW Eldar players are all packing the same fleet comp and the same skills/upgrades. Do we want that or should we promote more diverse play?
  • Likewise, we see that there is only 1 Corsair player in the top 50, and go hmm.. we need to look at evening the odds here somehow.
  • This can also be used to tell outliers, like this HERO guy asking for 270 arcs to be nerfed and a nerf to Eldar torps. Is he the only CW Eldar saying this, or are there more players of similar skill levels all using the same fleet comp/skills/upgrades?

I'm sure I can go on, but I feel strongly that when it comes to balancing and interrupting data, the data itself should be used to confirm the opinions of top players giving feedback. This is healthy and can only benefit the game in the long run.

last edited by LKHERO

@lkhero Since you are interested, I can provide data for the top league players. There are :
27 nid players, 19 Druhkari players, 12 SM players, 8 players for IN, Chaos, and orks, 4 players for Asuryani and Tau Protector, ad mech and Necrons have two players each, Tau merchant have one player and there are no corsair players.

@nemesor-xanxas said in Community update, looking at metadata and interpreting it:

@lkhero Since you are interested, I can provide data for the top league players. There are :
27 nid players, 19 Druhkari players, 12 SM players, 8 players for IN, Chaos, and orks, 4 players for Asuryani and Tau Protector, ad mech and Necrons have two players each, Tau merchant have one player and there are no corsair players.

Is this using the current leaderboards? Not to say it's completely useless, but I don't really care about the current leaderboards because it's collecting 1v1 and 2v2 data and factors in whoever spams the most games.

I care about actual ELO and the players who consistently perform well vs. other players of similar skill levels.

@lkhero It was yesterday morning (for me, its 4am here). Well, I thought it would be kinda helpful.

@nemesor-xanxas said in Community update, looking at metadata and interpreting it:

@lkhero It was yesterday morning (for me, its 4am here). Well, I thought it would be kinda helpful.

Yeah no worries man, I didn't mean to sound harsh to the data you provided. Thanks for breaking it down, but at the end of the day, it could also mean that those 27 Niddies and 19 DE played a crap ton more than anyone else!

@lkhero fair enough, I was more worried it might have changed since I last checked as I haven't had time to get on to check afterwords. You make a good point though.

Totally agree with the analysis (just a quick remark, a sample of 10 top players is a bit low), macro datas like global winrate is a thing but it can lead to misinterpretation if not backep up by detailed analysis.
For example everyone focusing on Nid carriers overperforming but is it related to Nid carriers ships being OP or to boarding being OP and nid carrier able to overabuse it from range?
Because with higher analysis of the top players if SM + Nid + DE are at the top it is more linked to boarding on which they all excels (easy crit + fast speed to come in range + better assautt boats/torps than other factions) than just nid carriers. Orks are a bit behind because they are too slow to use boarding at optimum value

@beernchips said in Community update, looking at metadata and interpreting it:

Totally agree with the analysis (just a quick remark, a sample of 10 top players is a bit low), macro datas like global winrate is a thing but it can lead to misinterpretation if not backep up by detailed analysis.
For example everyone focusing on Nid carriers overperforming but is it related to Nid carriers ships being OP or to boarding being OP and nid carrier able to overabuse it from range?
Because with higher analysis of the top players if SM + Nid + DE are at the top it is more linked to boarding on which they all excels (easy crit + fast speed to come in range + better assautt boats/torps than other factions) than just nid carriers. Orks are a bit behind because they are too slow to use boarding at optimum value

Oh yeah, totally. The Top 10 was the quick quote, but Top 25 would be more reliable and so on and so forth. The more data you look at, the better the aggregated data and the wider the breadth of coverage.

Awesome thread

Another major factor is current bugs.

Dark Eldar are UNDERPERDORMING considering reload and kin crewed batteries both do absolutely nothing yet greatly buff dark Eldar. When these bugs are fixed, dark Eldar will be performing better than they do currently.

9k only range for corsairs and craft world also means they underperformed due to bugs.

Reload stance being bugged means damage for almost all factions is also lower than It should be making many macro fleets like IN underperform.

The frequent boost bug/ability bug means mainly Eldar (most affected) currently underperform.

Faction power spikes due to unlockable abilities is also a big influence.

Necrons are trash tier until mass recall, and still probably low tier with it unlocked.

MWJ can also be a big spike for other fleets.
Etc...

last edited by CowGoMoo

Not to sure why you put such a high emphasis on 'the top players' but ok...

What suprises me is that even with them seeing these numbers and knowing the balance is completly screwed they arent really focusing on fixing the issue. Rather they are applying a band aid by increasing point costs and releasing the game anyway...

Balance generally should target the higher elo players. Balancing low elo players will come from factors such as making the default fleet options stronger. Adding auto cast abilities or simpler to use ones. Making the low level unlockables include the strongest abilities. Making game mechanics intuitive and easy to understand. Making skill ceilings and floors within reasonable ranges for each faction etc....

Unit stats and balance should focus on the upper echelons for the most part. not exclusively, but predominantly.

last edited by CowGoMoo

@Demoulius They are making more changes and they know what the balance issues right now are.

Just here to say that there is a tendency to hyperbole in balance discussions on the forum.

Balance isn't perfect (and never will be) but it is far from being screwed. I am having fun with the game right now and we just got the confirmation from the game director, that they are looking at both metadata and players feedback to fine tune the game.

I am satisfied all around and look forward to the full release on Thursday!

@Adm_Janus balance is definitely screwed lmao but that's ok as it is a a beta and the devs are listening the player feedback and are aware of the current issues.

@brohanbroski Maybe it gets worse in the upper leagues, I am doing ok in the adamantium bracket. There seems to be reasonable levels of cheese 😃

@Adm_Janus I'm 5th overall right now with my IN. Once you reach legendary you just start running into a lot more overpowering meta builds that overperform and effectively counter conventional fleets like 5 carrier bc synapse nids and slowing zzap orks.

I know that my 4 Styx BC's can kite apart most anything I come across. The problem is that a lot of what I fight is designed to be cheese. So I play keep away until my opponent rage quits. It's not fun for either of us really.

I'm sure they have the Data you're looking for, if they will share that? I'd probably say not.

last edited by brn4meplz

I have a feeling these numbers are just from all games in all modes.

I refuse to believe the winrate is this high on some of these factions in 2v2 games. AM is hot trash in 2v2, maybe they do ok in 1v1 though which if that's the case that's skewing the numbers.

Same for Necrons, SM's etc etc.

I'm confused on the leaderboards as well, are they just pooling any and all ranked games into one leaderboard? 1v1 and 2v2 should have their own Leaderboards.