Automated Ranking System for Offline Events


#1

Hey all,

As evidenced by my post count and join date, I’m fairly new here. Despite this, I am not new to relative strength ranking systems. In my previous fighting game endeavor, I put about a year and a half of work and research into a ranking system which only utilized wins and losses. It was nearing completion, but complications with my PhD came up and I have since retired.

The point is this: would you all be interested in an offline numerical ranking system? I know there’s an online ranking, but I’m sure there is an inherent difference between online play and offline play. If so, with some help I can complete the system I was working on before - the “TGL Project”. I do not yet know where I can get tournament results here, nor do I know what kind of information you guys record. If you all keep track of wins and losses, that would be good and bad. Good because of the added accuracy, bad since I’d have to restructure the program (thus causing delays to the first lists).

If there’s enough interest, I’d like to get my hands on approximately 1/4 to 1/2 a year worth of tournament results. That’d be enough to complete my research. Though it’d also be helpful to have some help from established community members to give me a general idea of who should be better than who (to have an intuitive check on the accuracy of the program)… but that’s a bit much of a newbie to ask of the community, heh.

Let me know what you think! I have a degree in Chemical Engineering and have a bunch of additional research done on these kinds of rankings. If this were to reach a high enough level of interest, I know a programmer who’d be willing to make a distributed version of it for regional use.

Regards,

Tuen


#2

I’ve been thinking for a while that the FGC would benefit from a ranking system similar to what Wizards of the Coast uses for Magic: the Gathering, i.e. an ELO based system that rewards the highest ranked players. In theory you could use nothing but majors (the tournaments that most consistently have the majority of the match results available) to track peoples performance. Challonge.com (I think) would be a good place to start.


#3

Anything would be better than the shitty system used by Evo last year. Win one tournament and you pretty much sealed a top 8 seed.


#4

You definitely want to get in touch with these guys:

They do a really great job for the smash community.


#5

I used to be in the smash backroom, actually. That’s what I meant by “previous fighting game experience”, but as I’ve heard regularly around the smash community, they’re commonly talked down upon when brought up in other communities, so I kept it vague.

That aside, the system they used is one of the weaker forms of ranking systems. It assigns point values to performances and takes your best 12 performances. I think this doesn’t account for enough data when determining relative ranks. It also doesn’t integrate new players very well. They also haven’t attempted to address the issue of drift… but to be fair, that’s what I got stuck on too (this is when one region drifts away from another). I also disagree with the use of monetary gain as a way to scale point gain, but I haven’t looked into that as much, research wise. I just philosophically think that a match that is deemed “serious” for one reason or another will indicate relative skill regardless of money ($100 to $15,000) gained.

If you guys want a closer look at what i’ve done, it’s all here: http://allisbrawl.com/group.aspx?id=9557

As it says, I stalled out when I tried to learn a programming language myself. Turns out, I’m pretty good at research and development… but I’m a terrible programmer. Heh.


#6

Winning a major should count for a lot, shouldn’t it?

I thought the Evo 2011 system was decent. The only thing I could take issue with it is that it favored people who had the means to travel to every stop on the list.


#7

Well, not all majors are created equal. Should Tokido have gotten the same number of points at an Australian Marvel tournament than he did at an American Marvel tournament? And yeah, it favored those who could travel a lot, too.

But I don’t have any suggestions myself, and I’m assuming that Evo is using the same system this year, so… oh well. I just feel that 64 points is hard to catch up to.


#8

you just had to mention your PhD didn’t you…

well since you’re new here, to be accepted into our community you have to beat Mike Ross in a FT5 set (any game). Then you’ll be allowed to make the official ranking system for the Evo tournament series and will be given a Stetson hat by Mr Wizard.


#9

This widely depends on the behavior of the community. If all matches are created equal, then after a certain amount of incentive is given (e.g. after a certain tournament size), then the assumption that any one given match result is legitimate becomes valid. In the super smash brothers community, there was always the issue of players goofing off and randomly throwing matches when the tournament size/payout/incentive was too low.

If the player behavior is consistent enough, I wouldn’t scale an event with bigger prizes any different than one with average prizes. If someone relatively unknown beats an awesome player, they deserve recognition (and a points adjustment) regardless.


#10

That’s pretty much how an ELO based system works. Everyone starts with the same baseline number of points and then gains or loses points based on wins and losses with the points being gained or lost being determined by their opponents point total and the size (value) of the event. Magic the Gathering for example (I think this is all still accurate; haven’t played in years) starts players off with 1600 points and has sanctioned tournaments at 8, 16, 24, 32 and 64 points. Meaning that in theory those point values are the most that can be gained or lost for a single match in that a specific event.


#11

Those metered point values are the ‘K-Values’, I believe. There’s another way to meter these out, actually. It’s the one I used to avoid dealing with the ‘which event is big enough’ question. Anyways, you can dole out the K values based on player strength. I haven’t really reviewed the details (my previous research ended almost 2 years ago), but I think it requires player ranks (i.e. anyone over 2000 is rank A, 1800-1999 is rank B, etc). You can only gain or lose an amount equal to your own groups K-Value, which decreases as you get better. This is one reason why chess players take YEARS to get to ‘grand master’ rank and such.

This also conveniently deals with the runaway issue. Say I beat XYZ superstar player at MVC (you know, right after the world ends this December). If you didn’t have metered K-values, the distributed points would be based on our rank separation. And given how new I am, it’d be HUGE. So the good player would take epic point punishment for losing to a nobody, and I’d skyrocket. It’d cause a lot of problems since there are a multitude of reasons that could be present for an upset… it literally could be a one-time incident, but it would still have a massive impact. Though the system you’ve described also mitigates this problem, so it may come down to preference when determining which to use. I just wanted to avoid a bunch of heated debates about which event warranted which rank. Though I could trump everyone and just make a set of event ranking rules and lean on my education as validation… heh.

Speaking of interesting corrections, I’ve also added a New Player Index (NPI). It helps integrate new players without impacting the current population. While the index is active, I as a new player can see all the gain/loss benefits in full, while you do not. It’s most helpful when a new player comes in and ranks surprisingly high. If there were 10 people in my region, and my ‘true rank’ was 5th as a newcomer, I could potentially ruin rank 6’s score, sending him/her to rank 8-ish, when the change was not warranted. The NPI allows me to climb to rank 5 without having an intense impact on everyone else. This is also useful for ranking resets, as people’s scores will all stabilize at different rates, and 20 games are required to erase your NPI.

And I’m getting off of the quoted topic… I could talk about this for a long time, as you see.

The next question for me would be this: where can I get tournament data? Also, what bracketing program do you guys tend to use? The smash community was real convenient in this regard, they almost all (and I mean approximately 99% of TO’s) use TIO. It helped a lot with regards to automation.


#12

Sup, Tuen!


#13

Yo! Can you or anyone else lead me to tournament results around here?