my view is this:
Online play is the only way to get statistically significant data. Tournament data is simply too sparse. Online has millions of matches, and that is statistically significant. So I think that in principle, online is a great way to get data.
My problem with the current rankings online is not lag … it’s characters! Rankings should be specific to each player/character combination. For example, let’s say Thelo is just picking up Dhalsim. The best system would show:
Under the current system, it just shows one combined ranking:
Under the current system, if Thelo starts playing Dhalsim, his combined rank will go down. This is bad, and not just for Thelo. It’s bad because the whole system is based on feedback. under the current hybrid system, Thelo’s rank might go down to
as he plays dhalsim. Then if he switches again to Honda, he’ll be playing at a 4800level when is he actually ranked at 2000! So his opponents will all get messed up rankings too.
It’s really important for each player/character combination to get its own ranking. That would be using the system as per design. Under the current, unfortunate implementation, rankings aren’t a very good measure. Let me list some of the problems with combined ranks:
*** Matchmaking gets messed up**
TrueSkill rank is supposed to match you up against players of equal skill. Let’s say you’re a master of Honda but a scrub with Dhalsim. You’ll now be matched up with intermediate level players. This is horrible. It’s too easy when you play honda and too hard when you play dhalsim. Under combined player/char rankings, you would correctly be matched up with master players when you pick honda, but bad players when you play dhalsim
*** Your rank gets messed up**
If you switch characters, your rank will be a wierd volatile mix of your characters.
*** Everyone else’s rank gets messed up**
You might think that you can just stick to one character and not have problems. But no. Let’s take the following example. Bob the shoto has a respectable rank of 4600 with his ryu. He’s pretty happy with his rank, and it reflects his skill. Thelo has been playing dhalsim and his rank is now an abysmal 2000. Thelo decides to play honda today. He gets matched up with Bob, and clobbers Bob.
Bob’s ranking will get HAMMERED, because Thelo (a 4800 level player) was playing with a rank of 2000. And beacause of the way the rankings are implemented, that loss will hit really, really hard. Thelo has played many many games, so the system is quite sure that Thelo’s rank is actually 2000. This means that it’s quite sure that Bob is less than 2000. So it might take Bob a very very long time to get his 4600 rank of ryu back. Better hope that Bruce Askew hasn’t been messing around with Fei Long, Bob!
So the current ranking system is terrible and doesn’t measure what its supposed to. In general, do good players get good rankings under the current system? Yes. In general, do bad players get low rankings? Yes. But it’s certainly not usable at the level of detail the OP wants from it. This could be easily solved with character/player rankings, though.