The Effect of Random Fluctuations on the Rankings
Louis Nel models how the positions of players in the Rankings change even if everyone's ability remains constant.
In the periodic discussions of our World Ranking system it does not take long before somebody mentions the relative performance of some particular players to make a certain point. I believe that to be a questionable practice, because it ignores random fluctuation. None of us can know how much the "good year" or "bad year" of a particular player is due to a change in form and how much due to random fluctuation. To emphasize the influence of random fluctuation I have carried out the following numerical experiment. I replayed - in a manner of speaking - the entire 2002 season worldwide under a certain change in the Laws of Association Croquet. The change is very simple to describe - each game was decided by a coin toss. So every player had a win probability of 50% in every game. Accordingly, every player was given the starting Grade and starting Index of 2000. There was no change in "form". The "true" Grade of every player remained 2000 throughout while the published Grade was calculated as usual. The familiar World Ranking table was produced at the end of the year (see the table at the end of this message).
I believe this Ranking table to be instructive. However, it must be used with caution, because the players started with a Grade that is 100% accurate - a highly artificial situation. As a consequence, players who played only a few games, would not have as much chance to deviate from their true Grade as players who played many games. Therefore, to put things in perspective, we have to note at the outset how the Standard Deviation of the Grades varied with the number of games played. The small table (right) gives an indication. (There were 740 players who played 10 or more games, 479 players who played 20 or more games, and so on).
It appears that the Grades generally drifted away from the super-accurate start (technically, the Standard Deviation got worse) for about 50 games and after that the Standard Deviation settled down.
In a normal player population there will not be such an artificially accurate start. All the Grades could be regarded as having reached the stage of randomness that the idealized population below acquired after about 50 games.
All told, it appears that Grades normally could be expected to have a Standard Deviation of more than 65, regardless of the number of games played. I believe our ranking system can be tuned to possibly better performance as follows. If a concensus can be established about what form fluctuation can serve as benchmark(s), then one could program that agreed upon form fluctuation into the behavior of idealized players and experimentally obtain values for the Step size (= Class Factor) and the Smoothing parameter that will minimize the Standard Deviation of the Grades (or even the Indexes). That seems preferable (more objective) than to look at the relative performance of certain individuals under different systems. Here is the mentioned table for the 2002 World Coin Tossing Rankings :)
All rights reserved © 2005-2017