New Skill Analysis Metric (Trial Phase)

[Trumpet63]

Status: Dormant
See Also: All Current Recorded Zenith Scores

zenith-score.xlsx (v.2.1.2)
If you have Excel, you can use this. Go to your level ranks page and copy the whole table, including the column titles (in Firefox, it's some kind of Ctrl+Shift with clicking and dragging from just outside the table). Paste it into B1 of Sheet1, go into Sheet2, and Sheet3, and all the results will be there. (Credit to Silvuh)

Alternatively, you can post in the thread and I will return the results to you, after which I can use your scores to improve the formula.

The Zenith Score is a measure of skill potential in units of difficulty.

This is the most recent scale by average:



Pretentious name is pretentious, I know. _Zenith_ was the first person other than myself that I used this on, and I thought it sounded cool. The graph above will be called the Zenith Curve, and the number you get from it will be called the Zenith Score. Zenith also had no idea I was posting this.

So far I'm only using Excel to do the calculations because I do not have a working knowledge of any programming language that can work with chunks of data such as this. In the end it would really only create issues with implementation if this became widely used.

Any feedback would be tremendously helpful. Having an opportunity to share this with people, and to get people excited about the enormity of the statistics here would make me very happy.

And just to give you a taste, the graph above indeed belongs to Zenith, and his Zenith Score is 72.01.

Volunteer Appreciation Here:
Tim Allen
PhantomPuppy
noname219
LethalMutiny
Callipygian
================

How the Original Works: (10 steps)

1. I copy the levelranks table (of course) and paste it into Excel.

2. Each score is sorted by the Good Count, and a value is assigned to other non-goods in the following way:
Perfect = 0
Good = 1
Average = 1.8
Miss = 2.4
Boo = 0.2
(Each of the P/G/A/M/B counts are multiplied by these values, then added)

3. Every score with 20 goods or above is removed from the table.

4. Each score is then assigned a new value, based on the following formula, hereafter referred to as the Weight.
Weight = (20-[Good Count])*[Difficulty]

5. Next, the weights for each difficulty are added together separately and arranged in a new table in the following way:
Column1 = "Difficulty" || Column2 = "Sum of Weights"

6. (At this point the graph could be made) The actual Zenith Score is obtained by first isolating all of the weights associated with the difficulty AFTER the highest weight value, or the "peak". i.e. All the values to the right of the peak.

7. Take the sum of all the values in the Sum of Weights column that have been isolated, this will be called Sum1.

8. Make another column that is the product of Difficulty and Sum of Weights:
Column3 = "DW Product" = [Difficulty]*[Sum of Weights]

9. Take the sum of all the values in this new DW Product column, this is will be Sum2 (again, only the ones to the right of the peak).

10. Zenith Score = Sum2 / Sum1

News and Progress:

9/30/14 I considered changing the Sum of Weights to an Average of Weights (the Y-value). Doing this was an effort to account for the fact that the difficulties in the game are not evenly distributed (refer to chart below). However, I found that when the sample for a particular difficulty is very small, the weight will become artificially high (i.e. one song played at D70 will imply you have AAA's on all D70 songs).



10/1/14 Thanks to the help of Silvuh, I have been able to conduct many more trials in much less time. After re-testing my Zenith Score, I found it to be 39.95, which, according to the current model of the scale, puts me borderline D2/D3. Since I was one of the last to be placed in the Tenth Official, and was placed into D3, but was previously D2, it is evident to me that I am in fact borderline D2/D3. Based on this and mr_candy and Zenth's scores agreeing with their current division placement as well, I am still hesitant to say the original formula is terribly flawed.

I postulate that a change in the division scale could be a more effective solution than developing a new calculation method. Nevertheless, I and Silvuh agree that an intuitive solution to the difficulty concentration problem would be to divide each weight by the number of songs in each difficulty. Yet more testing is necessary to rule out the usefulness of either method, and indeed I still find that the Average of Weights method conveys something with the graph that is still very powerful. For today I have to leave this issue of the three methods alone because of time constraints. I'd like to add that I don't like the idea of punishing players for playing in a difficulty range that has a lot of songs, or in effect making those scores less significant, and this is what this new counter-weight method seems to do.

10/2/14 A significant sample of 30 players has been assembled into a table to examine the viability of the Zenith Score and its variations for division placement. Adjustments to the scale should come next. In addition, two changes are to be made: 1) The length of the spreadsheet needs to be enlarged to account for the song additions caused by the Tenth Official. 2) A complete list of note counts for public songs needs to be assembled and integrated into the spreadsheet. It is likely that the note counts have to be arranged in alphabetical order because that would be the only arrangement that is consistent among all players.

10/5/14 Note counts added, sheet extended, and non-fully-played songs discounted thanks to the Excel work by Silvuh. We are adding a couple new dimensions to the Zenith Score by including new versions of the scaled score in a attempt to grasp at the concept of skill, and try to accurately predict division placements. Also the original scaled score is receiving an overhaul to correct the one-score-in-a-difficulty problem. After the spreadsheet is fully updated, we will conduct all of the trials on the table again, and afterward move into profiling entire divisions' scores. We may be looking for volunteers to conduct this large number of trials.

10/12/14 After many bugfixes in the spreadsheet, thanks to Silvuh, we can finally move into the next phase of development by making a complete survey of the tournament and its participants. Hopefully this will help us determine which of the six methods currently in use are best, and help us to gain a deeper understanding of the statistics themselves.

10/13/14 The entire tournament has been surveyed, thanks to several volunteers, and the work of Silvuh and myself. In addition, the larger portion of analysis as been completed. We have found the average, standard deviation, maximum, and minimum in each division, and for each flavor of Zenith Score. The next task is finding out the ideal scale of scores that results in the lowest error count. Current error estimations are at about 19% for W4, over all the divisions. After determining the best method to make the division scales, and finalizing the error count, the next step will be to hypothesize changes to the formula that may reduce the error count.