Rethink Our Chart Difficulties

[TheSaxRunner05]

I think with this new FFR Rank system, we need to rethink our difficulty system. In the past, it seems many charts are rated based on how hard they are to AAA, and not necessarily how hard they are to SDG/Teen. I made up a post with a bunch of examples, but then my toddler came up and smacked by keyboard, and losing everything I had typed. Some songs are just gimicky (giving an artifical boost to those skilled in a chart that only tests jacking, etc), some just have a low note count/dps compared to the rest of their range, making them easier to SDG.

So I'll just name some of the charts I was going to mention -

300
Yoshi's Cookie
Famouz
ABCDeath
The Divine Suicide of K [Heavy]
Fei Longer
Goatstep
Streets
Aim Anthem
Club
Novo Mundo
Within Life
RunnyMorning
Einstein-Rosen Bridge

Discuss? Should we change HOW we rate files because of our new ranking system?

[TheSaxRunner05]

Maybe we could use player data to generate a difficulty based on player ratings vs the score they acheived on the chart? While there would be outlier scores, in aggregate this could take subjectivism out of chart difficulties entirely for songs with over a threshold of complete plays.

[XCV]

Originally Posted by TheSaxRunner05

Maybe we could use player data to generate a difficulty based on player ratings vs the score they acheived on the chart? While there would be outlier scores, in aggregate this could take subjectivism out of chart difficulties entirely for songs with over a threshold of complete plays.

It might, MIGHT be impractical to predicate the song difficulties and player rankings on each other. Especially with FFR's throughput, you'd have to be very careful to ensure that feedback loops don't happen, although outside of lots of people playing the same song (tournaments) the changes should be minute enough to not cause any adverse effects.

That said, I like the idea of taking confidence intervals over score data, even if it ends up destroying the nice little 1-100 scale we have going on. Honestly though RATO's gonna be absurdly high no matter how you set this up, and it's a little hard to fit the beginner songs into this framework. Frankly I don't believe there's any meaningful difference between a 2 and a 5 on a linear scale. Maybe if it gets logarithmic down there you might have an argument but [MATH NERD JIBBLEJABBLE EXPUNGED]

[YoshL]

create a system akin to stepmania/ddr, where charts are rated on a bunch of terms such as: streaming speed, burst speed, density, one handed patterns/jacking, awkwardness

[XCV]

Originally Posted by YoshL

create a system akin to stepmania/ddr, where charts are rated on a bunch of terms such as: streaming speed, burst speed, density, one handed patterns/jacking, awkwardness

How dare you behave so spreadnormatively /s

This would also be totes really cool, but a.) it'd probably have to be done algorithmically, no big whoop (except for "awkward patterns" how even??), b.) as with the groove radar it'd be parallel to (but on the same plane as) the scale proper, and c.) actually implementing it into the engine would likely require a layout overhaul, which is only problematic because we've already hit 640x480 and cramming much more in there probably means tiny text or some other sort of eyesore. Either way we might have to de-emphasize the parts of the site that aren't that important w/r/t the engine (friends list???) to make room. However idk anything about AS3 or graphic design and someone else should chime in so that I can stop talking out of my behind.

[YoshL]

lmfao i actually didn't even think of spreadnormative i'm a poop

anyways, i think, even with all playstyles something that becomes a problem is songs that have notes that occur on one arrow very rapidly. could probably do something with that

should code .exe engine for maximum features and no more frash frash levorution amirite

[Guest15937]

Originally Posted by YoshL

one handed patterns

So, patterns, then

[LordCarlos]

Yeah man, it should be based on how well players do on the songs.

I would estimate about 7000 games are played on FFR daily (just a guess). And since there's a song library of about 2300, we could average that each song gets about 3 plays each day. However it's much more likely that some songs are attracting *a lot* more attention than others, especially high-level songs.

But there's a couple ideas for how we could create a novel ranking system (for both players and songs) independent of the FFR total points ranking system and completely from scratch.

The upsides: much greater accuracy in song difficulty and player ranking.
The downsides: much more complicated than the simple 0-99.

Basic Idea:
A player plays two songs. The %score is judged for both songs.

0) Both songs are temporarily set to undecided difficult ranking.
1) The song with the lower % score increases in difficulty ranking.
2) The song with the higher % score decreases in difficulty ranking.
3) A song less than x% does not count (x is some lowish percent to remove outliers, prevent abuse).

Each song thereafter is compared to the previous songs that the player has played.

Problem:
It would take a lot of judgements to accurately assess the difficulty of all songs.
Solution:
Use all players' one-week score history to evaluate the difficulties of the full library of 2332 songs.

Example:
Suppose around 50000 games have been played within the last week by 1000 active members. So on average, 50 games per person per week. This allows more than enough ranking judgements. Each person will yield a small profile of which songs he or she did good/poorly on. Consider Sarah got 100% on one song, 90% on another, and 60% on another. From just 3 songs, we can now rank those songs relative to each other. Now imagine that Steve got 90% on the same song that Sarah got a 60% on. Now we can rank Steve higher than Sarah.

Sarah has an average % score, and all songs are compared to her average % score for the week to make song-ranking judgements. Likewise, for any given song, there will be an average %score of all players who played the song in the past week. From there, player-ranking judgements can be made. Judgements will be accumulated, weighted, averaged, etc. Consider Mike gets 100% on Death Piano. Well, then Mike is the best.

So as you can see the number of comparisons that can be made is endless and the data available is huge enough to make some pretty killer player/song ranking assessments. Some amazing algorithm could sort the players and songs out to a crispy certainty.

----

Did some more pondering about this. Consider 1000 active players with 50 games each during the past week. The number of song comparisons than can be made per person is n(n+1)/2. Therefore on average, each person would offer 50x51/2 = 1275 ranking comparisons. x1000 = 1.3 million comparisons. Not too shabby! :3

[Guest15937]

Originally Posted by LordCarlos

Consider Mike gets 100% on Death Piano. Well, then Mike is the best.

No, just means Mike was using a bot and will soon be banned

[XCV]

Originally Posted by LordCarlos

all songs are compared to her average % score for the week to make song-ranking judgements. Likewise, for any given song, there will be an average %score of all players who played the song in the past week. From there, player-ranking judgements can be made. Judgements will be accumulated, weighted, averaged, etc.

This is the basic idea but

Any player's "skill ranking" or whatever it's called is dependent on their best 4 scores, which are weighted by song difficulty

Making song difficulty dependent upon player rankings, to however small a degree, would be inadvisable without some additional analysis or some sort of negative feedback mechanism because if some schmo goes off and whores SM for 3 months, then comes back and sightreads an FMO SDG or something stupid like that, then there's ever so slight a chance that the FMO will get easier because a comparatively low-ranked player busted some chops on it, thereby lowering the player's rank, lowering the chart's difficulty, et cetera, and now Perfect Cherry Storm's 10 out of 100, how did we get here.

Frankly I'm not sure if there's any efficient way to prevent this outside of updating rankings and/or difficulties less aggressively (once, at midnight) and that raises its own array of questions.

And honestly it's just a few numbers on a finger dance game on the Internet. Who really cares besides the 200 people left in this community. /ayylmao

[LordCarlos]

Originally Posted by XCV

And honestly it's just a few numbers on a finger dance game on the Internet.

True. I may have gotten overly excited.

Originally Posted by XCV

if some schmo goes off and whores SM for 3 months, then comes back and sightreads an FMO SDG or something stupid like that, then there's ever so slight a chance that the FMO will get easier because a comparatively low-ranked player busted some chops on it

It's just an outlier. All the other millions of comparisons will drown out weird occurrences like that. Also, if a low-ranking player does good on a high-ranking song, the player rank would be more affected than the song rank. If a person hasn't played a song in over a week, the song ranking wouldn't change.

The good news about such a system is that it would be completely harmless to try out. In fact, we would be able to scale the difficulties to a 1-99 system and side-by-side compare them to the original before making a decision to release it. The only thing that's needed is a 1 week score history.

Or I could create an unofficial ranking list of the songs/players.

[FF_rules]

Originally Posted by TheSaxRunner05

ABCDeath

I agree with this completely. There's no way this song should be even in the 50's range. Just played it for my first time and got 8good on it... when I can't even play most 50 songs with 30 goods.
Edit: #1 56 ABCDEath 50.630 375 8 0 0 2
REALLY???

-Reasoning; It's basically 30 seconds of piss easy D20 and 10 seconds of really tight mash (my 3rd play through I somehow mashed all perfects on the first up down streams, another time I got 12 goods.) Ofc I can see how it's high D50 if the song only had those 2 mash rifts for an entire minute but it's basically 50 seconds of D20-D40 and 10 seconds of mash.
http://www.flashflashrevolution.com/...replay/307809/
I mean look at that

[hi19hi19]

Face it, once you get remotely decent at the game (like, mid D1 basically), the game becomes about AAAing things.
I don't particularly enjoy that the game is constructed this way, but that's how it is.
The difference between the perfect and good windows is so colossal compared to how accurately humans can time, so FFR just winds up being all AAAs.

The difficulty system will likely remain primarily determined by AAA and low SDG difficulty until major changes in the game take place (read: higher FPS or MS timing implemented)


If you have specific charts in mind- for example I agree ABCDEath's rating is an out-of-place relic of Tass era- there's a forum specifically meant for that.

[FF_rules]

So I took the advice and posted a thread for ABCDEath in "Contested Chart Difficulties" subsection. But the thread isn't there after I posted it; does it just stay hidden until a mod approves it? (If that isn't the case; I'll just let someone else do it)

[XelNya]

Originally Posted by FF_rules

So I took the advice and posted a thread for ABCDEath in "Contested Chart Difficulties" subsection. But the thread isn't there after I posted it; does it just stay hidden until a mod approves it? (If that isn't the case; I'll just let someone else do it)

It does sit there until approved.

[Kanzas]

I would always get 1-0-0-1 on ABCDEath, like everytime. One day I miraculously was able to read it as a DR. Seuss book and AAAd the damn thing. 56? Doubt it.... A really complex high 40? Sure.

[TheSaxRunner05]

Originally Posted by XelNya

It does sit there until approved.

I sent somehing like two weeks ago lol

[hi19hi19]

I should note that, as a rule, files tend to wind up being rated by their hardest part.
This is a consequence of the game mechanics leading the scoring to be AAA/low SDG-focused.
For some extreme examples of this effect consider the ratings on party 4u "holy nite mix" v1 and Crowdpleaser, but of course it also holds true for less pants-on-head stupid difficulty spikes too.

So in general files with notable short difficulty spikes tend to be rated higher than it initially appears they should be- you typically quickly progress to the point where you have 98% of the file mastered, but that 2% will keep you from the AAA longer than you might think, raising the rating.

That said I'm glad the ABCDEath thread is made, it definitely needed one... I'm not pretending the system is perfect lol

[TheSaxRunner05]

What would be really helpful is if we could run a query on the songs that make the most people's top 4, and then decide whether there on so many people's top scores because they are actually good scores, or if they are overrated.

[Rapta]

Sounds like a good idea @post before mine