Scoring System Changes Implemented
Well, it's done.
Today, I implemented the display changes to show the new system in action.
You can read the previous entries in this blog to see how these changes came about.
First change is the voting forms. The old form has been redesigned with the new look and the wording has changed. It's very close to the temporary change that I implemented at the beginning of December, but with one major difference. The 10 is no labeled like before; the current version says 'Most Amazing Story', which makes it more attainable.
Another change is the reversal of the order of the grades. It used to be that the 10 was the first item under the mouse, now it's the furthest one. This is a subtle change, but one I implemented to encourage readers to think before they cast their vote.
The second change is the availability of the Expanded voting form. I designed the system to switch easily between the two forms and be able to set the default from within the form itself. This functionality relies on some JavaScripts. For those of you who run without JavaScript or your browser doesn't support the advanced functionality, if you want to use the Expanded voting form, you need to change the preference in the 'My Account' page. There is a new entry titled 'Edit Voting Form Preference'.
The third change is the display in the listings. I added a new column to most listings pages titled 'QScor' (the 'e' is missing from the word score to keep the column as narrow as possible).
For those of you who didn't read the previous blog entries, this column contain a new score calculated by comparing the story's usual score with the median of the scores of all the stories posted in the same period as the story in question.
For example, if the story you're looking at was originally posted in 2000, the Qscore is calculated based on the median (the midway point) of all the scores of all the stories posted between 1998 and 2001. After 2001, it goes by year, until today.
This calculation is designed to take into account the general voting patterns of the period to compensate for the constant upwards score creep. Stories posted in earlier years have lower scores generally, regardless of the quality of the story.
Both scores will be displayed side by side for the next year or so. After that, the average story scores (the old ones) will be removed and only the Qscore will stay.
Due to the change in wording of the voting form, newer votes will tend to be lower generally, resulting in lower average scores than what you're used to. The Qscore is designed to compensate for that, so for newer stories, the Qscore will be more consistent. So you should start getting used to seeing and relying on the Qscore instead of the average score.
Theoretically, a Qscore of 6 and over means a good story. A Qscore of 6 is the equivalent of an old score of 8.8.
The last change is the addition of the TPA score. It stands for Technical/Plot/Appeal score. It's a composite score derived from the new (optional) Expanded voting form. It works almost like the old form. It's an average of all the votes cast, but each category is on its own and no fractions. So if somebody gives a story a 9 for Technical Merit and 8 for Plot and 5 for Appeal, then the story will have a TPA score of 9.8.5 if another person give it 4, 5 and 10, that means its TPA score will become 6.6.7, the count of the TPA votes is to the left of the TPA score.
The TPA score is designed for authors and those readers who prefer to give detailed feedback to the authors.
The regular voting system still allows for 1 vote per reader per story and you can't change it. The new Expanded voting form allows you to change your vote. However, if you vote with an expanded form, you can't change your mind to an overall vote and if you vote an overall vote you can't change it to an expanded one.
It's a lot to digest, and the additional info displayed definitely makes the site's pages more crowded, which I don't like. But for a while, we'll all have to live with it the way it is now, until most of you have gotten used to the Qscore and the TPA scores.
Six months from now (maybe earlier, we'll see how things go), the top listings pages will get sorted by the Qscore.
A year from now the 'Score' column will go away.
90 Days from now, the overall voting form will lose the numbers and will have labels only.
Update:
I've posted a follow up to this article:
Scoring Changes Implementation Follow up
Today, I implemented the display changes to show the new system in action.
You can read the previous entries in this blog to see how these changes came about.
First change is the voting forms. The old form has been redesigned with the new look and the wording has changed. It's very close to the temporary change that I implemented at the beginning of December, but with one major difference. The 10 is no labeled like before; the current version says 'Most Amazing Story', which makes it more attainable.
Another change is the reversal of the order of the grades. It used to be that the 10 was the first item under the mouse, now it's the furthest one. This is a subtle change, but one I implemented to encourage readers to think before they cast their vote.
The second change is the availability of the Expanded voting form. I designed the system to switch easily between the two forms and be able to set the default from within the form itself. This functionality relies on some JavaScripts. For those of you who run without JavaScript or your browser doesn't support the advanced functionality, if you want to use the Expanded voting form, you need to change the preference in the 'My Account' page. There is a new entry titled 'Edit Voting Form Preference'.
The third change is the display in the listings. I added a new column to most listings pages titled 'QScor' (the 'e' is missing from the word score to keep the column as narrow as possible).
For those of you who didn't read the previous blog entries, this column contain a new score calculated by comparing the story's usual score with the median of the scores of all the stories posted in the same period as the story in question.
For example, if the story you're looking at was originally posted in 2000, the Qscore is calculated based on the median (the midway point) of all the scores of all the stories posted between 1998 and 2001. After 2001, it goes by year, until today.
This calculation is designed to take into account the general voting patterns of the period to compensate for the constant upwards score creep. Stories posted in earlier years have lower scores generally, regardless of the quality of the story.
Both scores will be displayed side by side for the next year or so. After that, the average story scores (the old ones) will be removed and only the Qscore will stay.
Due to the change in wording of the voting form, newer votes will tend to be lower generally, resulting in lower average scores than what you're used to. The Qscore is designed to compensate for that, so for newer stories, the Qscore will be more consistent. So you should start getting used to seeing and relying on the Qscore instead of the average score.
Theoretically, a Qscore of 6 and over means a good story. A Qscore of 6 is the equivalent of an old score of 8.8.
The last change is the addition of the TPA score. It stands for Technical/Plot/Appeal score. It's a composite score derived from the new (optional) Expanded voting form. It works almost like the old form. It's an average of all the votes cast, but each category is on its own and no fractions. So if somebody gives a story a 9 for Technical Merit and 8 for Plot and 5 for Appeal, then the story will have a TPA score of 9.8.5 if another person give it 4, 5 and 10, that means its TPA score will become 6.6.7, the count of the TPA votes is to the left of the TPA score.
The TPA score is designed for authors and those readers who prefer to give detailed feedback to the authors.
The regular voting system still allows for 1 vote per reader per story and you can't change it. The new Expanded voting form allows you to change your vote. However, if you vote with an expanded form, you can't change your mind to an overall vote and if you vote an overall vote you can't change it to an expanded one.
It's a lot to digest, and the additional info displayed definitely makes the site's pages more crowded, which I don't like. But for a while, we'll all have to live with it the way it is now, until most of you have gotten used to the Qscore and the TPA scores.
Six months from now (maybe earlier, we'll see how things go), the top listings pages will get sorted by the Qscore.
A year from now the 'Score' column will go away.
90 Days from now, the overall voting form will lose the numbers and will have labels only.
Update:
I've posted a follow up to this article:
Scoring Changes Implementation Follow up
Comments
Fractions in the TPA would be basically meaningless. There are three criteria. The extra precision won't help much. I was going to implement a 5 grade system with stars and allow fractions, but then thought about it and came to the conclusion that 4.5 stars for Technical Merit would be the same as a 9.
So I kept the scale at 10 and restricted the results to whole numbers. It's the same result, but more elegant.
You just messed up a simple system that told the reader what they wanted to know. Are you by any chance a government employee? This is what they do. Take a working system and screw it up.
You can always scale the average to 10, but I think 10 options are too many for the users.
Technical Merit:
Spelling, grammar and overall readability
Plot:
Creativity, storyline and character development
Personal Appeal:
Story’s emotional and/or intellectual impact on you
...succinct, concise, & 'stroke-factor' really is included (w/o even using the word "sex", amazing! ;-) being 'emotional' of course, yet doesn't 'dominate' Appeal. There'll be complaints, no doubt, but all-in-all a nice compromise all around. Am also glad you didn't put TPA in a 'combined-score', 'preciate it. Nice touch too the option of 'expanded voting' as default.
Now to see if there's actually more realistic, honest voting;
& thus if it can actually become an useful & usable tool...
thanx Lazeez; again, you've gone above & beyond, nicely done!
w_newd (editor for hire)
nonof_urbiz@sbcglobal.net
By the way, I scanned through a chapter or two of a couple of the stories. One caught my eye because the TPA was all 9s and 10s. In one chapter alone I found over 50 grammatical errors,(and I'm a horrid editor - I have 2 editors and 2 proofers doing my tales now and I still find mistakes or receive e-mails about them.) In other words, whoever did the rating either didn't read the story, or else didn't know what to look for.
Actually, it's about what I expected to see. To change voting habits, I still say you have to educate the people who are giving out the scores.
1) It looks like you have at least three lines of text in the TPA Score box. Why not separate the scores so you have the three categories all on their own line so they are easier to read? For example:
Tech = [1-10]
Plot = [1-10]
Appeal = [1-10]
2) As an author I strongly prefer more information from readers, so why not make the expanded voting the default for all users? So few people bother to vote so I hardly imagine that those who are willing to vote will refuse to do so because they have to rank three categories.
3) I agree that in the long run you shouldn't be able to switch votes. However, since this is a new system would it be possible to allow a week amenity so that I can change my overall votes to technical votes (i.e., if you want to switch, you must do so by Dec 31)? I vote for every story I read… which means I can't cast a technical vote for any of my favorite stories.
Rowan
I understand them, having read this blog, but this is not something you can expect everyone to do (or even know about).
The OLD FART
just-this-guy commented in his blog how he didn't understand some things. Looking at his recent stories, one posted 12/04 had Score=9.22/QScore=7.62 and one posted 12/11 has Score=9.26/Qscore=7.47. The high-low flipflop implies that they are being scaled differently. My guess is that there is a different scaling for 12/02-12/06 when Lazeez implemented his first pass at changes, which was reverted back to the original until today.
But this suggests to me a glitch in the system: stories posted close to a date boundary where the scaling changes presumably use the scaling for the day they were posted, but may have many or most of their votes actually cast after the scaling changed. Or is this accounted for somehow? Without recording each vote, the only way I see to do this would to remember separate averages for each scaling period, which is complicated and can't be done retroactively. Thus there may always be problems with some stories' Qscores.
Does the TPA contributes to the total score (I assume it does) and if so, how is it calculated?
Finally, I still the the 'A' should be advertised vs. appeal. At least one author I can think of has complained that when they wrote a story outside of their normal style , it received very low votes. It would be far better to have a way to give public feedback to the authors on how the quality of job they did with their synopsis and coding.
If the synopsis sounds good, the codes are appropriate to the type of story I want to read, and the plot is rated high, I will probably like the story. But with an appeal score, what other people think about the story is all I have to judge by, problem is without knowing anything about the other people who voted, it might as well be a random number.
If this score does become a representation of how well the author did the judge of telling me about the story, then all of the previous comments about the technical score apply. You need to be able to change this score if the author fixes the synopsis or the codes, then this score will also need to be changed.
Sorting issues have all been cleared and the search and sort by Q Score has been added to the relevant pages.
Even my proofers miss things. It's just a fact of life.
I do see where Technical could be useful to readers and even authors -- the authors who care enough to do anything about their problems.
However, when it somes to plot and appeal, that's another story.
I know that not all my plots are great, but I get downloads and I get positive feedback that tells me they're at least worth something.
And I know my stories won't appeal to everyone.
As long as they appeal to some or even the majority of my readers, which I also know from downloads and feedback, that's all that matters.
As a reader, I won't be using it to vote or to determine whether or not I read a story.
I write a note to authors to tell them how I feel about their stories and I let them know if they need help in the technical aspects of their story.
I choose to read stories by the descriptions and the codes not by what others think of the plot or how it appeals to them.
In other words, the entire TPA is pretty meaningless to me as an author and as a reader.
If the TPA score doesn't have any input to the Qscore, doesn't that mean the more people who use the expanded scoring, the more static/stale the Qscoring system will become? If the one does have an affect on the other, how will/does this work?
Thanks, ajohnson
Is not the simplest solution the best? You should be able to vote on every chapter, not just for a story as a whole. If you want to give the authors the opportunity to defend themselves against "undeserved" low ratings, fine - give them the ability to strike a vote. But also make sure there is some kind of penalty for doing so. If readers see a story has a score of 10, but has had 9 votes stricken out of ten votes, they'll know what's really going on.
One score is enough, we don't need to worry about the Russian voting bloc skewing the technical scores.
"Fractions in the TPA would be basically meaningless." Well, right now, all the scores are meaingless, including and especially the new ones. They're to difficult, too time consuming, too ridiculous and in no way do they actually solve the issue they were creating to discourage - unfair voting.
The scores hurt my eyes right now. I don't WANT to vote on a story if I have to read directions to do so.
I'm not electing a president here. I -want- to be able to vote up a story I like, or vote down a story I don't like, and I don't want to have to have an astrophysics degree to do so.
Look, let's use Frank Downey as an example. If I remember correctly, Frank is or was one of the most outspoken authors when it came to people giving him crap scores. It seems to the current system actually DOES work more often than not - despite those bad scores, he STILL ranks as one of the most popular authors on here.
If some scat fic gets voted through the floor, is that really a reason to change the system, or is that the sound of the majority making their collective opinion felt? I'm not saying that scat fics don't have a place, but I think the readers (the ones who make this place popular, and pay those premier fees) should have the right to make -their- preferences felt.
I've given high scores to multi-chaptered fics that had some scat in it. I suppose you could say the whole was greater than the sum of its poots. My ideal solution would be to score by chapter. I could rate all the chapters as I saw fit, and the authors could see what "sells" and what doesn't.
I don't think that would result in any real kind of homogeneity - rather, I think it forces authors to take a stand AND it forces them to bring us with them for the ride or pay the price. Look at John Wales, Morgan, Hungry Guy, Old Bill and ElSol - each writes VERY unique stories that are largely (if not entirely) on the far edges of the bell curve that comprises popular content on this site, and each has succeeded in their own special ways, because of AND in spite of their scores.
It seems to me that a lot of work has been done to make voting more difficult, but I don't see how the new system will make anything -better-. No fancy AJAXing or javascript is going to make the new scores more comprehensible, easier to use, or even as useful as the current system (or better, voting per chapter).
Above all, the new system doesn't do the two things that are most important: a) prevent or mitigate the effects of the "bad voters" b) allow the rest of us to simply express our level of enjoyment.
And frankly, being able to enjoy a story is the only reason any of us should be here at all.
How reliable can a score be if you (as a reader) can only vote 9 or 10 unless you want to feel like you are punishing an author who doesn't deserve to be punished?
With regards to the technical score being changed when an author fixes spelling mistakes, why not just give author's the option to reset their story score when they go through the repost story/chapter section? You can do that on MySpace and there doesn't seem to be any problems with it.
Of course, if you reset your score, it's technically a new story, so you should probably reset the download count too. But leaving such decisions to the author is probably best.
As usual, I've got no complaints about the new scoring. SOL is a free system and anyone who's complaining ought to consider exactly how much Lazeez is being paid for all his efforts on our behalf. If you've paid money to participate, by way of donations to the site, then complain all you want, or take your ball and go home, but if you're just here because it's free then keep all your criticism on the constructive end.
-PB1
I note that the lion's share of your detractors at this post are anonymous. I wonder why.
I've switched my preferences to the expanded scoring system. I'm onboard. I like it. In fact, I believe you have done about the best job available where a perfect solution just isn't possible.
I do have one suggestion. What do you think about rearranging the order of the numbers, from TPA to PAT? This puts the most heavily-weighted scores at the left - making the score weighting more intuitive, IMHO - and makes the composite number more readily sortable, should you decide to make that option available someday.
Just my $0.02 worth.
Thanks for your great work at SOL... you are appreciated.
Peter
Happy Holidays
I wonder why think I'm saying anything at all about Lazeez. I'm commenting on the scoring system - if you think I'm doing otherwise, well, that's your mistake.
Beyond that, a) what's the point of putting a name in the comments, since this system doesn't track with SOL's? b) who says we all have blogger accounts c) who says we want to reveal personal information? d) anonymity does not = insincerity.
Lazeez runs a great site, and I'm happy to see how hard he works at keeping the site fresh and add new features and all that.
But just because I don't like the new system (and posted such comments anonymously), there's no link between that and some people assertions that I'm against Lazeez or the site.
That's like saying people who are against the Iraq war are anti-american or anti-veterans. Those same "unamerican" people are the ones fighting to get injured vets more disability money than the $230 a month the Bush administration currently pays.
when I sarch I cnot find them
I, personally, like to let authors and others know what I like or don't like about a story without having to send e-mail feedback (that other readers don't get a chance to see, anyway).
Come on, folks, let's give this TPA thing a realistic chance so we can see if it works out without sabotage.
Merry Christmas, ajohnson
Thanks for holding up a clearer mirror.ajohnson
I would like to see something that excludes scores that are more than five or six points lower than the average scores the story has already received. Maybe that would work to prevent the sabotage that goes on by the fans of other authors. I suspect by don't really know that I've been a victim of that.
I'd also like to see, perhaps just on the author's page, a list of votes by score, or a percentage of votes by score. For example:
10 30%
9 50%
8 10%
7 5%
6
5
4
3 3%
2 2%
OS
After reading the comments and the blog from just-this-guy, I can confirm that Qscore is funny and unreliable as anything else that came before, but then I was expecting it to be nothing else! :-)
I was watching carefully the old and the Qscore and the votes since the announcement. With 770 votes, and old score of 9.58, my Qscore came to be 8.74, and then with two new votes, Qscore moved down to 8.52 while old score stayed the same, and then additional three votes brought the Qscore up to 9.16. Since the Qscore tends to lower the old votes and manipulates the new votes to a lesser degree, I think the latest votes have been at the extreme ends of the spectrum (1s and 2s together with 9s and 10s). So what has changed? Nothing! It's GIGO (garbage in, garbage out) again!
You know, if Lazeez is going to apply a mathematical model to pull down the scores in relation to some median value, why doesn't he look at the voting process itself and apply a cut off to votes that differ too much from the median? That is as good as the method he's using now, if not a superior method, because it would make it more difficult for the trolls and the cheerleaders who vote on the extremes of the spectrum, which is what has caused the inflation of scores as well as all the gripe from many authors, including me.
And like the old scoring system, the new TPA system is being inflated, too. The idea is good, and I have no argument about it, but it doesn't work, because the implementation is subject to all kinds of abuse that hasn't been adressed by either system (old or new), which has caused the implementation of Qscore.
.b was right about his assessment and comments on TPA. I've checked several of the stories with the TPA scores, and as expected there were numerous technical errors where the story was given a high T score! English is a second language for me, but I'm fluent, and make use of several editors for my own writing to make sure the typos and such are minimal. And if I had to score my own writing on the technical side, I would give it a 7 (or perhaps an 8 if I have to be a bit more generous), and that despite the fact that I receive mail from dedicated readers which list less than 4-5 errors in each chapter (where the lists end up being almost identical). I am also more forgiving when I read some one else's story which contains more errors, mentally correcting the material (such as replacing their with they're when the author meant they are), and have more trouble with crappy punctuation. Those technical errors don't bother me or interrupt my reading as much unless it gets beyond 20-30 errors in a 5-6000 word chapter, and then I throw up the towel. So when I see something with 20 errors in a 2-3000 word long segment, and see it is given a tech score of 9 or 10, it tells me that the system isn't working.
Some of the commentators said the old system is working! LOL! Right! It's useless to use as a criteria to find any good reading material. It's inflated. Qscore is useless, too. It's just an adjusted value that pulls the old scores down a bit, but in essence, it doesn't do anything. It's like when a country has too much inflation, and the money becomes worthless and has too many zeroes (like Italy after World War II). You can take out the extra zeros and instead of a 1,000,000 liret, designate it as 10 liret, but what did you change? Nothing! The inflation is still there!
I think Lazeez should put his thinking cap on and consider the pros and cons of the new system, especially the TPA system, and perhaps do not open it up to the public but a the paying members and authors and such. And he should consider how to address the core issues: how to deal with the votes that fall way beyond the average a story has scored, because that indicates there's a problem! Pulling the scores down to fit a median is as arbitrary as letting the old score system continue. I think cutting off votes that are extreme is a better solution... maybe it's not the best, but it certainly is better than what is being allowed to happen!
A few words on the TOP lists. Good changes are implemented. Yes, following up the downloads (for the last 7 days) is a bit of a hassle, and since the latest change it's almost impossible for many stories (old or new) to make it into the top weekly downloads lists unless they break the 2000 downloads limit! For example, unless the story is a serial with several chapters which averages something less than 1000 downloads per chapter, the story won't make it into the list except when the author starts posting more frequently to build up the downloads to 2000 figure. Depending on the author, that means he has to write mroe and post more frequently, or cut down the sizes of the chapters to icnrease the number of postings. Personally, I don't have the time or the inclination to spend more time on writing than I currently can, nor do I want to increase my posting schedule or change the sizes of my chapters. They will be what they are --to my recollection I haven't posted anything less than 6,000 words as a minimum for a chapter, and sometimes as much as 10-12,000 words, and my last chapter (56) was arbitrarily broken into two segments by SOL submission system, because it exceeded the length limit for a single chapter! But what this change in the system means is that we may end up seeing some adverse effects. Persoanlly, since the change, I began to spend more time checking the daily updates lists and stopped using top lists completely. I used to check the weekly top downloads list to see how a new story was doing, but that is no longer possible and since I don't rely on scores, I have hardly anything to use to filter stories. Now, it's back to carefully browsing through the daily updates lists and checking story codes and descriptions!
Maybe Lazeez should increase the TOP lists to include more than 20 stories, because as it is now, we only see the same 25-30 stories dominating the top 20 list!
Dai
Sure is a pity. From what I gather from all the comments, most people must receive great comfort in having other people make their decisions for them, as other people seem to find it empowering to determine reading selections for for the confused majority. Verrry interesting!
Scoring Changes Implementation Follow up