New Review System and Other Notes
I read all the comments on the previous post. One important issue that most of those expressing their opinions keep forgetting: the voting system should serve reader goals as well as author goal. While authors view story scores as a kind of feedback - which it is, an author should not forget that the score should also help readers decide what to read and what not to read. I know it's hard to think that there should exist a mechanism to help readers eliminate stories from their potential reading choices, but it is important.
As a reader, wouldn't you want something to help you wade through the thousands of stories on the site? There is no perfect system, but for a site like storiesonline where readers are limited in how many stories they can access per 24 hours, some kind of help must be provided for them to find something to read without wasting their download allocations. Codes and scores serve that purpose. No reader can judge each and every story on the site for themselves.
Specific issues:
Individual chapter scores: Don't work because the reader/voter has no real idea how a particular chapter fits within the grand scheme of a story, so while one chapter can be great fun to read, another can be hard to stomach, but necessary to the overall plot of the story. Readers should not be allowed to steer the direction of the story by a simple mechanism. An author should feel comfortable taking the story in any direction that they want without having to worry about the instant votes of readers that may not approve of the stuff in one or more chapters.
Author votes: Why shouldn't an author vote for their own story? The president of the US votes for himself. Plus, those who worry about skewing the score, come on, the author can vote only once; after 20 more votes, the author's vote becomes nearly irrelevant.
Thoughts on the Current Voting system
The current voting system, while not perfect, works well enough for most readers. But, authors looking for valid feedback from the score will be disappointed. The current score is merely an approval of the story in all facets combined by the readers' collective.
I'm contemplating dropping the voting system to a lower scale, from 1 to 5. I've been analyzing votes for the last couple of days and it seems that some values are almost never used. It seems that voters don't have enough to go on to judge in such small increments. I'm not sure about this step, but I'm contemplating it. And since the current voting system seems to be more of an approval system by readers, it would be better if the change was to be made, that the new scale would be of the '1 - Hated it', '2 - Didn't like it much', '3 - It was OK', '4 - Great fun' and '5 - Absolutely Loved it' variety. Of course choice of words would be thought out properly. Such a lower scale would be easier to understand by the voters, and authors won't mistake it for the extensive feedback system that they wish it to be. But, since the current system works well enough, I'm not really sure that I will be making such a change.
The New Reviewing System
The decision to re-implement the reviewing system has been finalized and I've started the feature draft.
Every reader will be able to submit a review for stories. The review will have the user's Login ID in it, so no anonymous reviews. That will cut down on those willing to submit a review, but since it's all anonymous to begin with, it's not really a step back in privacy, plus, anybody wanting to submit a review that affects the story and its author should be able to stand behind what they say. Also, since readers can change their Login ID, the old id and the new id will show if they do make such a change.
Reviews will have to be a comment (may have a size limit), and a three-criteria score. Plot and Character, Writing Quality and Appeal to Reviewer. There will be no stroke score; many stories have no sex in them so it shouldn't be a factor in judging a story.
I'm not sure how I'm going to combine all three scores to come up with one review score to be displayed in story listings. Of course, all individual scores will be visible in expanded-review view. Should grammar be as important as plot and character? after all, an author can have multiple editors going over their work, so grammar score would not necessarily mean that the author has a great grasp of the english language. I'm thinking 25% for grammar (can be greatly enhance by an editor), 25% for appeal to reviewer (reflection on personal taste of the reviewer) and 50% for Plot and Character (that's where the author's talent shows)
There will be no reviewer profiles, it's enough work to submit reviews, reviewers shouldn't worry about identifying their tastes and fetishes (which can usually be deduced from their reviews anyway.)
Every author will be able to choose whether to allow reviews for their stories or not (the way it currently is, I know for a fact that some authors prefer to refuse reviews.) So if an author chooses to refuse reviews of their story, nobody would be able to submit any review for their stories. However, the author cannot choose to accept reviews on per story basis, it's global on/off switch for reviews for all the author's works, not per story setting.
Anybody can volunteer to be a review moderator, but it's not automatic, they have to go through me to become a moderator.
A reader will be able to submit a review (it goes to moderation), edit an existing review (it goes to moderation before the changes take effect), delete an existing review that they have previously submitted and view the status and edit the reviews that they have submitted that are not on the site yet.
Each review will need to be approved by two moderators, so a moderator will be able to give first or second approval to a review, or reject a review. If a review is rejected by one moderator, it does not go to a second moderator and if it is accepted by the first moderator and rejected by a second one, then it does not go to a third moderator.
Authors cannot submit reviews of their own stories.
Authors who volunteer to be moderators cannot moderate reviews of their own stories.
Moderators cannot moderate their own reviews
Request for Suggestions
Of course, your suggestions are most welcome. If you think that I missed something important, don't be shy.
Update - 2004-11-09: 3:15pm
• The voting system is going to stay the same. There is no need for a change.
• Each review must have two approvals or two rejections
As a reader, wouldn't you want something to help you wade through the thousands of stories on the site? There is no perfect system, but for a site like storiesonline where readers are limited in how many stories they can access per 24 hours, some kind of help must be provided for them to find something to read without wasting their download allocations. Codes and scores serve that purpose. No reader can judge each and every story on the site for themselves.
Specific issues:
Individual chapter scores: Don't work because the reader/voter has no real idea how a particular chapter fits within the grand scheme of a story, so while one chapter can be great fun to read, another can be hard to stomach, but necessary to the overall plot of the story. Readers should not be allowed to steer the direction of the story by a simple mechanism. An author should feel comfortable taking the story in any direction that they want without having to worry about the instant votes of readers that may not approve of the stuff in one or more chapters.
Author votes: Why shouldn't an author vote for their own story? The president of the US votes for himself. Plus, those who worry about skewing the score, come on, the author can vote only once; after 20 more votes, the author's vote becomes nearly irrelevant.
Thoughts on the Current Voting system
The current voting system, while not perfect, works well enough for most readers. But, authors looking for valid feedback from the score will be disappointed. The current score is merely an approval of the story in all facets combined by the readers' collective.
I'm contemplating dropping the voting system to a lower scale, from 1 to 5. I've been analyzing votes for the last couple of days and it seems that some values are almost never used. It seems that voters don't have enough to go on to judge in such small increments. I'm not sure about this step, but I'm contemplating it. And since the current voting system seems to be more of an approval system by readers, it would be better if the change was to be made, that the new scale would be of the '1 - Hated it', '2 - Didn't like it much', '3 - It was OK', '4 - Great fun' and '5 - Absolutely Loved it' variety. Of course choice of words would be thought out properly. Such a lower scale would be easier to understand by the voters, and authors won't mistake it for the extensive feedback system that they wish it to be. But, since the current system works well enough, I'm not really sure that I will be making such a change.
The New Reviewing System
The decision to re-implement the reviewing system has been finalized and I've started the feature draft.
Every reader will be able to submit a review for stories. The review will have the user's Login ID in it, so no anonymous reviews. That will cut down on those willing to submit a review, but since it's all anonymous to begin with, it's not really a step back in privacy, plus, anybody wanting to submit a review that affects the story and its author should be able to stand behind what they say. Also, since readers can change their Login ID, the old id and the new id will show if they do make such a change.
Reviews will have to be a comment (may have a size limit), and a three-criteria score. Plot and Character, Writing Quality and Appeal to Reviewer. There will be no stroke score; many stories have no sex in them so it shouldn't be a factor in judging a story.
I'm not sure how I'm going to combine all three scores to come up with one review score to be displayed in story listings. Of course, all individual scores will be visible in expanded-review view. Should grammar be as important as plot and character? after all, an author can have multiple editors going over their work, so grammar score would not necessarily mean that the author has a great grasp of the english language. I'm thinking 25% for grammar (can be greatly enhance by an editor), 25% for appeal to reviewer (reflection on personal taste of the reviewer) and 50% for Plot and Character (that's where the author's talent shows)
There will be no reviewer profiles, it's enough work to submit reviews, reviewers shouldn't worry about identifying their tastes and fetishes (which can usually be deduced from their reviews anyway.)
Every author will be able to choose whether to allow reviews for their stories or not (the way it currently is, I know for a fact that some authors prefer to refuse reviews.) So if an author chooses to refuse reviews of their story, nobody would be able to submit any review for their stories. However, the author cannot choose to accept reviews on per story basis, it's global on/off switch for reviews for all the author's works, not per story setting.
Anybody can volunteer to be a review moderator, but it's not automatic, they have to go through me to become a moderator.
A reader will be able to submit a review (it goes to moderation), edit an existing review (it goes to moderation before the changes take effect), delete an existing review that they have previously submitted and view the status and edit the reviews that they have submitted that are not on the site yet.
Each review will need to be approved by two moderators, so a moderator will be able to give first or second approval to a review, or reject a review. If a review is rejected by one moderator, it does not go to a second moderator and if it is accepted by the first moderator and rejected by a second one, then it does not go to a third moderator.
Authors cannot submit reviews of their own stories.
Authors who volunteer to be moderators cannot moderate reviews of their own stories.
Moderators cannot moderate their own reviews
Request for Suggestions
Of course, your suggestions are most welcome. If you think that I missed something important, don't be shy.
Update - 2004-11-09: 3:15pm
• The voting system is going to stay the same. There is no need for a change.
• Each review must have two approvals or two rejections
Comments
I find it hard to believe that their preferences won't bleed into their "Plot & Character" scores as well.
Wouldn't it make more sense for people to only review stories that at least mostly matched their preferences?
Real book reviews are done in this way. Typically they have someone who knows and enjoys Science Fiction do reviews of those books, not someone who primarly reads romance novels.
Now, I'm not sure how you could enforce this, but it seems to me that if someone gave a review with a low "Appeal to Reviewer" score, that the entire review becomes nearly worthless.
My 2 cents...
LordGarth
I have been reading the comments with great interest. What I find interesting is they (the comments) exclude me and do not give me the opportunity to vote on a story I find appealing, interesting, and enjoyable. I am a commoner, by this I mean I hold a BA in Individual Studies. My knowledge is not at the level I notice EVERY misspelled word, missed comma, and most definitely each grammatical error in a story. I read (and write) for the pure pleasure of it. (Microsoft has already graded me a 9.5)
I have my personal preference of stories (as I’m sure each person visiting your site) and from the comments I feel I am not qualified to vote. I don’t hold a BA in Science, MBA in English, or PhD in Psychology. English is not my secondary language (ESL), and for that, I am grateful. I say that because ESL most definitely would preclude me from reviewing and voting.
Yes, I’m being facetious.
I’m a reader and author (that’s facetious also), but I look at the scores as a guide, not a grade. Would I like my stories to score the max, definitely, but that would mean my writing would have to appeal to EVERY reader, and be perfect in every way possible. I believe I am average as a reader, piss poor as a writer, and worse as a rater. I read a story, but I look at the codes beforehand so as not to be disappointed. Should the codes be inaccurate I feel cheated, or subjected to bait and switch, for a better expression.
Perhaps the simplest method of getting more accurate scores (readers opinions) would be to define the options of scoring.
10-Excellent (storyline, grammar, punctuation –no errors found)
9-Very Good (1-2 errors- storyline, grammar, punctuation)
8-Good (3-5 errors- storyline, grammar, punctuation)
7-Above average (good storyline, but minor errors in grammar, punctuation)
6-Average (storyline, and errors in grammar, punctuation)
5-Fair ((storyline, grammar, punctuation)
4-Needs work (storyline, grammar, and punctuation)
3-Poor (all areas)
2-Bad (all areas, perhaps you need to rethink your ability to write)
1-A bunch of jumbled words
I believe most authors appreciate the comments/emails received. In reality, these are more uplifting and mean much more than the score. However, those comments definitely coincide with scores.
Great site, and I appreciate your readers.
Sausage Dog
As for the voting system, taking the current score out of the voting box was a big step in the right direction.
I find I vote according to emotion. If a story made me identify with or even love/hate a character, it will get high vote from me. If I feel like crying when something bad happens to a character, I'll vote 9 or 10.
A decent plot will boost my emotional score.
Rather than hurt a story that has grammar/spelling errors with a low score, I just inform the author of the errors. Authors that care about their work will correct the problems.
I would gladly volunteer to be a review moderator. That way I could get the scoop on more stories than I could read on my own.
SOL member johnweatherby
But if a story is in progress, you can vote on it. That means the reader is NOT voting on the complete story which everyone seems to agree is important. So if you don't want chapter voting, you have to turn off the voting system until the last chapter.
I think changing the scale from 10 to 5 is a good idea. (By the way, how will you address all the old scores out there based on the 10 scale?) I would still like, as an author, to see how many 5's, 4's, 3's, etc. I got per story. If necessary, make it available only to the author in his/her stats. Maybe we could click on the score to get the details (like drilling down in a data warehouse).
As to reviews, I wonder how many readers will bother reading a review. One danger of doing so is that it may give away a surprise in the story. I think the rating system serves the reader better. If the author wants a review, that's different. That would be more like feedback. I don't know if this is okay to post here, but Desmonda's Fish Tank is designed just for that purpose -- for authors to review each other's work. But they have limitations since they expect short stories (under 5,000 words and no more than 10,000). Since I write longer stories it isn't a great resource for me.
As to grammar, structure, composition, etc. not being important, I for one don't enjoy a story that is hard to follow. No matter how good the plot is, if it's difficult to read then it's not a good story (in my opinion). Saying that, I know we're not professional writers and don't expect the story to be error free. But I expect it to be readable to be considered good. When the author states that English isn't their primary language I take that into account. But confusing pronouns, run-on sentences, poor paragraphing, switching from past to present tense, changing names, forgetting that someone was still wearing his pants, etc. is very disturbing to me (disturbing in that it distracts from the story).
Switch Blayde
I have been posting stories on other sites, but authors in an ASSTR Yahoo group I belong to praise SOL readers for providing much more feedback. They credit the feedback form on SOL for that.
I posted my 1st story on SOL and did it as an external link. Why? For two reasons. 1) I want full control over the formatting and 2) frankly, I don't have a text version. By postng it as an external link I don't get the advantage of the SOL feedback form. The story got high ratings, but no feedback. And, like others have stated, I'm more interested in the feedback than the rating.
Lazeez, you explained that it had to do with the frames and you simply link to the external HTML. But you have the voting on the bottom of the screen in a different frame. Why can't you also put the feedback form in that frame?
Switch Blayde
Voting scores are most useful for readers, reviews most useful for authors (as the system now stands). As a reader, I never look at the reviews (don't have time), but I do use the scores -- plus codes, my knowledge of the author's past stories, and the story's summary -- as a guide to what to take a look at, given how many stories there are and my limited time.
The real judge of course is the story itself. If I start reading a 9.5 story and don't like it, I won't keep reading it just because other people rated it a 9.5.
Regarding the scale of the voting system...right now, there is a discerable difference (to me) between a story voted 9.50 and 8.50. If you change to a 5 point scale, all you'll be doing is making readers adjust to understanding the differences between a 4.25 and 4.75 (unless you round to the nearest .5 or something like that...which might be interesting, but you could do this even with the current 10-point scale). If some of the lower numbers on the 10-point scale never get used, so be it: that's simply an aknowledgement that just stringing together a few words into something resembling a sentence, for the free enjoyment of others, is more than most of us are willing to do.
One thing I'd suggest doing is dropping the 5% of high and low scores: that would help get rid of the votes from people who just shouldn't have been reading the story in the first place.
Maybe when a serial story is completed, the old votes can be erased and it starts fresh? Or maybe each new vote for the completed story could result in an older vote being removed (first in, first out)? I never vote for serials until they're done, but I'd like to be able to.
Larger and beyond that, how about a system which combines scoring and the smaller comments/reviews? I'm envisioning something like this at the bottom of each story:
I give this a [select list with a choice of 1-10] on the basis of [select list with choices like "Plot," "Character," "Writing/Grammar," and "Overall Appeal"]. Optional additional comments: [textarea].
In other words, each reader can give a story a vote of 1-10 in a *single category* of their choosing. That's not too much more work for a reader than the current voting system, but might help give the authors more focused feedback: both from the scores themselves, and from what aspect people choose to comment on. The textarea (the comments entered here would be compiled for the author) is there if people want to make note of things they liked or didn't like, suggested edits, etc.
By all means have a more robust system for longer reviews, too, as I'm sure that's very useful for authors. (Just authors, be sure you realize the potential trap inherent: that you might start writing to please the official reviewers, not the readers.) But I as a reader would like to maintain a way of quickly determining what I might like, enhancing this if possible, and also finding an effective yet quick way of letting the authors know that I appreciate their efforts (because I want them to keep writing, of course!).
Seriously, thanks to all -- authors, editors, site maintainers. It's great to see so much thought being put into this.
matt
As fallible as the system is I think what you have in place is pretty good. Yes the scores seem to be top heavy but there still is a spectrum of scores even though it may not be broad. I think with the scores, the download numbers, numbers for those rating the story, the stats on downloads and scores for recent and all time stories along with the story codes, the reader has a lot of information from which to pick out a story.
For example any story that rates over 9.5 (with over 200 votes) will for the most part be a great read and between 9.0 and 9-5 will be a good read. Below 9 is usually a fair read and under 8 is taking a chance. Your site is called Stories Online and not Erotic Writers online. We as authors and sometimes the readers often lose sight of what we are and we try to be more. Most readers are looking for good stories not good literature.
I often have readers encouraging me to write like Heinlen, James Michener or Clive Barker that is not who I am. I am an amateur flexing his muscles. I just don't have that many muscles.
This is a great site and I know you are always trying to improve it but I really don't think the rating system is all that bad. A narrower voting system (1-5) might help. I like the idea of reviews that pass through a moderator system.
That system should be very sensitive to the author. Your site has always been author sensitive but there are reviewers out there who are on an ego trip where the object is not to inform the readers but rather to impress the readers and such can be harmful to the goal of providing a venue where beginning authors can grow.
I also like the option of the writer to turn off the review system. I would probably have mine turned off. Many of my readers have tried to be authorities and usually that turns me off. I have had a few readers that actually have a handle on what I am doing and can tell me where I need to improve. I also have trouble with someone who has never written a lick telling me how to craft a story.
Thank you Lazeez for your outstanding contribution to amateur erotica. I salute you.
Consonant Smile
I would like to see chapter voting, but your vote replaced your previous vote. I know you have mentioned that this would be a horrible mess, but let's face it; there is nothing from stopping an author from coming out of the gates with a great story, and then tanking it for the last 10 chapters of a 15 chapter story, and it's sitting there with a 9.6 because everyone voted early.
Chucking the old scoring system wouldn't be fair to the authors that have their stories on the top of the heap.
I still say that if you give your paying members a chance to create a personal top 20/50/100 list, and then create rankings based on what stories averaged out the highest using a reverse weighting system, you would have a more definitive scoring system. And, the extra labor for the scoring is at least being done for those that are supporting you. Otherwise, you have people out there that have voted "10" several hundred times, and nobody else has any idea how good the scorer really thought that story was. Was it better than Story X? Worse? And, as a member changes their list, it has the potential to change the rankings (which is maybe updated once a week, to save crunching the numbers?)
You are really in a no-win situation, so I can understand why leaving things the way they are is the real way to go. Right now, a "Good" story gets a 10. There was a time when you read stories that were over a score of 8, and new you were reading something that was pretty good, and something in the high 8's or low 9's was outstanding. Today, if it isn't over 9.5, it's probably not very good. Somewhere along the way, folks started voting for gymnastics. They start at a 9.0, and score up.
Garylian
I want to post a review... what hoops would I have to endure in order to get to a page where it will ask me for a review (and/or vote)? Will it let me think I posted a review without it actually being submitted (like you said some scores were processed)?
It seems like you've put the beginnings of a new policy, but haven't completely specified it out. Others have made a few points as well, which I won't duplicate.
cPuppy
On the subject of voting and scores... I think a reduced set of scores would be much nicer (1 to 5). Another option would be to keep track of ver low and very high scores and statistically correcting the average with a bell curve or something, which might give a more correct average value, since it would reflect to a great extent, the "real" distribution of votes, and fight against the vote manipulation.
And finally, I'm wondering if other authors would be inclined to comment on this next suggestion: why not allow an author to turn on or off voting for his stories. Some of us might actually like to deny readers the ability to vote. I know I'd like to have that option available to me.
Further suggestions... Although voting and scores help readers to select a story, I think that increasing the length of story description allowed for stories might actually help as well. I know that I look at story codes, and description despite a score, before deciding on reading a story. The current limitation of 512 characters is very limiting in that aspect!
Another possibility (perhaps in addition to the above), is to allow a sort excerpt from an actual chapter (chosen by the author) to be available to readers as a preview which might allow the readers to have an idea about what the story and the writing quality might be like. 1 to 3 KB of excerpt should be able to do it. The story list could have a link to a separate Preview area to have a quick browse of the actual text from the story. Further, the preview area might be set free from any limitations (download limits and such as imposed on free and premier membership reader limits).
Reviews... Reviews are all right, and most beneficial, but I'm wondering what would be the effect of that on ongoing serials or very long stories that are yet to be completed. If a long story has already several chapters posted, I think a preview-review might be beneficial, especially if there's enoguh material for the paritcular long story/serial already posted to form some kind of idea --and keep in mind that a regular reader of the particular story might have already formed an opinion about what's been already published, and since he has been following its progress regularly, he's in a position to make suggestions or recommendations on whether its worth reading/following, and why, as well as be able to tell other readers what to expect, and such.
I've said "preview-review" because the above description of the activity ('suggest/recommend reading an ongoing long story/serial, why to read it, what the story seems to be about, and such') may not fit intot he classic review situation (scoring the writing style, grammar, plot, etc... and because the story is unfinished.
Anyways... let's see what you think
Dai
I just forgot to add one more possibility... Keeping scores hidden from readers, but available to authors. After all, the authors use scores and ratings as a means of feedback, where as readers use scores to indicate likes and dislikes (most of the time, btu there's still the blatant votings, as well as psychology of rating a story relative to another --i.e. 'comparative scoring'.
The hidden-form-the-view scores could be used 'behind the scenes' when a reader is searching for the highest rated stories, and get the stories listed in ascending/descending order (but still without being able to view the scores or ratings). I believe this definitely will be able to blatant score manipulation. What's more, without seeing the actual scores, a reader would not be able to say 'by how much' story A is scoring higher than story B, and this might make it difficult for a reader to score story A relative to story B, because he doesn't have the actual scores or rates of stories A and B reltive to each other, except that A is higher than B. Let's not forget that, most of us do 'comparative scoring' and our votes are influenced and driven by tha actual numeric value assigned to a story, and comparing it to what should be the numeric rating another story should have. If the rating is unknown, then one of the factors (the actual rating) won't come into play when doing a comparative scoring. This might, in fact, be able to reduce the psychological mechanisms that drive a reader to score story B relative to story A.
When an author looks at top rated stories or similar score related searches, he should be able to get all the stats, like how many votes, and average scores, etc. thus enabling him to measure his own success and failure.
Dai
But more important, from a reader's perspective, the vote I most often want to give on that site is 4.5. But I either have to insult the 5's by honoring a flawed story with that score, or insult the author by grouping him/her with stories I felt lacking in one or more aspects while still being a good story.
I like getting the feedback here but when there are low votes bringing down a story, I never hear from them. Praise is rarely as useful as criticism, though hearing the praise tells me I've succeeded in what I intended. If any change, I would like to see a requirement that the voter provide feedback when voting under a 5. Of course, most of them would probably just say, "It sucked."
Anyway, please keep the 1 to 10 for those of us who will often want to vote 4.5 on a 5-point scale. - RPSuch
I also agree with the idea of allowing reader reviews. You might want to look at the readers' picks page on the Erotic Mind Control Stories Archive (www.mcstories.com) for an idea about reviewers defining their likes/dislikes; that is, have an optional section for the reviewers to state their preferences, etc. That way, if a reader finds a reviewer with similar preferences, he can know to look at the stories reviewed by that reviewer to help in the story selection process.
Lazeez:
If I’m understanding correctly, you are going to stop allowing ratings for individual chapters, which is great. It was always wrong to allow a ‘story’ rating for a chapter. When it is finished, then it can be rated, like every other story. If it’s something to be used to guide the author on an unfinished story, maybe you can consider readers to ‘rate this to date’, or rate this chapter, which has as much meaning as the current method.
As to the proposed new review comments.
1. I think it a mistake not to include the stroke score. Granted there are those stories with no sex, but what’s the harm of putting a zero in the stroke value? The reader can see for themselves the description codes.
2. You ask: “Should grammar be as important as plot and character?”
Maybe it shouldn’t be ‘grammar’. Maybe it should simply be ‘readability’. Many, if not most, reades find reading a story with errors, very inconvenient. Whether the authors edit stories themselves or even use hundreds of editors shouldn’t matter. In fact, using multiple should indicate pride in one’s work.
For what it’s worth, there is one thing that I’ve not seen suggested, and which would probably be helpful to the readers. If when I post a story I could prioritize the codes, it could give the reader more guidance.
The Purvv
One debate is whether there ratings should be more centered. Is there any harm in having a large 1-10 voting system in which low scores are rarely used? If we suppose that people are interested in their point of view becoming the official score, and they can see the current score of a story and know they can only have a small effect on the score, then they would have an incentive to exaggerate. If someone reads a story that's rated at 8, but thinks it was worth a 9, he might put 10 so that the score goes up to 8.5.
The problem is that when you generally have high scores, the voting mechanism is biased to give more weight to those with low scores. If the story is rated at 8, but the voter thinks it was really worth 7, this voter has a much stronger ability to affect the score than the guy who thinks it was worth 9. To lower a high score, a voter can go way down and have more of an effect on the average. Thus, the average score doesn't reflect the "typical" voter.
A solution to this first problem is to use the median score instead of the mean score. Basically, it means you use the score of the middle voter. If 5 people vote: 10, 9, 9, 8, and 1, the vote you show is the 9. The result is that no individual has an incentive to exaggerate their score, because it doesn't change the median score. The final score really reflects the view of the middle voter instead of the middle opinion (in terms of intensity of feeling).
Another debate is whether votes should be on a 1-10 scale or a 1-5 scale. Would people vote exactly as they do now? In fact, a smaller scale gives voters just as much ability to bring the average score up or down, as you're using averages. For example, an average score of 9.9 on a 10 point scale can be brought down just as much as an average score of 4.9 by voting 0. Of course, this problem would be resolved using a median score. The only problem with a 1-5 scale is that it's indivisible. It gets rid of certain subtleties. For example, I typically only read into the second chapter of a story I like. By the time I decide on voting, I know I like the story and it's really a question of distinguishing it from other stories I've liked. The big question is whether to go with 9 or 10. With a 5 point voting system, we lose this subtlety. I would say let voters enter their own numbers (including decimals), to make more subtle distinctions. Otherwise, going to a 5 point system should make no difference.
My example does bring up another question, however. It's that of generally high scores. This means that the average score is not near the center of the scale (for you statisticians, that the scores aren't normally distributed). We might find, as was previously mentioned, that very few people waste their time writing bad stories, so we rarely give 1s or 2s. In that case, our voting system gives an unbiased estimate of people's preferences. The one problem is that it becomes harder to see the difference between two good stories just by looking at the score people gave, as most scores get bunched together. This problem would only be exacerbated by a smaller, 5 point voting scale. Another explanation for the concentration of high votes might be that people just feel really bad giving such a low score. This kind of problem, in turn, would be reduced by a smaller, 5 point voting scale. It might, however, be that most people just don't get far enough into a story to vote unless they like it. The problem is that the "average score" system becomes so ridiculously biased that it stops reflecting the average reader's preferences. This basically negates the whole point of voting - as most stories will have high scores given by the few people who like them.
As a solution to this last problem, JIMC proposed automatically assigning the average vote if a reader doesn't vote on a story. This basically gets rid of the "vote only when you like the story" bias. But if the average preference for a story in people's heads (of everyone who read it) isn't really that middle choice, this will bias the voting system. For example, if everyone thinks a story deserves 8 (of 10), but a few people are lazy and don't vote and thus automatically vote 5, the score will go down closer to 5. So, automatically giving a story the middle score can screw things up horribly.
As for the final question, how to aggregate votes in different categories, let me just refer you to a quick theorem. It's been mathematically proven: there is no way to do it that doesn't involve "strategic voting". For example, if you have 3 categories of votes, and you average the "final" vote, you'll have people increasing their votes in a category (beyond their true preference) to raise the "final" vote. For example, someone who loved the plot but didn't really like the writing quality might think the overall story was so good (because of the plot) that they'll raise their scores in writing quality to increase the "final score". In this case, we no longer have a good measure of writing quality. The only way to deal with this is to not combine the scores. The ideal would be to let users see all 3 scores each time (or let them configure which score they want to see). Any ranking would have to be based on the category the user chooses, lest you see more "strategic voting". I know I would go for plot over writing quality or sex appeal most days of the week. So, unless you want to render each category useless as a measure of what it's supposed to be, don't average the different categories.
I hope this analysis was helpful. Sorry about being so long-winded. I assure you that this stuff I talked about is tested and holds true. It's actually used to design voting systems and used by dictators and politicians to manipulate voting outcomes. If you're interested, it's a branch of math & economics called social choice theory.
Hope you're feeling better after your surgery!
I like what you plan to do with SOL regarding setting up a reviewing system. Way to go!
As I suggested last time, it might be useful to _readers_ if they could choose one or more reviewers who share their tastes/fetishes/kinks/etc, and have the site create a composite "editor" out of the reviewers they choose as their favorites.
As for going from 1-10 to 1-5 system. As a fellow programmer, I have a suggestion: Continue to keep the scores internally on a 1-10 scale, but have the votes be 1-5. Then double the vote internally (for example, 5 becomes 10, 4 becomes 8, etc.) before factoring it into the existing average on the DB. Viola! simplified voting while compatible with the existing votes, and we authors still see the _average_ vote on a 10-point granularity!
--
Hungry Guy
A few comments on what was said in it:
I also don't like the automatic vote. I sometimes do not finish a story and come back to it at a later time. I'd hate to have an automatic vote made before I finish the story.
As to why there may not be a lot of votes on the bottom of the scale, if I don't like a story I stop reading it. When I do that, I don't vote. So the stories that I do vote on I like therefore when I vote they get scores at the upper end.
Finally, if SOL has separate scores for such things as readibility, plot, stroke factor, etc., I would not combine them. Personally, I would skip a story that received a low readibility rating. To me that's very important. And to me readibility isn't just spelling and grammar.
One last thing (tongue-in-cheek). If you don't want to upset your female readers, you better call "stroke value" something like "stroke/diddle value" or something like that. LOL
Switch Blayde
As a reader, I will use the scores, the codes and the summary to judge whether I read a story or not. In the two or three years I've been coming to SOL, I've found only ONE story I wanted to read by going through the reviews that are posted. I asked for a review of one of my stories, simply to see what was said and for the feedback. If you put in the "review switch" I would probably put it on just to see what was said.
One quick thought, If you do this, I feel an author should have some way of responding to a review. Perhaps notify the author that a review of his or her story has been done and give him or her ten days to read the review and respond. Which would then be posted right alongside the review.
It doesn't say in your post exactly what the moderators would be doing. In order to judge a review ACCURATLY, wouldn't the moderater also need to read the story? Obviously they wouldn't allow "flames" through, something that just attacks the author but stating the author has "screwed this story up badly" Is a very plausible review for a very messed up, unreadable story, but a moderator who has not read the story and has no experience with the writer being reviewed, may well see it as a flame. While a moderator who has read the story, or read other stories by the same author, may agree and allow the review through.
A pull down menu simply asking "Why did you vote this way? Was It:" just beneath where readers SCORE the story sounds like a good idea to me. The choices would be...
1) The story was excellent all the way around.
2) Did the story touch you even though it was not grammatically perfect.
3) That the story was simply unreadable because of poor grammar or poor planning.
See Sausage Dog's comments for this post for a pretty good idea of what I mean.
Something with maybe ten or so options about WHY the reader voted the way he/she did for the story, good or bad. And then, of course, give the author access to these comments and the scores given. This would be quicker than making the reader write an e-mail but give the author some feedback. Keep the e-mail option but put this in for those who choose not to write or for those who may be afraid to write. If the author gets the same comments over and over, he or she should get the idea that the story is great or needs work and how much work it needs.
Basically, keep the system the way it is, just give the reader an anonymous quick comment system to tell the author why they voted the way they did.
-Jefferson
For all intents and purposes, we already have a 5-value scoring system--I would wager that the overwhelming majority of votes are in the 6-10 range.
If a story is worth my finishing it and getting to the voting box, it's unlikely to be under a 5. Changing it to 1-5 simply means that, a year from now, we can have a discussion about how we need to fix the voting system, because almost all of the stories have an average between 3 and 5.
This being the case, then I would suggest that IF an author does not wish to see or have reviews done, then that author should also not be permitted to have stories voted on.
Why?
Simple really - just as reviews can be good or bad, so can scores - you can't honestly expect people to vote on your stories if you are not prepared to allow them to comment on them as well (IOW review them).
Food for thought.
StoryO'Mine
Lowering the score range from 1 - 5:
I think this has a psychological effect... no more 10 as a magical number. Some 'readers' take that 10 as a holy grail, while others see it as a reward; part of this might be the whole mystique behind a '10'.
I think the 1 -5 system can seem more like a restaurant ratings with stars; this is a 5 star story but doesn't detract in any way from some other 5 star story.
At this time, I'm leaning that readers will be more likely to hit the 4 because the 'bottom' range is closer... I can measure from the bottom up rather than from the top down. Therefore I'm not starting with a 10 and subtracting; I'm starting with a zero and adding.
Let's be honest, the problem we're trying to resolve isn't inflation... it's differentiation.
We want readers to differentiate between stories they 'LOVE' that have some faults, and stories that they 'LOVE' and the writer obviously takes care with his 'craft'.
There's a difference between a storyteller and a writer; the top scores should go to someone that has found a way to meld these two elements in a package that captivates the audience.
In 1 - 5 system, I would feel more comfortable handing out 4's then I do in the present system handing a 9 over a 10.
----
Moderation --
I think we all know that Lazeez would pick responsible individuals who would take moderation seriously. Let's give people a chance... my suspicion is that a moderator doesn't have to read a story to understand whether a reviewer is being fair and just.
Some slams are allowable otherwise we'd need to moderate out the 'sycophants'.
A good review should be measured, precise, and have reasons behind the position it purports.
I trust Lazeez to pick people that will take their job of making SOL a better place seriously.
--
Automatic scores -- NIX.
What if I don't 'notice' that I can score a story again or if I don't want to score a serial until it's over.
You're voting by proxy... bad idea.
--
I'm not sure about the whole Author responding to a review.
I think the system will take care of itself... if I see someone slamming 'Rewind' because Frank went away or 'The Favor' because MWTB took the story too far, I would be VERY likely to respond with a review of my own.
Writers have to learn to let it go... once you 'publish' a story half of it belongs to the readers.
A writer never has to ask forgiveness for what they've written, and they should never defend it.
ElSol
Why else do we publish a story, but to have it read.
Its like a crap game, we (authors) toss the dice and hope for a natural seven. We can’t suddenly race around the table and shout ‘my dice’. We must take the good with the bad. (I shudder to think I said that.) We write (if we publish on SOL) to a varied audience, and I must emphasize, a very, very supportive group of readers. (Note the increasing downloads.)
I do not believe any author will be perfectly happy with a review (they’d be closing the lid if he was.) A reviewer could fail to emphasize a point the author felt was obvious, and that would cause gastric eruption in the author. What we, authors, must realize is the reviewer is not looking at the story in the same direction. If he sees a different picture, we (authors) must remember that we had the brushes and, although we saw the picture clearly, we didn’t paint it clearly
Let’s let the reviewers/moderators have freedom to make open comments. After all, don’t we want feedback? Let’s face it; the reviewer has put his/her name on the remark as freely as did the author and his/her reputation is important. Additionally, Laz has the power to, lets say ‘curtail’ his/her remarks.
JimC said: If the mini-reviews are mishandled, I can easily ignore them, much like I do scores on my own stories…
Anonymous at 6:52 said... Reducing the scoring options from 1-10 to 1-5 is counter-productive. Changing it to 1-5 simply means that, a year from now, we can have a discussion about how we need to fix the voting system, because almost all of the stories have an average between 3 and 5.
Sausage Dog
I see some comments objecting to one moderator killing a review and wanting more votes. That would be ok if you had a surplus of responsible moderators. Another way...may be to pass the "voted down" review to a second moderator to see if he concurs as a reality check. If he didn't concur then it would go to a third for a tie break. ?What's different about this than the two review method. hmmm..oth..if you had a scale for the moderators and a review was a "1" (5 scale) then he could just kill it. Oh well, the more I think about this the worse it gets. Just use the one vote and out method. All methods depend on the moderators anyway, so if someone posts a review that gets killed, they could always rewrite and resubmit.
As for external links ...(Switch Blayde)......
I like the plain formating of SOL. Its easy on the eyes and doesn't distract from the words (which is what we're here for isn't it?). Indeed, when there's an external link, and on the ASSTR site (and others) where the author doesn't have stories in text on the ftp part, I almost always just copy the story and read it in a text editor to get rid of the (mostly) poor formatting. I'm not, however, objecting to personal expression. There should be external links allowed to another web page if the author desires, but the story should nevertheless appear in the SOL site the way Lazeez does it with the choice to go look at the other site AT THE READERS OPTION.
Scoring,,,,,hmmmm,,,,:
1. You can't make everyone happy.
2. You've got a lot more experience running a site than the rest of us.
3. It gives me a headache even thinking about it.
4. You've already spents years considering the options.
5. I'd use the easiest system for you to implement.
After all, you're doing the work.
Lazeez, thanks for all the time and effort you excert.
Mistaknly
Switch Blayde
To highlight the differences, suppose you were to read a movie review that gave the movie a rating of 3 stars out of 5. You would think it was probably a "good" movie.
Now, suppose instead, it had a rating of 6 out of 10. It doesn't seem so good now, huh. You'd think it probably wasn't worth seeing.
Now, suppose instead it had a rating of 60%. You'd probably think it "failed" (i.e. sucked).
Hopefully you are seeing my point here.
As to what I would prefer for SOL, I'd rather let people rate it as 1 to 5 stars. Then show the rating as stars (not 4.32), though you might want to display half stars in the results, just to reduce the rounding effect.
As to how to convert old ratings to the new ones, I'd simply subtract 5 from the current scores. A current "10" story would become "5 stars", and a current "5" story (if there even is such a thing) would become 0 stars. A "9.5" story would become 4 and a half stars. The bottom 5 values in the current system are virtually unused anyway.
LordGarth
Mistaknly (or is it Switch Blayde?) writes correctly that you can't make everyone happy. He asserts you should just go with the easiest system to implement.
Though this sounds tempting, I tried to explain in my long-ass post on voting systems that the design of the voting system really does make a difference on whether the ratings will provide truthful data. It matters only in as far as authors' and readers' need a good measure of a story.
So, let's get down and dirty (no, not like that). LordGarth (12:06 PM) explains that there might be a real psychological effect. I can see that this might have an effect. A 3 of 5 might be viewed differently than a 6 of 10 or a 60%. Then again, Americans thinking about 60% might think of their own grading systems in school. In India or in France, 60% is considered decent on a test. So, it's also a matter of what people expect grades to be... what the average and the distribution tend to be.
The thing we shouldn't forget is that there are other effects, not just the psychological, of changing the scale: using only whole numbers, there is much less room to distinguish the pretty good stories from the good ones. Though we might hope readers will use more of the entire range with a smaller scale, this won't happen completely. If we're currently only seeing the use of the top third of the scale due to a psychological effect, then that effect will still occur on a smaller scale. If readers don't want to give low scores on a 1-10 scale, then on a 1-5 scale they'll still avoid giving a 1 or a 2. Heck, the only way you could really be sure to get rid of that effect would be a yes/no voting system. Similarly, if people rarely get far enough into stories they don't like to feel they've read enough to vote, we'll see a concentration of votes near the top. In either case, we lose the fine distinctions at the top. Going to a smaller scale seems like a tradeoff between getting people to use the bottom numbers and scores that give a precise idea of how a good story compares to others. The point is, of course, to maximize the distinction between stories.
So, if we (or Laz, as I'm not sure this is a democracy or anything :) ) really want to go to a 5 point scale, we should allow for decimals (meaning, allowing people to vote 4.5 instead of just 4 or 5).
The other aspect is whether to use mean or median scores. I'm sorry, but I've got to continue on this point. JiMC says "It's less than useful for readers (only one of nine possible scores possible) and it's a bitch to program correctly." In fact, median scores allow for a range of possible scores (actually just a bit more than the number of possible scores people can vote on). If there is an even number of votes, the median is the mean of the middle two. So, if the votes are 2, 8, 9, & 9, then the median is (8+9)/2 = 8.5. On the other hand, the mean would be 7 (not really reflecting the average person's vote). If you allow for more precise voting, were users can put in more digits like 8.5 or 9.1, you get equally precise medians. For example, if the scores were 9, 9, 9.2, 9.3, the median would be 9.1.
As for the programming aspect, the system would simply have to keep a list of how many of each score was received, which is easy enough. For example, it makes a table of the various possible scores and how many of each it received. It says “I’ve received one ‘2’, one ‘8’, and two ‘9’s.” I haven't programmed in a few years, though, so tell me if this would significantly increase space usage. I imagine that it would add about 20 bytes per story. It would be insanely more complicated, of course, if you let people vote using any number, with no limit to the number of decimals. But I imagine you'd only let people vote up to one decimal (like 4.7 instead of 4.7092).
I come back to this point about using the median, simply because it really can bias the score otherwise. I think the goal is to reflect the average person's sentiment, rather than the average intensity. Please correct me if I’m wrong. So, instead of seeing a score of 6.3 for a story that received 1, 9, & 9, you would see 9. In addition, you would see less manipulative voting (a situation where people don't vote their real feeling about the story but vote in a way to affect the final score the way they want – thus spiking scores or voting overly high). When you use the median, people know they can't individually change the score. So, they have no incentive to untruthful.
PS: A final thought. If you store the median, you could also allow authors to view the distribution of scores their stories got. This would let them know if a bunch of readers loved their story at the same time as a bunch hated their story, or whether most people had the same sort of reaction.
However, something you wrote has given me the impression that scores posted in this manner are not counted. Is that the case? If yes, is there a way for people like me to make our opinion count?
Thanks - Clioclioclio
I think 1 to 5 brings reader scores closer to the author conception of score.
For instance... I've just been told by a 'longtime' fan that he voted a story of mine a 4, and that he was being 'kind'.
It raised of an issue on the spread of points. If it had been a significantly earlier story in my
writing, I would have understand but a recent story?
It also strikes me as strange because two minutes
later another person praised it straight to high heaven.
Now, my problem is not the score itself, but that
score in comparison to scores this reader has given
my other stories.
Let's say the 4 is true...
Then when he gives another story of mine a 10... that
story is SIX points better than the one he gave a 4.
I don't buy that, not within the context of just 'my' writing, if you're comparing my story to Aftermath... fine, but not compared to my stories especially when the faults in 'earlier' stories scream out so loudly.
I read the story and I say 'Off-voice, but grammatically sound, structurally makes sense...it's
quality is not faulty'.
This leads me to the idea that the vote is about
preference, and that it is COMPLETELY about
preference.
Now, I'm biased by about +1 point towards stories
that really excite me or authors that consistently
produce good work.
But this strikes me as nearly a four point bias... whether it be against this one story or for my other stories.
I'd like to see the 'range' of bias reduced... I can see a 1 or 2 point bias for 'preference' but 4+ is too much.
Sincerely,
ElSol
> of effort needs to go into moderating reviews? A
> moderator will need to know if a review is accurate; > are you willing to see if it's fair? Once you read
> the reviewed story, are you willing to supress your
> own thoughts on the story and fairly moderate the
> review? Reviews can come any time of the day, 24
> hours a day, 7 days a week. How often will you
> moderate?
Uh no...
I do not need to read a story to moderate a review. I don't expect these people to check for the review's 'accuracy'. In that case, why don't the moderators do reviews and we drop kick this whole reader review thing out the window.
All I expect is to see that the reviews follow some legible 'form', have some sort of reasoned justifications, and do not degenerate into an attack on the author or the subject matter.
If I think a story rated a 9.74 sucks because the writer can't find good grammar if I re-wrote the story for him... that's my READER opinion and that's the point of reviews.
"Hey... good plot 9, character development 9, stroke 9, grammar 2...
A good story absolutely ruined by tragic grammar that makes the reader work harder than necessary"
Do you need to go 'read' the story to see that this fair?
I don't think so... at most, you read a chapter or two and say 'Oh... okay, I'm not that nitpicky but I could see how someone would gag on 4 'was' in one sentence'.
If this reviewer included a couple of 'textual' examples, I wouldn't even bother reading the story.
ElSol
> score? Stop all authors, (ones who have published
> stories on the site) from voting.
A. Authors are also readers.
B. Authors are more likely than regular readers to criticize grammar, sound structure, and the details that make a good story, a well-written one.
C. Removing Author voting does not reflect a 'purer, honest scoring', it reflects a purer, reader-only scoring and I bet that the scores would inflate even more.
The worst Author bias is that we don't score crap for the crap it is... because we don't finish it.
ElSol
As to the comment of reviewers bring in personal bias in to the review. Well such is life and everyone filters what they see read and hear threw their on likes and dislikes that is part of what shapes their opinion. As for that any individual review is just that one persons opinion.
If we are only looking for positive feedback then maybe we are doing the wrong thing, readers have a right to their view and we, as writers should listen to that view. It does not mean we have to make changes to stories because of isolated views, it does not mean we get offended because someone doesn’t like our work.
It has been said that there is no accounting for taste, and readers are and should be encouraged to avoid stories that might disgust or offend them as readers. Even with that said there are many readers that will simply read a story they know they will not like and then review it motives for this aside it is just unavoidable.
Speaking only for myself I look forward to on line review of stories. As to the number system it to is open to the predispositions of the readers and will remain to be so weather it is a system from 1 to 10 or a system from 1 to 5 with all the same pluses and minuses of the review system.
Scores also have something to do with what the stories are about, my roommate and I both write here and post stories written together under one pen name average score for that pen name is a 7.15, while his individual pen names have scores from a low of 5.35 to a high of 6.67. My individual pen names range from a low average score of 6.67 (on my first account and my first attempts at writing) to a high average of 8.21.
Now this is where it gets really interesting and strange. the thing is that his account with lowest average score has the highest count of favorites and book marked stories. It has has the highest number of individual downloads of either us. So scores are scores but what about all those downloads, book marks, and favorites? What does that say? It says the content might be quite disturbing but there are a large number of people that like them weather they vote or not.
Well that is my opinion colored by own predispositions, bias, predilections and all other variations of opinion you care to throw in.
Okay being new to this blog thing I am wondering if it is something all the authors should have? It could be a wonderful way to get feedback or input.
Alexandra the Great