It’s long been one of my theories that the user ratings on Amazon are useless as guides to my reading habits. It seems like virtually every non-political book that I look up has a rating between 4 and 5 (out of 5). It’s not hard to understand how this would happen; I expect that most people don’t take the time to read and review a book unless they enjoy it. Self-selection would weed out the most negative reviewers before they pick up a book. (I probably wouldn’t enjoy Those Who Trespass, but I’ll never know because I’m not going to read it.) Furthermore, people who love a book are probably more likely to choose to review it than people who were indifferent.
I’ve taken a completely unscientific look at Amazon ratings. I looked at six categories: General nonfiction, general fiction, history, politics, classics, and “bad” books (evil, discredited, or worthless books, not trashy fiction). Most of the books in general nonfiction, general fiction, history and classics are books that I’ve read. (I’m interested in whether Amazon ratings are useful for me, you see.) I’ve read a few of the politics books, and none of the “bad” books. (A list of books and their ratings are here.) I know that this isn’t a randomized sample and that it’s biased around my tastes. I have no intention of defending this study’s methodology, except to say that I didn’t pick books in an effort to get results I wanted. It’s just for fun.
Here’s what I found:
| Category | Average rating | % of books rated 4-5 out of 5 |
| General nonfiction | 4.5 | 95% |
| General fiction | 4.2 | 94% |
| History | 4.1 | 85% |
| Politics | 3.6 | 35% |
| Classics | 4.3 | 100% |
| Bad books | 3.8 | 57% |
A few notes of interest:
– Amazon will automatically generate an average customer rating even if only one person reviews a book. With a handful books that nearly everyone would agree are evil, they seem to have turned off the average customer rating function. Amazon doesn’t present an average customer rating for Mein Kampf, White Power, or International Jew (by Henry Ford), and they’re not bundled with another book. (If this is intentional, they’ve missed a paperback edition of Mein Kampf. And they’ve bundled it with Alice Sebold’s The Lovely Bones, which I doubt the author would appreciate.)
– I had a hell of a time finding any books with an average rating under 3 stars. The only ones that I found were DOW 36,000 (average rating 2.5) and Dutch: A Memoir of Ronald Reagan (average rating 2.5).
– Political books scored lower than any other category. Most books are reviewed by their fans, but political books draw a crowd of partisans to give one-star reviews of books they haven’t read (sample, from the reviews of Big Lies: “Please, this book is a load of garbage. If this is a representation of the Liberal Left, then we’re in big trouble. This is nothing but pure hatred. Don’t bother buying it, but if you must, buy it used.”)
As a result, the accumulated reviews of a political polemic make the average blog look like the Algonquin Round Table.
Out of the sample that I picked, liberal books seemed to score better than conservative books. All of the books that got 4 stars or more were left-wing books. I’m not going to assert that that means anything, for reasons just stated.
– Plenty of nonfiction and fiction books had 5 star averages, but none of the classics did.
– User ratings seem to be much more useful on products like consumer electronics and kitchen equipment. People who get crappy equipment don’t have to invest hours and hours to find out about it, and they’re happy to share.
{ 30 comments }
Walt Pohl 09.12.03 at 5:41 am
I think textbooks have more reliable ratings: people who like the book review it, as do people who had it for a class and hated it, so you get a broad spectrum of reviews. (And since they’re textbooks, they are spared the ideological treatment that political books get.)
jimBOB 09.12.03 at 6:45 am
I basically see the star ratings as nothing more (or less) than a handy way of seeing where a review is coming from without reading it, or else before I read it. If I want to see what holes someone had poked in something, I’ll scan for one star ratings. If I’d like an exhaustive description of something, I’ll gravitate toward the larger multi-stars.
marz 09.12.03 at 7:12 am
How about the fools that do 500 or 600 reviews in a span of 2 years. Who are these people? I figure the limit for somebody reading serious books in a careful fashion has got to be well under 100 a year. Whats interesting is to pull up a review of something like Machiavelli’s Prince and see how everyone reads it as some kind of business manual or handbook to self-liberation.
Dave 09.12.03 at 10:37 am
If you think, “well this book’s been out for a while and it was quite well reviewed but as that’s not a reliable guide, what’s the word from the general public?” Then yes, Amazon is pretty useless.
However, the reviews do add something like the sense of being part of a reading community, and they certainly give me a reason just to browse the site. I’ve never tried it, but I suspect that if I wanted to try an author with a large back catalogue, then Amazon would give a clue as to which book to start with.
And it is very good for textbooks. I’m a programmer, and where there is a wide selection of books on one subject, the reviews do help me make a choice. There are some bad reviews out there: I remember one technical book had a three star rating from four reviews: two were British and were one star; two were from the US and five stars. I’m in the UK: I didn’t buy it.
catsailor 09.12.03 at 11:03 am
One thing that always puzzled was: how does Amazon prevent authors from promoting their books by writing rave reviews under different aliases?
This problem doesn’t come up that much or at all with fiction or the classics obviously, but it seems to be a dangerous possibility for promoting software books – and since the market in this field is very competetive and there are many books about the same program or problem, there is even the possibility of trashing a competitor’s book about the same subject by writing a few negative reviews that bring down the score.
Does anybody have some info on this?
markus 09.12.03 at 11:25 am
@cat sailor
IIRC John Lott used to do this as Mary Rosh, and Amazon took his review down once it became aware.
The solution a publishing company I know prefers is to get friends of the author and company to write rave reviews and to link to the book.
Which is why I only read negative reviews. I think I can distinguish substantial critique from agenda critique and I know what I don’t like in books, so I can tell whether the critique is “valid” for me.
G. DeeDee 09.12.03 at 2:09 pm
You have to actually look at the reviews to get an accurate gage of what people are saying about the book. People are very generous with stars but the real opinion comes out when you look at the review.
Most people tend to give 4 stars if it didn’t bore them. Perhaps that is the general feeling of what makes a good book. ‘OK’ seems good enough.
I’ve seen books that had the same comments from 1 star reviews to even a few 4 star reviews. It seems that the only difference was that the higher the rating is given by those who liked it slightly better. Or perhaps they are just the kind of people who aren’t ready to out and out trash a book.
I find it hard to give 2 stars to a book I didn’t like, but many people do just that.
Nicholas Weininger 09.12.03 at 2:40 pm
It would be nice if they could somehow “moderate” the reviews, but any such moderation system would probably produce more bias than it removes.
It continually astounds and irritates me, though, how often people see the review opportunity as a platform for ranting about their particular hobbyhorses. This means, for example, that the reviews of books about Israeli/Palestinian history are always completely useless.
spacewaitress 09.12.03 at 3:10 pm
Amazon.com has a tendency to delete negative reviews. I once wrote a scathing review of a completely worthless self-help book. The review was rated as “helpful” by at least 80 people before Amazon.com’s editors took it down. Amazon has publishing guidelines that appear to be enforced much more stringently on negative reviews than on positive ones.
Xhenxhefil 09.12.03 at 3:10 pm
I’ve never tried it, but I suspect that if I wanted to try an author with a large back catalogue, then Amazon would give a clue as to which book to start with.
That definitely works. I’ve spent a little while at that site figuring out which books to read by Heinrich Boll and Hermann Hesse, with no prior knowledge of them except book titles, judging by seeing how they affected people and how people understood them.
I have determined NOT to read “Group Portrait With Lady” or “The Glass-Bead Game”, whereas those would probably have been my first choices without looking at the customer reviews inasmuch as they have the coolest titles.
Xhenxhefil 09.12.03 at 3:13 pm
It continually astounds and irritates me, though, how often people see the review opportunity as a platform for ranting about their particular hobbyhorses. This means, for example, that the reviews of books about Israeli/Palestinian history are always completely useless.
One of the reviews of Anne Fadiman’s “The Spirit Catches You and You Fall Down” basically uses it as an excuse to rant about the failed notion of cultural relativism. Another one is by someone who seems to have actually read the book, but thinks it’s a novel, and thus gives it 1 star out of 5 for including all kinds of extraneous background information that isn’t germane to the plot and could be found in any number of nonfiction books. 1 out of 22 people found that review useful.
Ted Barlow 09.12.03 at 3:17 pm
Xhenxhefil,
I think the person who found it useful handled my mortgage application.
taak 09.12.03 at 4:56 pm
I’ve ALWAYS found that the highest rated book on a given subject was also the best book on the subject.
Several times I’ve bought a 4-star book instead of a 5-star book because it just didn’t “seem right”, and was disappointed, then bought the 5-star book and was blown away.
If you don’t believe me, pick a topic, cull Amazon for reviews and their ratings on all the most popular books for that topic, then go to the library and skim all those books. (This works well if you have a good library with an online catalog and/or has wireless internet inside.) I’ve done this a couple of times and found the reviews and ratings to be right on the money.
The problem is, as Ted points out, the scale of the ratings gets stuck between 4 and 5 stars. Thus the possible ratings are only 4, 4.5 and 5 stars, which is, needless to say, quite imprecise, but it does not follow that they are totally useless.
taak 09.12.03 at 4:57 pm
I’m talking about non-political non-fiction, by the way.
raj 09.12.03 at 5:13 pm
(i) Averages are only moderately interesting. What was the standard deviation?
(ii) I suspect that most people who go to the effort to read a book all the way through and take the effort to comment on it, will probably tend to like the book, at least to some extent.
(iii) I have never seen “bad books” as a category, akin to “general fiction” or “general non-fiction.” What constitutes a “bad book”?
Ted Barlow 09.12.03 at 5:36 pm
i. I didn’t calculate SDs, but maybe I’ll update with them later. I suspect that they’ll be relatively small.
ii. I agree.
iii. I’ve linked to the books I rated. Here’s what I used as bad books:
11 Septembre 2001 : L’effroyable imposture (9/11: The Big Lie) 5
Inside the Third Reich (by Albert Speer) 4.5
White Power 4.5
The Satanic Bible 4
Imperialism: The Highest Stage of Capitalism (by V. Lenin) 4
Manson in his own words 4
9/11 by Noam Chomsky 4
The Protocols of the Learned Elders of Zion 4
Chariots of the Gods? 3.5
Coming of Age in Samoa 3.5
International Jew (by Henry Ford) 3.5
Before the Dawn: An Autobiography by Gerry Adams 3
So, You Wish to Learn All About Economics? By Lyndon Larouche 3
DOW 36,000 2.5
markus 09.12.03 at 6:22 pm
what’s wrong with the Manson book? i read that during my internship at a forensic psychiatric clinic. It was useless, but what makes it a bad book? Just curious.
Doug Turnbull 09.12.03 at 6:35 pm
To Xhenxhefil, re: Hesse and Boll.
Those are two of my favorite authors, and I’ve read everything published by either in English. I don’t know exactly what your tastes are, but I’d recommend starting out with Boll’s short stories, which are excellent and which span his whole career. (His career can roughly be divided into two sections–the darker works about the war and the immediate post war, and the more satirical works about the later prosperity.)
I think his best novel is Billiards at Half Past Nine, which also spans themes from both periods, although it’s also probably his most difficult. Group Portrait with Lady is a great novel but is also rather long and slightly odd in its narration, so I can see why it got some poor reviews.
On Hesse, I’d start with Demian, then Steppenwolf, then Peter Camenzind (an early novel that doesn’t get much attention but is very nice.) On this, I agree that the Glass Bead Game is probably the worst of his major novels.
sara 09.12.03 at 6:50 pm
the accumulated reviews of a political polemic make the average blog look like the Algonquin Round Table.
Then what do the reviews of the accumulated reviews of a political polemic look like?
I’m not going to concede Chomsky as a “bad” author, though I will concede Lenin because of the unhappy history of Soviet Communism.
I’d like to know whether the customer reviews system at Amazon is automatic, so that when a leftist book accumulates a huge number of negative reviews by right-wingers, this merely reflects raw input, not moderation by Amazon.
If right-wingers’ books (take Slander and Treason by Ann Coulter, very bad books) don’t accumulate massive numbers of negative reviews by leftists, is this due to raw input or moderation by Amazon?
Since I saw the “Deck of Weasels” advertised on howstuffworks.com, a site with excessive product placement for an “info” site, I have been allergic to political bias in websites that supposedly offer neutral info or services.
Tom Andersen 09.12.03 at 7:14 pm
I’m happy to say that my book got a 5-star rating. But then again only one person reviewed it.
B 09.12.03 at 7:58 pm
My impression is that the Amazon rating system works well for technical books. Somebody who bought a book to learn something from it and found the book to be useless, outdated, erroneous, etc. is likely to make the effort to give it a bad review. It’s actually quite useful when selecting among the many books on, say, Java Server Pages or XML.
Mark 09.12.03 at 8:16 pm
As an Amazon reviewer, I’d like to agree with what most people are saying. I review books I’ve read and music and DVD I enjoy. Guess what, if I’ve bought it and invested time into it, chances are that I think I’ll like it before I even start it. And usually I’m right. So almost all of my reviews are 4 or 5 star reviews.
I have found the best way to see if I really will enjoy something is see what else the person has reviewed. If they have tastes similar to mine, odds are I’d feel the same way about it they did. This is why someone who writes lots of reviews is more helpful in selling what they review then someone who only reviews one author.
Chris 09.12.03 at 8:18 pm
I agree that most people post positive reviews…I am an Amazon reviewer (for music) and I do myself. Basically, if I were a professional reviewer there are a ton of CDs I might be assigned that I think are really bad. But I am not a professional, and so I’d rather spend my time promoting music that I feel passionate about rather than tearing down crappy music….after all, what’s the point of giving Britanny Spears a bad review. You won’t change anybody’s mind. But a positive review of some out of the way jazz artist or avant-garde musician, might inspire the adventurous to buy more out of the way albums, thus helping the artist and the art in general.
Also, the average ratings on mainstream books, CDs and such are pretty useless, but less so I think on less frequently reviewed material.
Shai 09.13.03 at 12:00 am
I agree with g.deedee. you’ll have to read the comments to get a sense of anything. At least on Amazon.
The tomatometer rating at rottentomatoes works because theres a selection process that determines who will or will not be included as a reviewer. But if you don’t match the demographic of the average movie critic they might be a little off (I tend to agree with ebert 80% of the time — but never when he’s trying to channel a 14 year old).
In my experience imdb scores aren’t all that bad with exception to some subcultures such as “japanese animation” where hackneyed tripe will receive a quite decent score. For most movies with a few thousand ratings you can break down the score for specific demographics like sex and age to get a better idea.
But in general I think the comments are a more accurate way to judge; however, I don’t restrict myself to comments that are on amazon, I also do a google search for ‘”book title” review’, look for newspaper reviews, check the scholarly indexes if it’s a monograph, whatever.
re technical books, I wonder whether the ratings are more accurate than the average review as some have suggested. I notice many reviews by people who rate a book one star simply because they didn’t understand it. eg. they’re reading “physics for people who actually know calculus”, when they should be reading “physics for communication majors: what you should have learned in high school”. I see the same pattern in student evaluations for courses that don’t have rigorous prerequisites: 50% of the students will write that the professor is a bad lecturer and the course was quite hard when all of the material is elementary, if not utterly simplistic if you bother to pay attention and do the readings/problem sets/etc.
Bernie 09.13.03 at 12:53 am
understanding of the reviewing mechanics, it should make extrapolating useful information from the review possible irrelevant of the surface ratings.
bernie 09.13.03 at 12:55 am
The article is somewhat negative. With all the insight and understanding of the reviewing mechanics, it should make extrapolating useful information from the review possible irrelevant of the surface ratings.
bry 09.13.03 at 11:35 am
I have to say that your analysis of amazon’s helpfulness is not very useful either. The classification bad books pretty much invalidates the whole thing.
I don’t think I’d trust the classification system though, as I can’t think of many instances where I would rate. Only if I bought the book on amazon would I feel any reason to go back and rate it, I would however only be impelled to rate a book in extreme circumstances, if I thought it was brilliant, or if I thought it was absolute dreck. I would also be tempted to negatively rate works that I thought were being overrated. I suppose that if I ever rated a book on amazon I would be a corrective to overvaluation system, but as I never do I am not. sad really.
Jim Allan 09.13.03 at 6:19 pm
I haven’t done many reviews at amazon.com.
But today I received an email from the publisher of a version of a book I reviewed favorably a couple of years ago offering me a free copy if I would review the new version for amazon.com.
Is this perhaps now common practice in the industry? Are some prolific reviewers at amazon.con simply giving good reviews of every free book provided to them by a publisher?
jFrog 09.14.03 at 4:21 pm
One of the neat features of being an Amazon reviewer is that Amazon provides a chat group caller ‘Customer Reviews’ where you can buddy up and join a not very secret group who votes for each others reviews.
It seems that reviewers are more intent on raising their rank than in actually writing good reviews. Keep in mind when you read a review with a lot of positives that there are good odds the reviewer cheated.
Rebecca 09.15.03 at 9:31 pm
Hi,
As far as reviewing goes, it is much easier to see the negatives in life than the positives.
However, why would “most” reviewers bother reviewing anything below three stars? While some reviewers enjoy reviewing books in a negative fashion, there does seem to be a place for reviewers who enjoy promoting their favorite books, movies and CDs. Which is what many of the Amazon about-you-areas show.
The only time I really love reading negative reviews is when I absolutely disliked a movie. It is great for a laugh.
I’ve found that the reviews at Amazon are much more trustworthy unless they are an obvious attack on the author.
At times, I love buying a book just to see if it “really” is that bad. So, I figure negative reviews also help sell books in a big way.
Mostly, I find how you view an Amazon review has more to do with YOU personally than with how the reviews are written.
If you tend to be a more positive person, then you will agree more with the positive reviews. If you are more negative, you might tend to enjoy the negative reviews and relate more to those aspects.
I’m somewhere in the middle. I take both sides of the picture and try to form an opinion of the reviews in that manner.
If you have been reading Amazon reviews for a couple of years, you learn to weed out the unhelpful ones and take the helpful ones more seriously.
So far, I’ve found the reviews at Amazon to be very helpful and may Amazon.com rule the web forever!
Power to the people ;)
~TheRebeccaReview.com
P.S. I think your reasons for why conservative books have a lower average (I can’t verify that due to my own time constraints), could be because liberals tend to be more active in reviewing books they disagree with than conservatives? In my limited
years online, I’ve met far more liberals online than conservatives. All my “liberal” friends are also great at writing. For some reason they tend to be more creative. At least the friends I’ve met online excel in those areas. So maybe that helps answer your question.
P.S. II My mother wants me to read: “Unfounded Loyalty” by Wayne Perryman.
P.S. III You will find that if you can find books that you love and you can review them in a positive manner, authors will appreciate the fact that you found their “gem.” I love finding books I absolutely can give 5 stars to because those are the books worthy of extensive and well-thought out reviews. Anything below a three star rating is going to be a book I’m not really interested in to begin with.
Comments on this entry are closed.