PDA

View Full Version : Review system creation



Darklord
16-04-2010, 12:40 PM
OK folks, think I have a free couple of hours coming up in 4 years time so wnated to get the review system up and running. I know this has been discussed briefly elsewhere but I wanted to get everyones opinion in one place.
The system will be split into 2: 'official reviews' and member reviews. The official reviews will be done by an elite squad trained in the art of tea making (possibly) and will follow a set scoring pattern. Any product/service that achieves a high enough score will be awarded a wamp recommended medla (or something like that) Official reviews will be done on any product offered up by a willing company.
Member reviews can be done by anyone on any product. they may or may not follow the set scotring system (upto the reviewer) but wont receive any wamp recommended items if they achieve a certain score.
So what I want is suggestions on scoring. What aspects should we specifically rate with a score?
Cast Quality will be one
Value for Money, Originality, maybe considered, we only need 3 I would say as other points we would cover wihtin the review (such as ease of purchase, skill level required etc)
How would we score these? each out of 5 or 10 or as a percantage?
There will be an overall score, not necessarily the total of the sub scores but a general indicator of the product. again should this be scored out of 5,10 or a percentage?
Finally as mentioned there will be a wamp recommmended medal awarded for high scorers. Do you think we should go for just one or maybe have 3 levels such as gold, silver and bronze? If so what score would be needed to get these?

Of course any other suggestions, opinions or recommendations you have I would love to hear.

Once all this is in place we shall begin audtions for reveiwers (have you got the Wamp Facotr?!!) The plan is to have a panel of 3 - 5 official reviewers, spread worldwide if possible ( so companies dont need to ship overseas so much)
And I shall send out a press release to companies.

Iacton
16-04-2010, 01:42 PM
Cast quality
Ease of assembly - sliding scale here might be useful - novice, amateur, intermediate, experienced, expert
Value is highly important really.. Makes a difference if you are wavering over buying 1 model for a tenner or 2 or 3 at 3 or 4 each...

If you have a percentage score overall with bronze, silver, gold is a good touch - Gold over 95%, Silver over 85%, Bronze over 75%. Quite high for a gold, but would make it more of something to aim at :wink:

Darklord
16-04-2010, 01:49 PM
yeah thats the sort of scoring Im favouring. Shane has done some mock ups for me, inclduing a 'wamp rejected' one - not sure whether we should use that though?
Value is pretty important, a 30 mini isnt necessarily expensie compared to a 5 if its a 150mm thing with flashing lights that doubles a s a beer fridge

NathanS
16-04-2010, 01:53 PM
I would like it to be each of the categories be a possibility of 5 pts with a total of 15 pts. It would make it easier and I would love to get in on this.

Iacton
16-04-2010, 01:57 PM
Wamp rejected is ok. We aren't an official body, but opinion matters, and if it's warranted, then fair enough. TBH it's unlikely to happen unless you bear a personal grudge against the model/range, or if you chopped the end off your finger removing mouldlines...

Ulfgrimr
16-04-2010, 02:03 PM
A percentage system might give more flexibility than a points system limited to 5 points and is easily understood by most. I also like the idea of gold, silver and bronze (as Iacton said) and might give companies an incentive to achieve better next time. Would a 'rejected' label discourage manufacturers from submitting for reviews in the future?

The most important things for me are cast quality, dynamism of the pose/originality and value for money in that order.

Hope that helps.

Iacton
16-04-2010, 02:10 PM
I know (from feedback whilst in the mixer) that ease of assembly and cast quality are the things that make and break models. Really simple, easy to clean models will always sell well - look at Hasslefree for example, and even GW to an extent - companies that look after this will always have popular models.

TaleSpinner
16-04-2010, 02:10 PM
I don't have a lot of input, but one thing I'd like to see along with any review is ease/how to aquire said product in various parts of the world. Being a UK based site, products that you guys find and obtain readily, we in the US and probably Australia too have difficulties getting without paying double the price in shipping. I'm sure that the reverse is true products made in the US and Australia.

2 cents.

Andy

Hinton
16-04-2010, 02:15 PM
As I've mentioned before, I really like the idea of a review system and the scoring you've suggested (percentages) seems to be a good one. Definitely like the "WaMP Seal of Approval" idea.

To be honest, I'm a bit against the "WaMP Rejected" label. If something scores low enough, with a well-done review people will understand the problems with the mini and know that it's not good.

Darklord
16-04-2010, 02:27 PM
what puts me off the wamp rejected is as Bill has said it may put companies off associating with us in the future.
I like the percentage scoring as it gives more flexibility. look at cmon for example - an 8.0 is not the same as an 8.9 but on a out of 10 sytem both would be 8/10. I think out of ten would work for the sub scores then percentage for the total?
@Talespinner - i think this sites more canadian than anything!! ease of purchase would be mentioned (if not scored) last year we had a couple of sponsored contest that lost entries due to them bein hard to find on one side of the pond or the other.


TBH it's unlikely to happen unless you bear a personal grudge against the model/range

someone in mind?!!

Iacton
16-04-2010, 02:31 PM
http://i247.photobucket.com/albums/gg148/Iacton/Smilies/angel2_smily.gif absolutely, er, not :tongue:

shanerozzell
16-04-2010, 02:56 PM
Firstly most of the reactions I have read assume that it'll only be miniatures being reviewed. I hope this is not the case, retailers, paint's, tools, brushes, display bases, etc. can't all be reviewed in the same way. A points system can work but if we (by we, I mean DL:biggrin:) decide to use the gold, silver, bronze thing the points will have to be devisible by three for eg. 1-5=bronze; 6-10=silver; 10-15=gold.

While I agree that the reject thing shouldn't be used (on the whole) for specific products it comes into play with dodgy retailers and the like. Maybe used as a kinda of smilie where, on a thread, people use it to indicate what service they received, thus fore-warning other WAMPERS not to use them.

Darklord
16-04-2010, 03:06 PM
The plan is to review any hobby related products including all those mentioned and even maybe things like light tents or carry cases, books, dvd etc.
Not sure I understand the divisible by 3 bit Shane?

The reject as a smile is a great idea!

shanerozzell
16-04-2010, 03:27 PM
I choose three simply for easy of use. The divisible by 3 thingy is to give a even spread to the points given to the reviewed item. 5 points to bronze, 5 to silver etc. I think by doing it this way the scoring system will maintain a certain continuity

For example lets say we are reviewing a miniature.
The things people (IMO) want to know are;

Cast quality (Flash lines etc.)
Sculpt quality (Design, pose, etc)
Assembly (mainly for multipart mini's)
Game integration (can it be used in any game system or is it spcific)
If each of these points is given a score of 1 to 5 and in this e.g. we'll be totally fair and give them all a three. That means the mini has 12 points so according to what I posted earlier it'd get a silver star.

If on the other hand the reviewer just gave a star rating for each of these points the mini would have a total of 4 stars, each different. In this case the product company would be less likely to use the review system in its own marketing because the 4 different stars, each a different colour, are less visually appealing than just the one.

ScottRadom
16-04-2010, 03:39 PM
Man I got tons to say if I did some reviews. Good and bad. But I'd rather focus on the good.

Darklord
16-04-2010, 04:09 PM
I see what your saying Shane and it makes sense. I do think a percentage for overall is the best option, but how to tie that in with the sub scores like you say.

maybe make the overall and the sub scores out of 10 but with a decimal so a mini could get 9.2 overall - this would create unity while still giving as much flexibility as the percentage system

NeatPete
16-04-2010, 04:17 PM
I think ?/10 or a persentage gives you more options than ?/5. sometimes you want to go between 3 or 4 but you cant so you have to round one way or the other, if its out of 10 you can easilly say 7!

I would think the categories would have to be pretty vague/transfereable accross different product, Mini's, paints, brushes, w/e.

I would love to be an official WAMP reviewer.

Darklord
16-04-2010, 04:21 PM
I would love to be an official WAMP reviewer.

Once the system is set up anyone interested will get the chance to audition

ScottRadom
16-04-2010, 04:36 PM
audition! Sweet! I got this thing where I can tuck my junk between my legs and then... well... nevermind.

shanerozzell
16-04-2010, 04:43 PM
LOL.... Scott you're a very scary man!:eek:

Darklord
16-04-2010, 04:44 PM
well im putting the feelers out. Shane - ill give you a shout later about the medals mate

Iacton
16-04-2010, 04:55 PM
Man I got tons to say if I did some reviews. Good and bad. But I'd rather focus on the good.

Well, it's nice to be nice, but in a review it's better to be honest...

Vern
16-04-2010, 04:57 PM
I choose three simply for easy of use. The divisible by 3 thingy is to give a even spread to the points given to the reviewed item. 5 points to bronze, 5 to silver etc. I think by doing it this way the scoring system will maintain a certain continuity

For example lets say we are reviewing a miniature.
The things people (IMO) want to know are;

Cast quality (Flash lines etc.)
Sculpt quality (Design, pose, etc)
Assembly (mainly for multipart mini's)
Game integration (can it be used in any game system or is it spcific)
If each of these points is given a score of 1 to 5 and in this e.g. we'll be totally fair and give them all a three. That means the mini has 12 points so according to what I posted earlier it'd get a silver star.

If on the other hand the reviewer just gave a star rating for each of these points the mini would have a total of 4 stars, each different. In this case the product company would be less likely to use the review system in its own marketing because the 4 different stars, each a different colour, are less visually appealing than just the one.

Yeah, kinda agree with this - think that each medium (ie figure or paint or dvd etc) should have fairly rigid review criteria too

Vern
16-04-2010, 04:59 PM
Well, it's nice to be nice, but in a review it's better to be honest...

Totally agree - any 'official' award would loose creditability otherwise

RogerB
16-04-2010, 05:51 PM
In the case of value for money it should be taken into account if it's for a game and how many you would need. Typically historical stuff for games has been cheaper as well.

Darklord
16-04-2010, 06:19 PM
yeah the value for money has to fit the situation of the product - a regiment in 6mm has different considerations to that of a dipslay only single 90mm piece

exilesjjb
16-04-2010, 06:28 PM
I think info on the company would be great eg web site, supplier in europe, USA. then I would agree /10 is better, like the idea of an overall % which leads to a gold/silver/bronze. maybe they could lead to annual awards , best figure, best paint, best....

Darklord
16-04-2010, 06:31 PM
yeah we'll have some company info for eachone. IM personally thinking 3 x sub scores out of 10 and overall also out of 10 (the overall having the opportunity to be 9.2 or 5.8 score)

Hinton
17-04-2010, 05:24 AM
Thought about this some more and I think I'd prefer a review system that doesn't use any numbers or percentages.

Giving things a rating of Poor, Good, Great and Outstanding (or some such) allows for a bit more interpretation for the reviewer. Assigning a number to something (say, x/10) can create unwanted - or unwarranted - backlash. Just look at the scoring system on CMoN; people gripe about that constantly. And some even argue decimals, wondering why their mini is scoring a low 8 when they think it should be a high 8 or something.

While there should be consistency in reviewing, it'll be difficult since there isn't consistency in reviewers. What one person hates another may absolutely love. Just look at movie critics; most of them can't agree on anything. Siskel and Ebert used to go round and round about movies because one loved it and the other hated it.

As usual, I'm probably over-thinking this - and no matter what system is used, unfortunately someone's not going to like it.

shanerozzell
17-04-2010, 11:26 AM
Giving things a rating of Poor, Good, Great and Outstanding (or some such) allows for a bit more interpretation for the reviewer. Assigning a number to something (say, x/10) can create unwanted - or unwarranted - backlash. Just look at the scoring system on CMoN; people gripe about that constantly. And some even argue decimals, wondering why their mini is scoring a low 8 when they think it should be a high 8 or something.

Totally agree with you there mate but I think the main problem with CMON is just one overall score. Here there are scores for different criterea as well as the reviewers opinion so that should make the final result fairer. I know DL is going to "audition" four or five permanant reviewers but I posted in another thread about a guest reviewer. Has anybody got any thoughts on that?
The idea was that for each item being reviewed the main reviewers would get their mits on it first then a randomly choosen "Guest WAMP Reviewer" would have a go at it. This would keep WAMPERS more engaged with the whole review process and give them a chance at e new product whatever their skill level..... just a thought.

Iacton
17-04-2010, 11:55 AM
A non-score based rating sounds good to me.
As for guest reviews, I would have thought that anyone who wanted to review an item could do so, in an official form, providing it was submitted to DL, and/or the reviewing panel (if that is what is created), for ratification/editing. (Please read that as editing and not censoring!) That way we can avoid a duplication of reviews, mainly...

Reading Hinton's post, this is where I think that scoring on dynamism/style etc, would be a bad thing. This shuffles into the realm of artistic critique, when really we want to know whether it's nicely made, goes together easily, comes with spares, (base, magic beans), value for money. The actual artsy aspect should then be down to the punter.
I am sure that comment on the aesthetics is perfectly reasonable within the review - style, dynamism, proportions, scaling etc; but it should not be "scored" by the reviewer, just an opinion.

Hinton
17-04-2010, 02:38 PM
I've actually been working on a review for the Twilight Knight Pin-up Model from Kingdom Death. Maybe I'll be able to put it together and get it posted today (I guarantee nothing, though), just to kind of show my take on things.

Hinton
19-04-2010, 01:04 AM
Ok, did a review of the Twilight Knight Pin-up Model. And yeah, I did it as a video review, just for kicks. Also tried something a little different.


http://www.youtube.com/watch?v=lg-RQG4Gt7o

NathanS
19-04-2010, 02:30 AM
When can we check it out Hinton I can not wait to see it.

edit: never mind I found it I will check it out when I get home.

Hinton
19-04-2010, 02:33 AM
Should be able to see right now.

NathanS
19-04-2010, 03:57 AM
That was the most spectacular and entertaining review of a mini I have ever seen. I give this review the NS seal of approval rating Outstanding.

breadhead.
19-04-2010, 08:57 AM
I really enjoyed that, the idea of a video review seemed a bit weird to me but you can actually pull it off!

Ulfgrimr
19-04-2010, 11:01 AM
Great stuff Hinton. All manufacturers should be made to include a Hinton video review, complete with his alter ego.

shanerozzell
19-04-2010, 12:15 PM
I'm giving the Hinton review a gold seal of approval. Man, you should be on the telly!

Iacton
19-04-2010, 12:35 PM
Hmm... It would appear that you have a habit of annoying yourself... and disapproving of your own actions...
I shall consult my Freud texts and get back to you.

(Nice review by the way)

Hinton
19-04-2010, 01:22 PM
Thanks, everyone. It was a lot of fun doing that and it's great to hear that people like how it turned out.

I was going to do just a text review, but this idea popped into my head, so I thought I'd give it a shot.

Wonder what Poots thinks about it...

NeatPete
19-04-2010, 06:01 PM
Can't wait to get home and watch this with sound!

NeatPete
20-04-2010, 03:47 AM
Ha! Well done! Another dope video from Hinton! I was chuckling, and most other people would have annoyed me, but I enjoyed it all.
Always love your vids. I can see why you're such a good actor as well. Go Cubs!

Hinton
20-04-2010, 03:52 AM
Thanks, Pete. Glad you enjoyed it.

The response to this has been kind of overwhelming. So many people like it and it's racking up more views than any of my other videos (even picked up a couple of subscribers at YouTube because of it) that I'm seriously tempted to continue to do videos with my "alter-ego". Maybe make "him" the star of the show, so to speak.

And yes, go Cubs!

ScottRadom
20-04-2010, 04:43 AM
Nice stuff Hinton!

Jabberwocky
21-04-2010, 12:30 AM
Fantastic review. Laughed quite a few times...I look forward to your next review!