Boardgames To-Go

How accurate is Fairplay's Essen scouting?

Every year the folks from the German game magazine Fairplay set up a booth at Essen. Besides pitching subscriptions, they collect informal rankings for the new games from Essen attendees. These are shared on the web after the event (during?). The ranking system isn't the familiar 10-point scale we commonly use in the English-speaking world. Instead, it's based on a school grade system, only rather than A-F or a 4-point scale it's a 6-point scale, with 6 being worst.

It's simple enough--just look for the games scoring between 1 and 2, but today I tried inverting & transforming 2003's Fairplay results to a 10-point scale. Once I did that, I could then compare their average Essen scouting scores with the game ratings on Boardgamegeek. (I used the raw averages from BGG rather than the Bayesian ones. I don't think it really matters.)

The point of all this is to see how good of an indicator the Essen scounting reports are. Do they do a good job finding the best games in just a few frantic days? Or are there good ones that slip through? Worse, are there games with good buzz that later fall flat?

I look at this comparative data and come away with the conclusion that the Essen scouting is pretty good, but not great. Put another way, it's good overall, but with some notable exceptions. Last year, the Essen scouting kind of missed Santiago. You can't really say the same about Carcassonne:The Castle--there was lots of attention paid to it--but the later opinion of that game was notably higher than the early buzz at Essen. Over-expectations, perhaps? Attika is somewhere in between. It didn't have the attention of The Castle, but was much more visible than Santiago. It's just that the opinion of the game rose even higher after the event. (It was still a Top Ten game in the Essen scouting, accurate enough.)

Yinsh is special in that it's a niche game. Abstract fans love it. Not just abstract fans, but it's reasonable to expect a game like that will suffer a bit in broad polling. I'm less sure what to make of Einfach Tierisch, the German remake of Knizia's High Society. Why didn't it get higher marks in the scouting report? Maybe just because it IS a remake, and attendees of the fair are more interested in new titles. Is Warcraft a niche game, too, or just one that Americans/Brits with more fantasy wargame background can appreciate.

Perhaps more significant are the handful of titles the Essen scouts really liked, but whose buzz cooled off pretty quickly. Last year, that was Maya, Ludoviel, and Logistico. How can you guard against that when looking at this year's scouting report? I'm not sure.

Picknick Panik is what happens when the scouting report includes only a limited number of votes from a skewed sample of fans. Not that many of these ratings are statistically valid--this is just an extreme case.

I don't think too much should be made of discrepancies or any other analysis of the low-end games. Yeah, BSZZZ! shows lower in BGG ratings, while King Arthur shows a bit higher. We might be dealing with degrees of mediocrity, though, so it's not worth much attention.

Looking forward to 2004's scouting report, then, what should we watch out for:

1. Niche products. The people that already like these will like them, but the broader polling results will make their scouting scores a bit lower.

2. Low number of votes. Fairplay tends to screen out games receiving less than 10 votes in their online posting, but I think you need 25-50 votes before you can be very confident with the results. Many of the games receiving few votes are coming from small/unknown publishers where your risk is higher already.

3. Kids games. I already filtered out the games intended just for children in the results below, but Fairplay doesn't.

4. Over-expectations. If there's a game with lots of anticipation (Carc:The Castle, King Arthur), then if it turns out to be merely good, not great, the ratings will be lower due to disappointment. But it still may be a fine purchase. (Perhaps the Tresham & Wallace games could apply here.)

5. Remakes. Looks like the Essen attendees aren't wild about remakes, maybe because they're sometimes surprises. But there's nothing wrong with a good remake. (Watch out for Razzia/Ra.)

Even with all of that, there will be inaccuracies. What else can you expect? Last year, Maya was rated by Essen attendees significantly higher than Santiago, and now that looks like a bad call. That'll probably happen again, and the only way to avoid it is to wait much longer before you spend your money. Best of all, of course, would be to try before you buy.

Here's a graph showing my 10-point converted Essen ratings versus what's on Boardgamegeek. Following the graph is the full table showing all of the titles and their two scores.

Game	BGG	Fairplay
Princes of the Renaissance	7.9	8.5
San Juan	7.7	8.4
Yinsh	8.1	8.4
Maya	6.6	8.4
Fresh Fish	6.9	8.2
Railroad Dice	6.8	8.0
Attika	7.6	7.8
Finstere Flure	7.2	7.8
Pingvinas	6.9	7.6
Ludoviel	5.8	7.6
Zauberschwert & Drachenei	6.2	7.6
Feurio	6.6	7.5
Iglu Pop	7	7.4
O Zoo le mio	7.1	7.4
Subulata	7.1	7.4
Logistico	6.5	7.4
Tom Tube	7	7.2
Age of Mythology	7.1	7.2
Bridges of Shangri-La	7.2	7.2
Global Powers	6.9	7.1
Die Fugger	6.8	7.0
King Me!	6.5	7.0
Santiago	7.6	7.0
Dracula	6.6	6.9
Die sieben Siegel	7.1	6.9
Flaschenteufel	7.1	6.8
Cronberg	6.4	6.8
Schwarzarbeit	6.1	6.7
Fantasy Pub	6.7	6.7
Picknick Panik	4.2	6.6
Industria	7.2	6.5
Top Speed	7	6.4
Kogge	7.1	6.4
Carcassonne - The Castle	7.7	6.4
Lawless	6.2	6.4
Mamba	7.2	6.3
InterUrban	6.8	6.3
Anno 1503	6.6	6.3
Lucky Loop	5.5	6.2
Borgia	6	6.2
Bonobo Beach	6.4	6.1
Ruhe in Frieden	5	6.1
Maka Bana	6.7	6.0
GRO: Battle for the Petri Dish	5.8	5.9
Sunda to Sahul	6.5	5.9
Alexandros	6.4	5.9
Intrige	6.9	5.9
Nuggets	6.2	5.9
Einfach Tierisch	7.4	5.8
Die wilden Fußballkerle	6.3	5.6
BSZZZZ!	4.2	5.6
Yellowstone Park	5.2	5.4
Fluxx	5.6	5.4
The Kookaburra Game	5.2	5.2
Hoppla Lama	5.4	5.0
Clue - The Card Game	6.1	5.0
Turbo	4.7	4.9
Warcraft	6.8	4.8
Tomb Raider	5.8	4.6
Coyote	6.8	4.6
King Arthur	4.8	4.4
Crocodile Pool Party	5.2	4.3
Fliegen	5.9	4.0
Secret of the Tombs	5.1	3.3
Wo ist Jack the Ripper?	5	3.1

Posted by MarkJohnson at October 21, 2004 03:29 PM