Jeff Atwood, one of the founders of the StackExchange network, chose to mark his last (day-to-day) day at the company with a post disparaging identification questions. As Jeff pointed out in his post, the [tag]story-identification[/tag] tag is the second most common on our site right now, so I’d like to point out why our community feels that these are a valuable part of the site, where we disagree with Jeff’s viewpoint, and hopefully give you a bit better understanding of how decisions about what’s on-topic are made (and then re-made, and re-made, and …).
The community consensus gets hashed out on “meta” – there have been quite a few discussions about story identification questions (starting way back at the launch of the site, up to and including during the recent moderation elections), so you should head over there if you’d like to make your opinion heard, or read in more depth (good places to start are here and here) – what’s below is heavily based on the content from there (I’ve borrowed some sentences verbatim).
This is my personal opinion as a user of the site, not an official statement.
I disagree with Jeff’s post right from the outset (all bold quotations are from Jeff’s post):
Our engine is great at these kinds of questions
Actually, while the StackExchange engine is great at some types of question that aren’t generally suited to the sites (e.g. polls), I find that there are many problems with using it for story identification, for example:
- A great fit is a question that can have a few (perhaps two or three) really great answers that all get highly voted, leaving readers of the question able to learn from each of them. A story identification question needs exactly one correct answer.
- A great fit means that it’s easy to know whether to vote up (or down) a question/answer. Although it can seem otherwise, story identification questions on scifi.stackexchange.com generally score a bit lower than other questions. Voting on the question isn’t too hard; voting on answers, however, is hard. If you’re familiar with an answer, then you can judge how well it suits the description in the question, but you don’t actually know whether it’s the right match or not. What happens if there’s an accepted answer – do you just vote for it (it’s obviously correct)? Do you vote other answers down?
- A great fit are questions where wrong answers get voted down or deleted. With story identification, if it’s very close but not exactly right, these tend to get votes and stay around, even after an answer is accepted.
- A great fit is when you know that your answer is right because it exactly matches the problem in the question. However, with story identification, sometimes the description in the question isn’t quite right, so you should add your answer anyway.
However, even though these aren’t a great fit with the StackExchange engine, they are a valuable part of our site, and they’re better than the current best the Internet has to offer: the Usenet group
rec.arts.sf.written is well-known as a go-to place for story identification – they work ok on a Usenet group, but not so great: sometimes the thread diverges and the answer is buried amid hundreds of off-topic posts. Consider how similar this problem is to the state of programming questions before StackOverflow came along.
A half-remembered description of something you vaguely recall is not what I’d call a practical, answerable question.
The “practicality” question (“solving problems“, or are we a “how to” site) has come up a few times on our meta. It’s true that if you’re considering “practical” to mean “I need this for work” (and for StackOverflow, the first, most important, and most successful StackExchange site, that’s mostly the case), then (other than in rare cases) we’re not doing that.
However, understanding a development in a fantasy book I’m reading really is a “practical, answerable question based on an actual problem” that I face. Wanting to know what order I should show my son Star Wars to maximise his enjoyment is also a practical (“of or concerned with the actual doing or use of something”), answerable question based on an actual problem that I face. Being able to get back to sleep because I can finally stop trying to figure out what that book I read 30 years ago that I really, really, want to find again is also a practical, answerable question based on an actual problem that I face.
Story identification questions are clearly practical: they are concern with “doing” (finding the story) and using something (re-consuming the story). If the question is too vague, too light on details, could match vast numbers of works, then yes, it’s not answerable. But that’s ok! There are lots of questions outside story identification that are like this too – we have a process for these: we vote down the question, commenting on it what it needs to be improved. And if the question is really terrible (“I read a book and it maybe had a green cover”, it can be closed, on merits, not because of its nature. “Unanswerable” isn’t a problem that’s limited to story identification questions.
2. Guessing game questions don’t help others
Actually, these are extremely useful to others. The same “vague, broad, half-remembered descriptions” (or at least an overlap) will be used by others searching the web for this great story that they don’t quite remember. They find the answer on our site (and once they’re there, they realise what a great resource it is, and join in contributing knowledge). Good answers to story identification questions go into a little detail about the story, and so the page on our site has not just the ‘vague’ description from the asker, but also more complete information from the answerer.
The goal of Stack Exchange is not to construct un-findable single-serving questions that only help one person, but that’s exactly what guessing game questions tend to do.
We get the occasional “me too” non-answer on story identification questions (if you browse around, you won’t see these, because they get cleaned up because they aren’t answers – some survive as comments). This shows that story identification questions are useful to people other than the asker. I don’t have statistics for these, but I think they’re more common for story-identification than on average. Here’s a sample such non-answer:
Thank you so very very much for answering the question.
I stayed up nights for years trying to remember the name of that book so I could track down the second book and read it.
The question was highly relevant and direct as I had exactly the same question now finally answered. Thank you… Now I can get on with my life. 🙂
We get people finding these questions when looking for the same work themselves much more than Jeff realises (here’s just one more example). In some cases, we even feel that it’s worth “seeding” a story identification question, because it took so much work to find a story.
If we allow vague and insubstantial questions, we are explicitly opening the door to “do my work for me” questions (or worst case, Yahoo Answers) — no need to expend effort, do research, provide examples …
This is just like the “unanswerable” point above – it’s not specific to story identification questions. Good story identification questions explain the research and effort that has already been done. It’s often the case that there’s just a key word or phrase that the asker isn’t remembering that makes all the difference in finding the story. Consider this anecdote:
I know that about 10 years ago, I was trying to find an old time radio show, A Pail of Air, but kept searching for “bucket” instead of “pail.” It took me a long time to find the story because I kept using the wrong word on my searches. Then, later, when I found out it was on a broadcast of “X Minus 1,” I found a website that mentioned this is “one of those stories.” It’s one that everyone remembers parts of, but they forget the title and author.
Answering these questions isn’t necessarily hard work, either. A lot of users of our sites have an extremely broad knowledge of science fiction and fantasy works, and can recognise the work in question without having to do a lot of research. This doesn’t make the answer any less valuable to those that are needing the answer, any more than a core Python developer answering a Python question on StackOverflow without needing to do “hard work”. (In both cases, the hard work was done before the question was asked).
Also, an expert in the topic should be able to have at least some confidence that the answer he’s writinganswers the question.
This is just the “unanswerable” point again. Yes, if there’s not enough information in the question (any question, not just a story identification one), then it’s a bad question. Most story identification questions on our site have enough information that an expert is able to be confident in their answer.
But I would also argue that these questions aren’t educational in any way, because there’s no way to learn about the process of discovery. A particular community member, by virtue of their experience in the field, just happens to be able to take the limited information you remembered and fill in enough of the blanks to guess the correct answer.
Jeff’s totally wrong here. If it’s left such a good memory to someone else that they want to find it again, that’s a recommendation. It’s not like we get a lot of people asking about stories that they thought were really bad! These are stories that are so great that the memory of them has stuck with someone for many years, even though they often only read/watched it once. If you wanted to find a list of lesser-known stories that are worth trying, this is the best place to start (certainly the most common tags are no use for this – you probably already know about them!).
The “just happens” part is sometimes true, but again it’s true for other types of question as well, including ones on StackOverflow). When research was required, often the answer goes into detail about it, and we learn about Amazon’s “Search Inside”, the ISFDB, how to better do web searches, and more.
I urge you to click on the guessing game tags yourself and take a long, hard look at the artifacts these guessing game questions are producing. After a year I am convinced that guessing game questions do not meet our goal of making the Internet better.
I agree with the first sentence. Take a look at the story identification questions on our site (maybe you can answer some! maybe there’s something you should read!). If you’re convinced that they don’t add any value, then click on the “meta” link and share your opinion (by voting or answering) there and add the tag to your “ignored tags” list. Where Jeff and I disagree is with the final sentence: story identification questions do make the Internet better.
I can’t speak to the quality (or lack thereof) of identification questions on other StackExchange sites. However, within our site, about three times fewer story identification questions get closed (and then deleted) than other questions. Although the votes are slightly lower than the average, they aren’t extremely low. I’ve seen the interpretation that story identification are not policed and so poor questions (with too little detail) fall through the cracks. This does not agree with my experience. These questions are inherently difficult – difficult is ok! – but we still manage a decent response rate (90% have an answer that’s at least vaguely plausible).
Story identification is also great for promotion: this question form attracts both novices (who want their question answered) and experts (who want to show off their culture). I hope that we can organically recruit more experts, so that we can claim to be a place where story identification questions get answered – and those experts will contribute in other ways, too.
Bonus tidbit for anyone that gets to the end of the post. The title of this post refers to Asimov’s The Last Question, of which he said:
Frequently someone writes to ask me if I can give them the name of a story, which they think I may have written, and tell them where to find it. They don’t remember the title but when they describe the story it is invariably “The Last Question”. This has reached the point where I recently received a long-distance phone call from a desperate man who began, ‘Dr. Asimov, there’s a story I think you wrote, whose title I can’t remember – ‘ at which point I interrupted to tell him it was “The Last Question” and when I described the plot it proved to be indeed the story he was after. I left him convinced I could read minds at a distance of a thousand miles.