The Last Question

Jeff Atwood, one of the founders of the StackExchange network, chose to mark his last (day-to-day) day at the company with a post disparaging identification questions.  As Jeff pointed out in his post, the [tag]story-identification[/tag] tag is the second most common on our site right now, so I’d like to point out why our community feels that these are a valuable part of the site, where we disagree with Jeff’s viewpoint, and hopefully give you a bit better understanding of how decisions about what’s on-topic are made (and then re-made, and re-made, and …).

The community consensus gets hashed out on “meta” – there have been quite a few discussions about story identification questions (starting way back at the launch of the site, up to and including during the recent moderation elections), so you should head over there if you’d like to make your opinion heard, or read in more depth (good places to start are here and here) – what’s below is heavily based on the content from there (I’ve borrowed some sentences verbatim).

This is my personal opinion as a user of the site, not an official statement.

I disagree with Jeff’s post right from the outset (all bold quotations are from Jeff’s post):

Our engine is great at these kinds of questions

Actually, while the StackExchange engine is great at some types of question that aren’t generally suited to the sites (e.g. polls), I find that there are many problems with using it for story identification, for example:

However, even though these aren’t a great fit with the StackExchange engine, they are a valuable part of our site, and they’re better than the current best the Internet has to offer: the Usenet group rec.arts.sf.written is well-known as a go-to place for story identification – they work ok on a Usenet group, but not so great: sometimes the thread diverges and the answer is buried amid hundreds of off-topic posts.  Consider how similar this problem is to the state of programming questions before StackOverflow came along.

A half-remembered description of something you vaguely recall is not what I’d call a practical, answerable question.

The “practicality” question (“solving problems“, or are we a “how to” site) has come up a few times on our meta.  It’s true that if you’re considering “practical” to mean “I need this for work” (and for StackOverflow, the first, most important, and most successful StackExchange site, that’s mostly the case), then (other than in rare cases) we’re not doing that.

However, understanding a development in a fantasy book I’m reading really is a “practical, answerable question based on an actual problem” that I face.  Wanting to know what order I should show my son Star Wars to maximise his enjoyment is also a practical (“of or concerned with the actual doing or use of something”), answerable question based on an actual problem that I face.  Being able to get back to sleep because I can finally stop trying to figure out what that book I read 30 years ago that I really, really, want to find again is also a practical, answerable question based on an actual problem that I face.

Story identification questions are clearly practical: they are concern with “doing” (finding the story) and using something (re-consuming the story).  If the question is too vague, too light on details, could match vast numbers of works, then yes, it’s not answerable.  But that’s ok! There are lots of questions outside story identification that are like this too – we have a process for these: we vote down the question, commenting on it what it needs to be improved.  And if the question is really terrible (“I read a book and it maybe had a green cover”, it can be closed, on merits, not because of its nature. “Unanswerable” isn’t a problem that’s limited to story identification questions.

2. Guessing game questions don’t help others

Actually, these are extremely useful to others.  The same “vague, broad, half-remembered descriptions” (or at least an overlap) will be used by others searching the web for this great story that they don’t quite remember.  They find the answer on our site (and once they’re there, they realise what a great resource it is, and join in contributing knowledge).  Good answers to story identification questions go into a little detail about the story, and so the page on our site has not just the ‘vague’ description from the asker, but also more complete information from the answerer.

The goal of Stack Exchange is not to construct un-findable single-serving questions that only help one person, but that’s exactly what guessing game questions tend to do.

We get the occasional “me too” non-answer on story identification questions (if you browse around, you won’t see these, because they get cleaned up because they aren’t answers – some survive as comments). This shows that story identification questions are useful to people other than the asker. I don’t have statistics for these, but I think they’re more common for  than on average. Here’s a sample such non-answer:

Thank you so very very much for answering the question.

I stayed up nights for years trying to remember the name of that book so I could track down the second book and read it.
The question was highly relevant and direct as I had exactly the same question now finally answered. Thank you… Now I can get on with my life. 🙂

We get people finding these questions when looking for the same work themselves much more than Jeff realises (here’s just one more example).  In some cases, we even feel that it’s worth “seeding” a story identification question, because it took so much work to find a story.

If we allow vague and insubstantial questions, we are explicitly opening the door to “do my work for me” questions (or worst case, Yahoo Answers) — no need to expend effort, do research, provide examples … 

This is just like the “unanswerable” point above – it’s not specific to story identification questions. Good story identification questions explain the research and effort that has already been done.  It’s often the case that there’s just a key word or phrase that the asker isn’t remembering that makes all the difference in finding the story.  Consider this anecdote:

I know that about 10 years ago, I was trying to find an old time radio show, A Pail of Air, but kept searching for “bucket” instead of “pail.” It took me a long time to find the story because I kept using the wrong word on my searches. Then, later, when I found out it was on a broadcast of “X Minus 1,” I found a website that mentioned this is “one of those stories.” It’s one that everyone remembers parts of, but they forget the title and author.

Answering these questions isn’t necessarily hard work, either.  A lot of users of our sites have an extremely broad knowledge of science fiction and fantasy works, and can recognise the work in question without having to do a lot of research.  This doesn’t make the answer any less valuable to those that are needing the answer, any more than a core Python developer answering a Python question on StackOverflow without needing to do “hard work”.  (In both cases, the hard work was done before the question was asked).

Also, an expert in the topic should be able to have at least some confidence that the answer he’s writinganswers the question.

This is just the “unanswerable” point again.  Yes, if there’s not enough information in the question (any question, not just a story identification one), then it’s a bad question.  Most story identification questions on our site have enough information that an expert is able to be confident in their answer.

But I would also argue that these questions aren’t educational in any way, because there’s no way to learn about the process of discovery. A particular community member, by virtue of their experience in the field, just happens to be able to take the limited information you remembered and fill in enough of the blanks to guess the correct answer.

Jeff’s totally wrong here.  If it’s left such a good memory to someone else that they want to find it again, that’s a recommendation.  It’s not like we get a lot of people asking about stories that they thought were really bad!  These are stories that are so great that the memory of them has stuck with someone for many years, even though they often only read/watched it once.  If you wanted to find a list of lesser-known stories that are worth trying, this is the best place to start (certainly the most common tags are no use for this – you probably already know about them!).

The “just happens” part is sometimes true, but again it’s true for other types of question as well, including ones on StackOverflow). When research was required, often the answer goes into detail about it, and we learn about Amazon’s “Search Inside”, the ISFDB, how to better do web searches, and more.

I urge you to click on the guessing game tags yourself and take a long, hard look at the artifacts these guessing game questions are producing. After a year I am convinced that guessing game questions do not meet our goal of making the Internet better.

I agree with the first sentence.  Take a look at the story identification questions on our site (maybe you can answer some! maybe there’s something you should read!).  If you’re convinced that they don’t add any value, then click on the “meta” link and share your opinion (by voting or answering) there and add the tag to your “ignored tags” list.  Where Jeff and I disagree is with the final sentence: story identification questions do make the Internet better.

I can’t speak to the quality (or lack thereof) of identification questions on other StackExchange sites.  However, within our site, about three times fewer story identification questions get closed (and then deleted) than other questions.  Although the votes are slightly lower than the average, they aren’t extremely low.  I’ve seen the interpretation that story identification are not policed and so poor questions (with too little detail) fall through the cracks. This does not agree with my experience.  These questions are inherently difficult – difficult is ok! – but we still manage a decent response rate (90% have an answer that’s at least vaguely plausible).

Story identification is also great for promotion: this question form attracts both novices (who want their question answered) and experts (who want to show off their culture). I hope that we can organically recruit more experts, so that we can claim to be a place where story identification questions get answered – and those experts will contribute in other ways, too.

Bonus tidbit for anyone that gets to the end of the post.  The title of this post refers to Asimov’s The Last Question, of which he said:

Frequently someone writes to ask me if I can give them the name of a story, which they think I may have written, and tell them where to find it. They don’t remember the title but when they describe the story it is invariably “The Last Question”. This has reached the point where I recently received a long-distance phone call from a desperate man who began, ‘Dr. Asimov, there’s a story I think you wrote, whose title I can’t remember – ‘ at which point I interrupted to tell him it was “The Last Question” and when I described the plot it proved to be indeed the story he was after. I left him convinced I could read minds at a distance of a thousand miles.

Site News

8 thoughts on “The Last Question

  1. Are we able to do analysis of the types of traffic that story-identification questions bring to SFF.SE? I feel like that knowledge would help address or explain some of Jeff’s points. If we are getting a steady flow of Google referrals due to people typing vague story descriptions into Google, then we are making the internet better.

    • I know that other venues tend to attract people for story identifications. I’ve seen that many times back when I lurked on rec.arts.sf.written, people would come to that Usenet (or lately Google Groups) thingy as a last hope to ask the title of that story. (YASID: Yet Another Story Identification.) And often get an answer within five minutes. Now what I don’t know is whether that’s just drive-by traffic or there is user retention. Another point to consider is that story identifications attract experts who want to show off how well-read they are. Rasfw has this informal competition of who will be first to answer the most YASIDs correctly.

  2. Like I said in the comments to Jeff’s blog-post: it comes down to the discoverability issue. Can we show that people are finding existing [story-identification] post (show, not guess)? If so they’re good. On the other hand, if people aren’t finding them they they are not doing any good.

    At this point we have few duplicates, which bodes well, but I haven’t looked to see how [story-identification] compares to other tags on that metric.

    • We can show that other people are finding existing questions because they leave “me too” answers like the one Tony quoted. My informal impression (being a mod on scifi, I’ve deleted several of these) is that they’re more common for story identifications than average.

  3. Well, look at the data.

    571 questions with views of 500 or more.

    Of those ONLY SIXTEEN, 2.8% are [story-identification].

    That’s a little odd, don’t you think, considering [story-identification] is the second most common tag on the site?

    Like I was saying.. “The goal of Stack Exchange is not to construct un-findable single-serving questions that only help one person, but that’s exactly what guessing game questions tend to do.”

    I know they’re fun in a way, but a) we’re not shy about demoting fun when it isn’t producing useful artifacts and b) you have to take a cold, hard look at the utility of the artifacts that are produced.

    • Here’s another way to look at the data:

      Of all tags, [story-identification] currently sits around the middle in terms of number of views. That seems about right to me – these aren’t questions with mass appeal (they’re not [star-trek] or [doctor-who]), but they aren’t right at the very end of the long tail, either, like [the-almighty-johnsons]. We’re trying to help people, not be popular (the community, at least – presumably the company is trying to have both).

      I didn’t claim these were fun – they are practical, answerable, educational, and they bring in new users. In the opinion of the community (at least, the majority of those that have spoken up on meta), those are useful artifacts.

      The community has taken a “cold, hard look” at the utility of these questions. This isn’t the first time we’ve used the API and data explorer (as well as the tools in the main site itself) to consider their merit.

  4. @Tony scoping that query to tags with at least 20 questions so there’s sufficient data, [story-identification] ranks 41st out of 43 total tags in average views per question.

    (add having Count(*) > 20 to the query clause)

    So I guess it’s a “success” that the 2nd most popular tag on the site isn’t, uh, dead last in average views. It did beat [fantasy-genre] and [the-new-52] at 202 and 166 average views per question.

  5. I’m on the fence on these questions, but I just want to bring up one more angle to look at them from. Has anyone gone through all the answers to see if the same stories are being sought using different “vague, broad, half-remembered descriptions”? Out of 380 [story-identification] questions on the site right now, I only see two that were closed as exact duplicates. If people are finding half-remembered stories without posting lots of duplicates (presumably by searching and finding an earlier description), these questions would seem to be at least somewhat useful. If, on the other hand, you have lots of different vague recollections of the same story, then I’d have to say that the questions are only useful to the people who post them.

Leave a Comment