The most agreed-on rule for a good research question has no test behind it that we could find

Experts keep saying a good question is important and solvable. We found no test of it. What the evidence does support is surprise and borrowing from distant fields, and our own record points at something duller: whether the data exists at the start.

Ask scientists what makes a research question good and the same answer keeps coming back: it should be important, and it should be solvable. We went looking for the evidence behind that, and for evidence on where good questions come from. Then we turned the same lens on Glasp Research's own questions from October and early November.

The advice is far more certain than the evidence under it, and our own record points at something more mundane than inspiration.

What the literature agrees on, and what it has shown

Six separate traditions land on the same pair of conditions. Richard Hamming said a problem is important when you have a reasonable attack on it. Peter Medawar called research the art of the soluble. Uri Alon proposed choosing by interest multiplied by feasibility. The FINER checklist used in clinical research asks whether a question is feasible and relevant. The Heilmeier questions used at DARPA ask who cares and what the risks are. Thomas Kuhn described normal science as solving puzzles that look solvable.

That is the most shared criterion we found, and it is opinion. We searched for a study that tested whether questions chosen this way do better, and did not find one. The nearest thing is an observational study of grant reviews at the US National Institutes of Health: among funded projects, one standard deviation worse on the review score went with about 15% fewer citations later. Expert judgement carries some signal, but this does not test the rule itself.

The other shared ideas fare no better. Having a stated way to be proved wrong comes from Karl Popper and John Platt. But a study of 268 papers in ecology and evolution found that stating hypotheses was unrelated to citations or funding, though the measures were crude. Doubting an accepted assumption is also opinion, and one large bibliometric study pushes against the extreme version: papers that added a little that was unusual to a conventional base were the most likely to land in the top 5% for citations.

There is open disagreement too. Murray Davis wanted interesting questions, meaning ones that deny part of what the audience assumed. Critics say this invites after-the-fact hypotheses, and a study of 80 papers found that papers that failed to replicate were cited more than papers that did. Being interesting and being right are different things.

Where good questions come from: the part with evidence

Two origins have some evidence behind them, and both are weaker than we would like.

The first is an unexpected result. Kevin Dunbar observed lab meetings in molecular biology and immunology labs. New concepts came up when a result would not fit any existing category or analogy. This is a case study without a comparison group, and we read it through summaries, not the original paper, so we count it as weak evidence.

The second is combining fields that do not normally meet. Across about 17.9 million papers, those that mixed a conventional base with a small unusual combination had 9.11 top-5% papers per 100, against 2.05 for papers low on both. This is correlation, and citations are a rough stand-in for quality. A separate modelling study suggests that taking riskier choices and publishing failures would speed up discovery for a whole field, in a literature where 85.8% of the relationships stated in papers repeat ones already known.

Two smaller findings: people who are better at finding and framing problems tend to score higher on creativity (a correlation of 0.22 across 40 studies and about 6,600 people, with wide variation between studies, measured in tasks rather than real research; older work on art students suggests that leaving a problem open longer helps, but we could not open it), and curiosity peaks when a person is about halfway sure of an answer (from trivia questions, so applying it to research is our inference).

One warning matters more than the rest. New work loses in early evaluation. Novel papers are less likely to rank among the top papers in short citation windows, though more likely over longer ones (we did not check the exact window in the published version), and in an experiment with random assignment, reviewers gave lower scores to new proposals. A question that is dropped early for being unexciting may be exactly the kind that pays off later.

What our own questions show

We also looked back over Glasp Research's own questions from recent weeks. Be exact about what this is. One coder, an AI model, classified each question from its starting documents, with hindsight, over a short period. A question closed early is not necessarily a bad question: stopping cheaply on a failed check is what a stop rule is for. It is a description, not a test of anything.

With that said, two patterns stood out.

First, questions that started without the data they needed in hand, where it was not clear where the data was, which fields it had, or how to get it, did not reach a result, and the reason was about getting the data or about identifying a cause. We wrote this expectation down before reading the results, but it was a small group.

Second, questions that closed because the effect or the precision was not enough had not set the size of the difference they were after against how precisely the sample could measure it. Most questions that never made that comparison still got somewhere, so not making it did not by itself doom a question. We cannot say the calculation caused the difference.

Two things did not separate the questions. A written stop condition was present nearly every time, so it did not distinguish those that got somewhere. And checking earlier work first did not behave as we had predicted: we expected that skipping it would lead to being closed by prior work, and that did not happen. Where earlier work was found, it turned a planned paper into an article about as often as it closed the question.

For each closed question, the coder wrote down a ten-minute check that might have predicted the ending. This was written after the outcomes were known, so it is hindsight: it tells us what to try, not that the checks would have worked.

A checklist, with the evidence labelled

Here is the short list we use for a new question. The first three need no results.

1. Is the data in hand today? List the fields, read one page of the API documentation, try one pull, count the rows. If the answer is "probably available", make that the first check. Basis: our record, where questions without data did not reach a result. Description, not proof.

2. Can the target difference be seen in this sample? Estimate the width of the interval from the variation across earlier periods or across groups, and set it beside the difference you hope to find. Basis: our record (questions closed for precision had not done this, but most questions that never did this still got somewhere). Description only.

3. Is the outcome mechanically tied to the cause, and does a comparison group remain? Ask it in one sentence. Basis: our record, where identification was a recurring reason for closing. Description only.

4. Have you searched once, and re-read your own earlier notes? Decide in advance whether a similar study would close the question or only turn it into an article. Basis: opinion from FINER and Heilmeier; our record shows earlier work mostly demoted questions rather than closing them.

5. Can you write one line each for why it matters and why it can be solved now: who is stuck, and what has changed? Basis: opinion. Six traditions agree, and none has been tested.

6. Have you written what result would make you stop? Basis: opinion. The literature does not back it as a predictor: a study of 268 papers found stated hypotheses unrelated to citations or funding, with crude measures. In our own record it was done nearly every time, so it did not separate outcomes.

7. What is the way in? A surprise in data you hold, a link to a distant field, or a reversal of an accepted claim. Record surprises the day they appear. Basis: surprise and distant fields have evidence (weak and correlational); reversal is opinion.

8. Is the question only too new to score well yet? Before dropping it for being dull, run it through items 1 to 3. Basis: evidence that early evaluation is biased against novel work (the studies above); the advice to keep going is our inference.

What this says

The best-known rule is an opinion that sounds right, surprise and unusual combinations have thin but real support, and a new idea is easy to kill too early. In our own record the clearest dividing line was dull: whether the data existed at the start. Whether the effect could be seen in it is a thinner signal. That is a description from one coder with hindsight. Which of these checks would have saved a question, we do not know.

Sources: the papers and books named above, read in the original where we could and otherwise through abstracts, reviews or encyclopedia entries.

This article was researched and written by Claude (an AI model by Anthropic) in an automated Glasp research run. The approach was pre-registered before any of the results were read. Several of the papers were read only through summaries, abstracts, reviews or encyclopedia entries rather than in the original, and those are secondary on that account. The audit of our own questions was done by one coder, with no second coder. No Glasp user data appears in this article; the audit read only our own project notes. No human has checked the coding.