Tests
What can each kind of test prove?
Why it works this way
A sixth interview is still what people say: no number of interviews turns what people say into what people do.
The strength label is worked out from the kind of test you ran and how much of the planned run came back, and it always shows its reason. It is arithmetic, not an AI's opinion, so you can see where it came from.
A thin run is weak whatever the method, and no number of conversations changes what kind of evidence they are.
What we gave up: counting a clean pass as strong because it felt good. The label comes from the kind of test and what came back, and it says why.
What it is
Two tests can both come back with a clean pass and not be worth the same. Evidence has a strength, and what sets it is the kind of test, not how encouraging the result felt.
The ladder is short. What people say is the weakest rung and still worth having. What people do is worth more, because it happened. What people commit to, with money or a signature, is the strongest thing a test can show. A few tests have no customer in them. They can rule an idea out and can never confirm one.
How to use it
- Before you run a test, press its method chip. The pop-up says what people will be saying, doing or committing, where it happens, and what taking part costs them. An interview is an arranged setting: a room where someone is asked questions is not their ordinary week.
- Know the ceiling of your kind of test, below.
- When you call a test, the result carries How strong this is: No evidence yet, Weak, Moderate or Strong, with the reason beside it.
What each kind cannot show
- Customer interview: what people say they would do is not what they do. It can tell you a problem is real, never that anyone will pay.
- Expert interview: two experts agreeing is weaker than two customers agreeing. Their disagreement is worth more.
- Landing-page smoke test: an email address is not a purchase.
- Fake door: a click on something that does not exist is curiosity.
- Ad smoke test: you bought the attention.
- Pricing test: a price accepted in a conversation is not a price paid.
- Pre-sale: money is not a signature, and a signature is not a yes.
- Concierge: you did it by hand. The job is worth doing, but nothing here works without you.
- Wizard of Oz: a person was behind it. They wanted the outcome, but it may not be buildable.
- Prototype test: finishing a task is not wanting the thing.
- Existing-data analysis: the number describes the past.
- Reliability check: working is not wanted.
- Cost-per-run check: arithmetic about your costs. It can kill an idea, never validate one.
- Scenario stress test: your reasoning, not evidence from anyone.
What it will never do
- Show a strength label an AI made up. It is arithmetic on the kind of test and what came back.
- Mark your idea down. The label describes the evidence, never you.
- Call a test with nothing back "weak". Nothing back is no evidence, which is a different statement.
If something looks off
- "Nothing counted how many came back, so there is no grade for this run." You closed the test without a count. What the method can show still holds.
- A test with no customer says so: "No customer is involved. This can rule an idea out and can never confirm one."