Why You've Never Won an SBIR

If you've submitted five Small Business Innovation Research (SBIR) proposals and lost all five, here's something nobody at the Department will tell you: it may not have been your idea.

It may have been who read it, and for how long.

At AFWERX, I watched seven fellow evaluators try to get through hundreds of proposals each in a single cycle. You spent forty hours on yours. It got read in forty seconds. Then we opened evaluation to the entire Air Force, which fixed the math and broke the judgment. The other services never opened it at all, and their few evaluators still give each proposal the same twenty to thirty-five seconds. Same verdict, reached two different ways.

BLUF: SBIR doesn't reward the best technology. It rewards the best proposal writers, which is why a tiny set of “SBIR mills” keeps winning while the real builders lose. Every service gets there a different way — the Air Force democratized evaluation and drowned judgment in a crowd; the others never democratized and still skim each proposal in seconds with too few reviewers — but the outcome is identical. The fix doesn't care which failure you have: stop asking reviewers to agree on what's excellent, because they never will, and ask them what's terrible, because they almost always agree. Filter out the lemons with the masses, then grade the survivors on evidence, not prose, with a small bench of experts.

Why you keep losing

Every federal agency with a large extramural research budget pays 3.65% into SBIR and the Small Business Technology Transfer (STTR) program, and only gets it back by funding small businesses.^[1] AFWERX's Open Topic opened that money to thousands of companies that had never touched defense. It worked so well Congress made the rest of the Department copy it.^[2]

Then ten to twelve thousand proposals a year arrived, and the rule AFWERX ran under called for three independent evaluations of each. A handful of reviewers couldn't do it, so proposals got skimmed. When evaluation opened to the whole Air Force, thousands of volunteers with no shared standard took over. Scores tracked enthusiasm and time of day more than merit.

In a system like that, the proposal that wins is the one that's easiest to score in forty seconds. That's a skill, and it's not the same skill as building technology. It's why SBIR mills exist: firms whose real product is the proposal. They win again and again, and the founder with a working prototype and a clumsy write-up loses again and again. My old Defense Innovation Unit (DIU) colleague David Rothzeid laid the numbers out: the multi-award winners are under one percent of participants but take more than ten percent of Phase II dollars, and from 2011 to 2020 just twenty-two firms captured more than ten percent of them.^[3]

There was one silver lining in the open model. Fully anonymized, distributed evaluation was hard to protest, because nobody knew who reviewed what. Keep that in mind. It's half the answer.

The other services are the mirror image. They never opened evaluation, so they have no crowd problem — just too few evaluators giving each proposal those same twenty to thirty-five seconds. Over-democratize and the crowd picks badly; under-resource and the handful skims. The answer isn't more reviewers or fewer. It's using the crowd for the one thing a crowd is actually good at.

Everyone agrees on terrible

Here's the insight that fixes it. Put a hundred reviewers in a room and they'll never agree on what excellent looks like. But nearly all of them will agree on what's terrible.

Mark Klein and colleagues at MIT turned that into a method called the Bag of Lemons.^[4] It inverts the vote.

Each evaluator chooses how many proposals to review. Say twenty.
They get the same number of tokens. Twenty reviews, twenty tokens.
Tokens mean no, not yes. A reviewer can dump all twenty on one terrible proposal or spread them thin.
Every token comes with a one-line justification, logged.
Proposals with low token counts advance. The lemons don't.

The point isn't to pick winners. It's to clear the floor so experts spend their time on the proposals worth it.

Here's why that matters to you if you've never won. In a lemons pass, polish stops being an advantage. A beautifully written proposal for a bad idea still collects lemons. A clumsy write-up of real technology doesn't, because reviewers can usually tell terrible from merely awkward. The mill's edge disappears in Stage I. Call this Stage I: the whole service clearing the floor.

We already do half of it

If token-based funding sounds exotic, it isn't. The Air Force already runs it.

In 2018, General Goldfein set aside $64 million as the Squadron Innovation Fund (SIF).^[5] Today it runs as a marketplace: squadron commanders allocate virtual tokens to the initiatives they believe in, including ones outside their own squadron, wing, and major command.

That's a preference market. The Bag of Lemons is the same machinery with the sign flipped. We've already proven commanders will use tokens responsibly. Now let's use them where the Department has the worst signal-to-noise ratio it owns.

Stage II: the bar rises with the money

The proposals that survive the lemons go to a much smaller group: trained evaluators from operations, acquisition, and contracting. In this 3OS series I call them Pathfinders. And yes, I know that has a different meaning in the Army than in the Air Force. I didn't pick the name, but I like what it implies and I'm going with it.

The cadre of evaluation experts then won't grade prose. They grade evidence, and the evidence they ask for depends on the stage you're at. A Phase I applicant shouldn't need a working product, because Phase I exists to prove feasibility. A STRATFI applicant should need a lot more. Either way, Stage II grades the same four things on every award — technical merit, qualifications, commercial potential, and mission fit — and only the bar rises.

Award	Technical merit	Qualifications	Commercial potential	Mission fit
Phase I	A credible feasibility argument; no product required	Prior work showing the team can plausibly build it	A plausible path to a customer	A named mission thread it serves
Standard Phase II	What Phase I actually proved, against what it promised	Delivered Phase I on time and to spec	An early demand signal	An operational user engaged on the thread
Direct-to-Phase II	Existing feasibility evidence, since there's no Phase I record to check	A track record of shipping comparable work	Market or government pull already shown	A named mission thread and user
TACFI	A working prototype against Phase II results	Demonstrated delivery under contract	One committed match: programmed (POM'd) government funds or outside private capital	A program office or operator in the loop
STRATFI	Capability demonstrated at relevant scale	A delivery history that de-risks scaling	Committed outside private capital, required	Programmed (POM'd) government funds committed, plus a named end user and program office
Sequential Phase II	Test-harness and deployment data from the prior Phase II	Repeated delivery across efforts	A sustaining path beyond the program	Continued operator demand
Phase III	Test-harness pass rates	Proven at operational scale	No SBA money left; it runs on real contract dollars	Programmed (POM'd) government funds, required at this stage; a program office ready to carry it

The lemons filter is the same at every stage, because terrible is terrible whether it's asking for Phase I or a strategic award worth tens of millions. What changes is the bar after it.

That's the fair path in. A research-stage founder competes on whether the idea connects to a real mission. A company asking for strategic money competes on whether people with checkbooks already believe in it.

Conflicts are handled the boring way. A Pathfinder who's the technical point of contact, end user, or customer on a proposal recuses, and another Pathfinder evaluates it.

The Small Business Administration (SBA) still sets the rules. Its policy directive, and the Department's own solicitation, codify three evaluation criteria in descending order of weight: technical merit, the qualifications of the team, and commercial potential.^[6] Mission fit isn't a fourth criterion on paper — the Department folds it into technical merit and enforces it at the topic gate. I break it out anyway, because it's the one judgment that doesn't come from the lab. The applicant may own the market; the operator owns the mission, lives it every day, and draws on a body of knowledge the applicant likely doesn't have. It's also why Pathfinders can't be drawn from the acquisition community alone — acquisition doesn't understand mission any better than the scientist does. You need the operators in the room.

The objections, answered

"Three reviews per proposal is too thin a signal." That's the system we have now. Today, three reviewers who are often underqualified pick winners. The Bag of Lemons asks those same three people an easier question, one they're actually qualified to answer: is this terrible? It also adds guardrails the current system doesn't have. A proposal is eliminated only when at least two of its three reviewers flag it, and token counts are normalized so a harsh grader and a generous one carry the same weight.

"What stops someone from aiming tokens at a competitor?" Evaluators are government personnel, not companies. They're assigned proposals at random, so nobody can go hunting for a rival. Every token carries a logged justification. And reviewers whose lemons consistently diverge from consensus get flagged and down-weighted, which is the same math that made the open model protest-resistant.

"Won't this just reward bland proposals?" Stage I doesn't pick winners. It only removes the obvious failures. Bold proposals that a few reviewers dislike survive, because one dissent isn't enough to eliminate. Excellence gets judged in Stage II, by people qualified to judge it.

"I don't have a product yet. Doesn't 'grade evidence' lock me out?" No, and this is the most important design choice in the model. Phase I asks only for a credible path to a real mission thread. Product evidence comes into play at Phase II and beyond, when the money is large enough to justify it. If you're research-stage, Stage I protects you from losing to polish, and Stage II judges you against other research-stage companies, not against firms with three years of revenue.

“Your expert panel will just make expert-flavored mistakes.” Sometimes it will. But it's less likely than the current broad panels are to make mistakes. I once watched an Army SBIR panel downgrade a proposal because they were certain it defied physics — when the proposal carried the mathematics and the live-demonstration data showing it did no such thing. The applicant was an MIT PhD who was right, and the evaluator who dinged the evaluation was wrong. Sitting at the rank to be an evaluator doesn't make you better at math than the astrophysicist. That's exactly why Stage II grades demonstrated evidence — live demos, test data, the math — and puts it on the record, instead of trusting a reviewer's gut. Expertise earns a seat; the evidence still decides.

The fix, and what you can do about it

Run the next SBIR call in two stages. Stage I is the whole service voting lemons out, with randomized assignment, logged justifications, and a two-of-three rule. Stage II is a small bench of trained evaluators grading evidence that fits the stage. No new authority is required. The tax is already collected, the evaluators already exist, and the token machinery already runs in SIF.

If you're a founder who's never won, you can't change the evaluation model yourself. But you can make the case that it should change:

Tell your story in the comments. Tell me on LinkedIn how many proposals, how many losses, and what feedback you got. A pile of real stories is evidence; one blog post is an opinion.
Tell your congressional delegation. SBIR is a congressional program. Members respond when constituents who create jobs say it isn't working for them.
Ask for a debrief: what the evaluators scored, and why.

SBIR should reward the company that built the best thing, not the one that wrote the best description of it.

What you can do Monday

Unit commanders: Put your sharpest operators forward as Stage II evaluators. They know the mission threads better than any proposal does.
PEOs / PAEs: Publish which topics you would actually absorb in the next four quarters, and volunteer technical points of contact for them.
SBIR program managers (AFWERX, NavalX, Army, and others): Run one topic as a two-stage lemons pilot and publish the results.
Service personnel offices: Count evaluation duty as a performance bullet, so reviewing well is worth doing.
DIU and the service innovation organizations: Publish evaluator workload: proposals per evaluator and minutes per review.
OSD: Issue guidance that explicitly allows a two-stage evaluation model across the Department's SBIR programs.
Congress: The 2026 reauthorization already caps proposals per company starting in FY2027^[1]. That’s a volume fix. Pair it with an evaluation fix: require agencies to report evaluator workload and time per review.