How to Run a Startup Evaluation Panel That Holds Up Later

The number that looks comparable and is not

Here is the most common way an incubator's selection process quietly stops working.

Two reviewers each score an application "out of ten". One gives it a 7. The other gives it a 7. The average is 7, everyone moves on, and the spreadsheet looks tidy.

But the first reviewer was thinking mostly about the team, and the second was thinking mostly about whether the market was real. They agreed on a number and disagreed on the company. Nobody notices, because the average still computes.

Now do that across eighty applications, five evaluators and three weeks, and you have a ranked list that feels rigorous and cannot be defended. The problem is not that your reviewers are careless. It is that "out of ten" is not a question.

Start with the criteria, not the scale

A rubric is just the list of things you have decided actually matter, each with a weight. Writing one is a half-day of argument and it is the highest-value half day in the whole selection process, because it forces the disagreement to happen before the scoring rather than after it.

A workable structure for most incubation programmes:

Criterion	Typical weight	What the reviewer is actually judging
Problem and market	25%	Is this a real problem, for enough people, that they will pay to fix
Team	25%	Can these specific people do this specific thing
Product and traction	20%	Does something exist, and has anyone used it
Fit with the programme	20%	Can we actually help, with what we have
Feasibility of the ask	10%	Is the plan for the next twelve months plausible

The weights matter less than the fact that they are written down and the same for everyone. What you must avoid is the criterion nobody can score consistently - "innovativeness" is the usual culprit. If two reasonable people would score it differently for reasons they cannot articulate, it is not a criterion, it is a mood.

Give each criterion a scale with words

Numbers alone drift. A five-point scale where each point has a sentence attached drifts much less:

• 1 - No evidence at all, or evidence that contradicts the claim
• 2 - Asserted but unsupported
• 3 - Supported by something the applicant did, not just said
• 4 - Supported by evidence a sceptic would accept
• 5 - Exceptional, and rare enough that most cohorts have none

The point is not precision. It is that "3" now means roughly the same thing to two different people, which is the whole game.

Brief the panel properly, and brief them once

Most panels are briefed by forwarding a spreadsheet. Twenty minutes at the start of the round saves a fortnight of reconciliation later. Cover four things:

1. What the programme is for. A reviewer who does not know whether you are selecting for growth potential or for local employment will optimise for the wrong thing, confidently.
2. What each criterion means, using the scale above, with one worked example scored live in front of them.
3. Conflicts of interest. Who declares what, and what happens when someone knows an applicant. Decide the rule before the case arrives, because deciding it afterwards always looks like it was decided to suit the outcome.
4. That they score independently first. This is the one people ignore, and it is the one that matters most.

Score independently, then discuss

Independent scoring produces a spread. The spread is the useful part - it tells you where the panel actually disagrees, which is exactly where the conversation should go.

If everyone scores an application 4, 4, 4, there is nothing to discuss and you can move on in ninety seconds. If the scores are 2, 3, 5, that application deserves ten minutes, and you have found it automatically rather than by someone remembering to raise it.

Reviewers may change their scores after discussion. That is not a failure of process, it is the process working. Record that the change happened, who made it, and when. A score that moved after a discussion is fine. A score that moved with no record of why is the thing that will be difficult to explain later.

Keep the breakdown, not just the total

This is the part that gets thrown away, and it is the part you will need.

When a funder, a governing body or an unsuccessful applicant asks how a cohort was chosen - and eight months later, one of them will - the useful answer is not "they scored 7.4". It is:

Five evaluators scored independently against a five-criterion weighted rubric. On market, the scores were 4, 4, 3, 4, 5. On team, 2, 3, 2, 2, 3. The team score is why they did not make the cohort, and here is the rubric as it stood that day.

That answer is only available if you kept the per-evaluator, per-criterion breakdown. If your system averaged everything into one number at the point of submission, the information is gone and no amount of care afterwards recovers it.

The same applies across cohorts. If you keep the breakdowns, you can eventually answer a much more interesting question: which criteria actually predicted which startups did well? Most incubators cannot answer that about their own programmes, because the data was averaged away every round.

Scope your external evaluators narrowly

Good panels usually include people from outside your organisation - an operator, an investor, a technical specialist. They need to see the applications for one programme. They do not need your funding records, your other cohorts, or your startup directory.

If the only way to give someone access is a full account, you end up choosing between an over-permissioned stranger and running evaluation over email attachments. Both are bad, and the second one is how the audit trail disappears.

The right shape is an evaluator who is attached to a specific programme, sees only its applications, and cannot reach anything else - without consuming a full staff seat or requiring an administrator to remember to remove them afterwards.

A checklist you can use this round

• The rubric is written down and weighted before any application is read
• Every criterion has a scale with words, not just numbers
• Nobody scores an application they have declared a conflict on
• Evaluators score independently before any discussion
• Disagreement above a threshold you set triggers a conversation
• Score changes after discussion are recorded with who and when
• The per-evaluator, per-criterion breakdown is retained, not averaged
• External evaluators can see one programme and nothing else
• The rubric version used is stored with the round, not overwritten

Where the software part comes in

None of the above requires software. You can do all of it on paper, and small programmes reasonably do.

It stops being practical at roughly the point where you have more applications than one person can hold in their head, more than three evaluators, or a funder who will ask questions later. At that point the things that break first are always the same two: the per-evaluator breakdown gets averaged away, and the external evaluator gets an account that can see everything.

If you are choosing a system, those are the two questions worth asking a vendor before any of the others.