FAQ
Everything you need to know before you submit. Jump to a section below or scroll through.
How Verdict AI scores anything
Verdict AI runs every programme through one engine. What changes between programmes is the criteria pack — the list of criteria, the topics they group into, and the rules that turn them into a certification. The mechanics below are identical whichever programme you submit to; the numbers are in your programme's own section further down.
Every criterion is scored 0 to 5. Criteria are grouped into topics, and a topic's score is the average of its criteria. There is no total, no percentage and no weighting — your certification comes from how many topics reach each band, never from a points tally. Nothing in Verdict AI is marked out of a hundred.
A strong submission demonstrates a working system end to end, not a description of one. Submissions that lean on stubs, mocks or hardcoded responses score in the lower bands regardless of polish, because the scale reserves 4 and 5 for work that is genuinely considered rather than merely functional. Working-but-unremarkable is a 3 by design — a rubric where everything competent scored 5 would teach you nothing.
Your report is written against your own source. The scores say where you landed; the written assessment says why, and What To Do Next says what to change first.
How to submit
- Programme: pick yours from the dropdown on the submission form. It selects which rubric you are scored against, so submitting a Java service to the Gen AI programme will not go well.
- Format: a single .zip of your project. The file picker accepts only .zip; the server rejects anything else.
- Size: 20 MB maximum — the server refuses anything larger. Strip node_modules, .venv, target/ and build artefacts before zipping.
- Workflow run ID: only some programmes ask for one, and the field appears only when yours does — if you cannot see it, your programme does not score a workflow run and there is nothing to provide. Where it is asked for it stays optional to fill in, and is always verified when you do: that the run is yours, that it completed, that it is a run of the workflow you submitted, and which of your nodes actually executed. Your programme's section says what omitting it costs.
- Identity: your name and email come from your Bodhi login (read-only). You cannot submit on someone else's behalf.
- Runtime: most evaluations finish in one to three minutes. The progress view traces each workflow node live; no need to refresh, and you can leave the page.
- Result: a verdict screen with your certification, every topic banded, the written assessment and what to do next — plus a Download report (PDF) button. The submission is also added to your History tab.
- Iteration: re-submit as many times as you like. Duplicate detection only blocks a project identical to one from another account, never your own re-runs.
How your certification is decided
There is no score out of anything and no percentage. Four things are applied in order, and any one of them can decide the outcome on its own.
First, the gates — the programme's automatic-fail requirements. Fail any single one and you land at the lowest certification regardless of every score you earned. Your programme lists its own.
Second, the floor. Each programme sets a minimum standard below which a submission is not a weak attempt but a different assignment. The mechanism differs by programme: some name a couple of criteria that can never drop below a threshold while tolerating a few weak ones elsewhere, others require every criterion to clear a bar. Yours is described in its section.
Third, topic counts. Each topic averages its criteria and lands in one of four bands, using the thresholds in the table below — these are the same for every programme. Your certification is then a question of how many topics reached which band, plus in some programmes a bonus capability or a specific topic that must itself reach a band. Your programme's exact rule is in its section.
Fourth, the maturity cap — the one that surprises people most. See the next section.
Across every programme, Advance is the level required to complete. Mastery is deliberately harder than the programme asks for, and is not the expected outcome.
| Level | Topic average | What it means |
|---|---|---|
| Beginner | below 2.0 | The area is absent, or present only as scaffolding. This is also the band a criterion is forced to when a gate fails against it. |
| Standard | 2.0 and above | The area exists and works, with gaps a reviewer would raise. Assembled rather than engineered. |
| Advance | 3.5 and above | Done deliberately, with the reasoning visible. This is the band that counts toward completing the programme. |
| Mastery | 4.5 and above | Production-grade, including the parts most submissions skip — measurement, failure handling, and the case for the choices made. |
The maturity cap
Every programme classifies your submission on a maturity ladder from L0 to L4, and that classification puts a ceiling on your certification which no amount of topic scoring can lift. This is the most consequential mechanic in the engine and the most common reason a candidate improves their scores and does not move.
The ceiling is the same everywhere: L0 caps you at Beginner, L1 at Standard, L2 at Advance. L3 and above lift it entirely. What differs by programme is what each rung means, and your report names the level you were placed at along with the reasoning.
Maturity also works a second way that is easy to miss: below certain levels the rubric caps individual criteria outright, and capped criteria drag down every topic containing them. So at L1 the ceiling often never even comes into play — the criterion caps have already done the damage.
The practical consequence: if your report says maturity is holding your level, polishing topic scores will not move you. Raise the maturity of the system itself first. Your report names which criteria are being capped and what the ceiling is.
Gen AI 201 L2 — what we expect
24 criteria grouped into 13 topics. At minimum your ZIP should contain a clearly stated problem with a named beneficiary, a runnable interface, an agent loop that calls real tools (retrieval over a vector store, MCP-published APIs, or both), deliberate guardrails, and enough observability that a reviewer can trace what happened on a request.
Two things carry more weight than most people expect. Your use case has to be relevant to Publicis Sapient — see below, because an irrelevant one holds your score down however well it is built. And a workflow run ID is what proves the thing actually runs: this programme gates on it, so omitting one holds you at Beginner.
Gen AI — what makes a use case acceptable
The programme exists to produce things Publicis Sapient can use. Exactly two shapes qualify, and anything else holds your use-case score low no matter how good the engineering is.
- A client engagement. This needs BOTH a stated business outcome AND a specific named PS client or account that could use the POC or project. "A retailer", "the banking sector" or an invented company identifies no beneficiary and does not count.
- An internal developer-productivity tool that measurably improves how PS engineers work.
- Document it, do not imply it. Reaching the upper bands requires the case written down — who it is for, the outcome it targets, and how success would be judged. The top band also wants the value quantified and a path to putting it in front of that client or team.
- Generic chatbots, simple Q&A, summarisers and hello-world projects are rejected by a gate before any of the above applies.
- Use case is one of two criteria that can never fall below 3 on this programme, so getting it wrong is not a few lost points — it decides the verdict.
Gen AI — the 13 topics
Each topic is the average of its criteria on the 0–5 scale. Topics count equally toward certification — no weights, no points, so a weak topic costs you the same wherever it sits.
Whether the project serves a named PS client with a business outcome, or improves developer productivity — and whether that case is documented rather than implied. See the section above.
Routing, the control loop, and multi-agent delegation. A ReAct-style tool-calling loop is the framework default and scores as competent rather than advanced; the upper bands want routing you designed, with the selection criteria visible and the wrong branch handled. Delegation means an orchestrator handing work to subagents, not one agent calling several tools.
How well you use the current framework rather than how much of it you hand-roll. Middleware or a custom graph should appear where the default agent is genuinely insufficient, with the reason evident. Deprecated entry points — AgentExecutor, initialize_agent, the old create_react_agent — count against the top band.
Tools, their schemas, and MCP. Typed input and output schemas that are actually validated, with malformed model output repaired rather than crashing. MCP is scored on transports, on primitives beyond Tools (Resources, Prompts, Sampling), and on tool-shadowing defences.
Reranking, metadata filtering, query rewriting or semantic chunking beyond naive splitting — and at the top, hybrid search plus an actual measurement of retrieval quality rather than an assertion of it. A deliberate chunking pattern can fold in here as a bonus.
Persistence and goal tracking, plus context engineering: what actually enters the model's window on each call. Retrieval scoped to the step that needs it, an explicit budget, and compaction or subagent isolation chosen against a real constraint rather than by default.
Decomposition, planning approach, and self-critique. The critique matters more than its mechanism: which outputs get checked, why those and not others, and evidence the check actually changes an output rather than always approving it.
Can you tell whether a change made the agent better? A fixed test set the agent is actually run against, automated scoring of tool-call and routing correctness, and at the top a regression gate in CI against a tracked baseline. Ad-hoc manual prompting does not count. This is the topic most submissions score zero on.
Error handling, guardrails, human-in-the-loop and observability. Two things are commonly missed: containment for a confidently WRONG answer as opposed to a thrown exception, and a demonstrated injection attempt actually being caught. Observability means traces a reviewer can read — any OpenTelemetry-conformant backend qualifies; no vendor is required.
Model calls routed through the Bodhi gateway, with retries and deliberate model selection; an enforced per-request cost or token budget at the top. Calling a provider directly fails a gate outright, and this is the second of the two criteria that can never fall below 3.
The interface and how results reach the user — streaming with back-pressure handled, and multi-channel delivery at the top band.
Whether a reader can understand the design and your reasons for it without opening every file. Documentation of the use case itself is scored separately, inside the use-case criterion.
Measured from your submitted Bodhi workflow rather than judged by hand: conditions, repeat, subworkflows, multi-agent, variables and preview. Demonstrate fewer than five of those six and the topic is held below Advance. It is also the only topic that can block certification by itself — see the tier rule below.
Gen AI — the gates
Fail any one of these and you are at Beginner however well the topics scored. Most of them also zero or floor the criterion they relate to, so the damage shows up in your topic scores too.
- A Bodhi Workflow Builder artefact is present and can be parsed.
- The submission is technically sound — it is a real project, not fragments.
- It meets the type requirement for the programme.
- Model calls go through the Bodhi gateway. Direct OpenAI/Groq/Anthropic calls without a Bodhi base_url fail, and so does provider routing a reviewer cannot determine from the source.
- The use case is not trivial. Generic chatbots, simple Q&A, summariser-only projects and hello-worlds are rejected here.
- A workflow run was verified. This is the run ID gate — no run ID means no verification, which means Beginner.
Gen AI — the floor
Two criteria are non-negotiable and must reach 3: your use case, and Bodhi gateway integration. A trivial use case or a bypassed gateway is not a weak submission, it is a different assignment.
Beyond those two, up to three criteria may sit below 3 before the floor fails. One weak area costs you bands, not the whole verdict; four does.
Gen AI — the tier rule
Of the 13 topics, how many reached which band. Advance is what completing the programme requires.
A "bonus capability" means Agentic Microservices or use of a Cloud AI Platform. Neither is scored as a topic — they exist only to satisfy the rules below.
| Level | How it’s decided | What it means |
|---|---|---|
| Beginner | anything short of Standard | Also where you land if a gate fails, the floor fails, or no workflow run was provided — however well the topics scored. Your report names which of those it was. |
| Standard | 4 topics at Advance, and 11 of the 13 at Standard or above | The pieces are assembled and working. Breadth is the binding constraint rather than depth: nearly every topic has to be at least functional. |
| Advance | 10 topics at Advance, plus one at Mastery or a bonus capability — and four topics that must reach a band in their own right: Workflow Builder Mastery, Reliability & Safety and Tool Integration at Advance, Evaluation & Testing at Standard | The level required to complete the programme. Those named topics are the point: counting alone let a submission skip human-in-the-loop, or observability, or MCP entirely and still certify. They cannot be averaged around. |
| Mastery | 12 topics at Advance, 7 of them at Mastery, a bonus capability, and Workflow Builder Mastery, Reliability & Safety, Tool Integration and Evaluation & Testing each at Advance | Production-grade across nearly the whole rubric. Deliberately hard, and not the expected outcome of the programme. |
Gen AI — common pitfalls
The patterns that consistently cost bands, in rough order of how often they appear.
- No workflow run ID, so nothing demonstrates the project runs. Beginner on its own.
- A node wired into the workflow but never reached on any run. Source analysis cannot tell the difference; the execution log can, and the report names the nodes that never ran.
- No way to tell whether a change helped — no test set, no scored trajectories, no regression check.
- A use case with no named beneficiary. Technically strong projects fail here regularly.
- Fewer than five of the six Workflow Builder features, which holds that topic below Advance and blocks Advance certification outright.
- No handling for a confidently wrong answer, as opposed to a thrown exception. Guardrails that only catch crashes are half a guardrail.
- Missing validation of model output — JSON parsing without a fallback, no schema check on tool calls.
- No observability, so a reviewer cannot trace what the agent did on a request.
- Retrieval returning irrelevant results because chunking, embedding or search was never tuned to the corpus, and never measured.
- MCP tools declared but not registered, or exposed without input/output schemas.
- Infinite loop potential — the agent can re-plan or re-call itself with no step or cost cap.
- Everything dumped into the context window on every call, with no budget and no stated reason for what is kept.
- Deprecated framework entry points left in place, which cap Framework Depth however well the rest is built.