zeromodel

Local Vision Model → ZeroModel Arcade Policy Bridge

Overview

This example demonstrates how a local vision-language model can operate as an external observation provider for a ZeroModel policy.

The local model observes a rendered arcade frame and reports a small set of visible state variables. ZeroModel validates those observations, converts them into a stable policy-state address, and selects an action from an independently compiled Visual Policy Map.

rendered arcade frame
        ↓
local vision-language model
        ↓
bounded text observation protocol
        ↓
deterministic parsing and validation
        ↓
stable ZeroModel policy row
        ↓
identified VPM policy
        ↓
VPMPolicyLookup
        ↓
LEFT / RIGHT / STAY / FIRE

The model performs perception.

ZeroModel performs state validation, policy addressing, action selection, and trace construction.

The model is never given the action policy and is never asked to choose LEFT, RIGHT, STAY, or FIRE.

Every case is additionally persisted as a ProviderEvaluationCaseDTO through the video action-set RMDTO aggregate (see docs/architecture/provider-evaluation-rmdto.md), so exact-state correctness and policy-action correctness stay separate, closure-validated, queryable evidence rather than only living inside cases.jsonl.

Why this example exists

ZeroModel can compile a bounded policy into a deterministic artifact and execute that policy without invoking a model at decision time.

That still leaves an important upstream question:

How does a runtime observation become the stable state address required by the compiled policy?

There are several possible answers:

direct instrumentation;
deterministic visual protocols;
conventional computer vision;
learned visual models;
human or system-provided observations.

This example tests one of those paths:

A local vision-language model observes the frame, while ZeroModel retains authority over the downstream policy decision.

The experiment deliberately avoids turning ZeroModel into a general-purpose perception framework.

The local model remains a replaceable provider outside ZeroModel Core.

Architectural boundary

The central rule of the example is that perception and policy remain separate.

Local model responsibility

The local model receives:

one rendered frame;
a description of the visible objects;
the bounded observation vocabulary;
the required text response format.

It reports:

the player tank column;
whether a target is visible;
the target column when present;
whether the weapon is ready or blocked;
a self-reported confidence value.

ZeroModel responsibility

ZeroModel:

validates the model response;
rejects missing, contradictory, malformed, or out-of-range values;
constructs a stable policy row identifier;
addresses the independently compiled VPM policy;
selects the winning action;
records the observation, policy identity, action, and evaluation trace.

What the model does not receive

The local model does not receive:

the action values;
the policy table;
the action-selection rules;
the expected state;
the expected action;
the ground-truth row identifier.

This distinction matters. The Provider.predict boundary itself is declared as (image: bytes, render_mode: str) -> ProviderReply - there is no truth parameter to pass by mistake. tests/test_local_model_zero_arcade_provider_isolation.py proves this at runtime (a spy provider records every argument an end-to-end run actually passes to predict()), not just by inspecting the declared signature.

For example, the model may report:

TANK_COLUMN: 0
TARGET_COLUMN: 6
COOLDOWN: READY
CONFIDENCE: 95

The deterministic adapter converts that observation into:

tank=0|target=6|cooldown=0

The compiled ZeroModel policy then returns:

RIGHT

The model observed the scene.

ZeroModel selected the action.

Why the provider uses a text protocol

The first local-model experiments attempted to require schema-constrained JSON responses.

That path was unreliable with the tested Ollama and local-model combination. Some requests returned empty visible content even though the model had processed the image.

The working design uses a minimal labelled-text protocol:

TANK_COLUMN: <0-6>
TARGET_COLUMN: <NONE or 0-6>
COOLDOWN: <READY or BLOCKED>
CONFIDENCE: <0-100>

This keeps the model-facing interface simple while preserving a strict deterministic boundary.

probabilistic model output
        ↓
small human-readable protocol
        ↓
deterministic parser
        ↓
strict typed state

The parser may tolerate harmless formatting differences, but it does not tolerate ambiguous meaning.

Responses are rejected when they contain:

a missing required field;
conflicting values for one field;
a tank column outside 0..6;
a target column outside 0..6;
an invalid target-absence marker;
an unsupported cooldown value;
a confidence value outside the declared range;
content that cannot be mapped unambiguously into the policy state.

The model is not trusted merely because it produced syntactically valid text.

Arcade state space

The bounded arcade fixture contains seven horizontal columns.

Each policy state consists of:

tank column
target column or target absence
cooldown state

The complete state surface is:

7 tank columns
×
8 target states
×
2 cooldown states
=
112 policy rows

The eight target states are:

target absent
target in column 0
target in column 1
target in column 2
target in column 3
target in column 4
target in column 5
target in column 6

The two cooldown states are:

0 = ready
1 = blocked

The candidate actions are:

LEFT
RIGHT
STAY
FIRE

The policy behavior is bounded and explicit:

no target → STAY;
target aligned and weapon ready → FIRE;
target left of the tank → LEFT;
target right of the tank → RIGHT;
target aligned and weapon blocked → STAY.

These rules are compiled into the VPM policy before the visual model is invoked.

Files

Executable example:

examples/local_model_zero_arcade_test.py

Living example documentation:

docs/examples/local_model_zero_arcade_test.md

Recorded labelled smoke result:

docs/results/local-model-zero-arcade-smoke-v1/README.md

Local generated output:

local-results/<model>-zero-arcade-<timestamp>/
├── images/
├── cases.jsonl
├── summary.json
├── report.json          # only with --compile-report
└── run-manifest.json

The generated local-results/ directory is environment-specific evidence and should not normally be committed.

Persistence and reporting flags

--store {memory,sqlite}   default: memory
--sqlite-path <path>      required when --store sqlite
--compile-report          compile the run into a VPM report via the external adapter

--store memory (the default) keeps the run in an in-process InMemoryVideoActionSetStore for the duration of the script. --store sqlite --sqlite-path <path> persists identity, episode plan, observations, and the provider-evaluation run to a durable SQLite database through the existing zeromodel-sqlalchemy runtime composition. Either way, the script reloads the saved aggregate immediately after saving it and fails if the reload does not reconstruct byte-for-byte identically - a closure round-trip proof, not just a write.

--compile-report runs the aggregate through examples/provider_evaluation_report_adapter.py and writes report.json with the compiled report/adapted-report/VPM artifact ids; this import is lazy, so the default path has no dependency on zeromodel-artifacts.

A curated result may instead be preserved under docs/results/ with:

the exact command;
model identity;
model digest;
runtime version;
ZeroModel commit;
summarized metrics;
explicit claim boundaries.

Requirements

The example requires:

a local ZeroModel checkout;
the repository development dependencies;
Pillow;
Ollama;
a locally installed vision-capable model.

Install the repository dependencies:

python -m pip install -r requirements-dev.txt
python -m pip install pillow

Check the local Ollama runtime:

ollama --version
ollama list

Inspect the intended model:

ollama show qwen3.5:latest

The selected model must support image input.

The example is local-only and is not intended for GitHub Actions.

Verify the harness first

Before invoking a real local model, run the fake provider:

python .\examples\local_model_zero_arcade_test.py `
  --backend fake `
  --mode smoke `
  --render labelled

The fake provider is a wiring check.

It verifies:

frame generation;
state construction;
policy compilation;
row addressing;
action lookup;
result serialization;
summary generation.

Expected result:

attempted
accepted
exact states
correct actions
rejections

The fake provider does not perform perception and must not be cited as visual-model evidence.

Run the local-model smoke test

Use the exact model tag reported by ollama list.

Example:

python .\examples\local_model_zero_arcade_test.py `
  --backend ollama `
  --model qwen3.5:latest `
  --mode smoke `
  --render labelled `
  --confidence-threshold 0.0

The smoke suite contains eight representative states covering:

no visible target;
both cooldown states;
an aligned ready target;
an aligned blocked target;
targets to the left;
targets to the right;
extreme columns;
intermediate columns.

The confidence threshold is intentionally 0.0 for initial evaluation.

The model’s confidence value is self-reported and should not be treated as calibrated until a separate calibration experiment has been performed.

Labelled and unlabelled render modes

Labelled mode

--render labelled

The frame includes:

lane numbers 0..6;
a textual ready or blocked status;
visible lane boundaries;
strongly contrasted sprites.

This is the easiest supported perception condition.

It primarily tests whether:

the model receives the image correctly;
the text protocol works;
the parser works;
the VLM-to-policy boundary works;
the complete state can address the policy.

A successful labelled run is an integration result, not a robust-vision result.

Unlabelled mode

--render unlabelled

The frame retains the seven-lane geometry but removes the printed lane numbers and explanatory status text.

Run:

python .\examples\local_model_zero_arcade_test.py `
  --backend ollama `
  --model qwen3.5:latest `
  --mode smoke `
  --render unlabelled `
  --confidence-threshold 0.0

This is a more demanding perception test.

It asks whether the model can infer spatial lane positions rather than simply reading the printed labels.

The labelled and unlabelled results must be reported separately.

Complete canonical-state evaluation

After the smoke test behaves correctly, run the complete 112-state surface:

python .\examples\local_model_zero_arcade_test.py `
  --backend ollama `
  --model qwen3.5:latest `
  --mode all `
  --render labelled `
  --confidence-threshold 0.0

A complete run measures the model over every declared combination of:

tank column;
target absence or target column;
cooldown state.

This is substantially stronger evidence than the eight-state smoke suite.

The complete evaluation should report:

attempted observations;
accepted observations;
rejected observations;
exact-state accuracy;
action-equivalent accuracy;
tank-column accuracy;
target-presence accuracy;
target-column accuracy;
cooldown accuracy;
response parse failures;
confidence-threshold rejections;
latency distribution.

A wrong state can still produce the correct action.

For that reason, exact-state accuracy and action accuracy must remain separate measurements.

Output files

Each execution creates a timestamped local result directory.

`images/`

Contains the exact rendered frames supplied to the provider.

The image filenames encode the ground-truth state for local evaluation.

The model is not given the filename.

`cases.jsonl`

Contains one detailed record per observation.

Each record may include:

image path;
image digest;
ground-truth state;
raw provider response;
parsed observation;
confidence (provider_confidence_basis_points, the canonical 0..10000 integer, alongside a derived provider_confidence float in 0.0..1.0 recomputed from it - the float never participates in case identity);
predicted row identifier;
expected row identifier;
predicted action;
expected action;
exact-state result;
action-equivalent result;
provider metadata;
model metadata;
latency;
rejection reason.

This is the most important diagnostic output.

When a summary result is surprising, inspect cases.jsonl before drawing conclusions.

`summary.json`

Contains aggregate measurements for the run.

Typical fields include:

{
  "attempted": 8,
  "accepted": 8,
  "rejected": 0,
  "exact_state_correct": 8,
  "action_correct": 8,
  "action_equivalent_count": 0,
  "action_changing_count": 0,
  "run_id": "sha256:...",
  "factor_accuracy_over_accepted": {
    "tank_column": 1.0,
    "target_present": 1.0,
    "target_column": 1.0,
    "cooldown": 1.0
  }
}

action_equivalent_count and action_changing_count come straight from the persisted ProviderEvaluationSummaryDTO (see docs/architecture/provider-evaluation-rmdto.md): action_correct (exact_state_correct + action_equivalent_count) can exceed exact_state_correct whenever a wrong predicted state still lands on a row whose policy action matches the ground truth. A nonzero action_changing_count means at least one case crossed a policy boundary - the predicted action itself was wrong, which is a materially different failure than an action-equivalent miss. run_id addresses the full aggregate in the configured Store (--store sqlite makes this durable).

`run-manifest.json`

Records the execution arguments and policy artifact identity.

A curated result record should additionally preserve:

git rev-parse HEAD;
ollama --version;
model tag;
model digest;
model parameter count;
model quantization;
operating system;
Python version;
protocol version;
prompt identity when available.

Interpreting results

Exact-state accuracy

Exact-state accuracy requires every state factor to match:

tank column
target presence
target column
cooldown

A prediction is exact only when the generated policy row is identical to the ground-truth row.

Action-equivalent accuracy

Action accuracy measures whether the predicted state addresses a row whose winning action matches the action from the ground-truth state.

For example, several different states may all select LEFT.

Therefore:

correct action

does not necessarily imply:

correct perception

Both metrics are required.

Factor accuracy

Factor-level measurements identify where perception fails.

Examples:

high target-presence accuracy but poor target-column accuracy;
correct spatial positions but unreliable cooldown classification;
accurate labelled frames but poor unlabelled lane counting.

This is more informative than a single aggregate score.

Rejections

A rejection is not automatically a system failure.

The adapter should reject observations it cannot map safely into the bounded state vocabulary.

A useful provider may prefer:

correctly reject uncertain observations

over:

confidently address the wrong policy row

However, rejection behavior must be measured rather than assumed.

Confidence

The confidence field is generated by the local model.

It is not a probability produced by a calibrated classifier.

The persisted aggregate stores confidence as a canonical scaled integer, provider_confidence_basis_points (0..10000), not a float - this keeps the case’s content digest free of float non-determinism. cases.jsonl and summary.json also expose a derived provider_confidence float (0.0..1.0, recomputed from the integer) for convenience; that float never carries identity and must not be relied on for exact comparisons.

Do not interpret:

CONFIDENCE: 95

as a verified 95% likelihood of correctness.

Confidence becomes operationally useful only if evaluation shows that it separates correct from incorrect predictions on held-out canonical and perturbed observations.

Validated labelled smoke result

The first recorded local-model smoke experiment used:

Property	Recorded value
Model	`qwen3.5:latest`
Model size	9.7B parameters
Quantization	Q4_K_M
Runtime	Ollama
Render mode	Labelled
Cases	8
Accepted	8
Exact states	8
Correct actions	8
Rejections	0

The recorded run achieved:

exact-state accuracy: 100%
action accuracy:      100%
tank-column accuracy: 100%
target presence:      100%
target-column accuracy: 100%
cooldown accuracy:    100%

The evidence and exact environment identity are preserved at:

docs/results/local-model-zero-arcade-smoke-v1/README.md

The strongest supported statement from that result is:

A local Qwen 3.5 vision model correctly observed all eight labelled canonical arcade smoke-test frames, communicated the complete bounded state through a parsed text protocol, and enabled an independently compiled ZeroModel VPM policy to select the correct action in every case.

That claim is limited to the recorded fixture and environment.

What the example establishes

Within a successful bounded run, the example can establish that:

a local visual model received the rendered frame;
the model produced image-dependent observations;
the observations crossed a small declared protocol;
malformed responses could be rejected deterministically;
accepted observations could be converted into stable policy rows;
an independently identified VPM policy could select the action;
the observation provider and policy artifact remained separate;
the decision path could retain provider, state, policy, and action evidence.

The architectural result is:

learned perception
        ↓
validated bounded observation
        ↓
identified deterministic policy

This is the intended integration seam.

What the example does not establish

This example does not establish:

general image understanding;
open-world perception;
general visual intelligence;
reliable object detection outside this fixture;
robustness to arbitrary visual variation;
real-time gameplay;
production readiness;
safety-critical suitability;
calibrated confidence;
successful rejection of all unfamiliar frames;
performance across all 112 states unless the full run is executed;
unlabelled spatial reasoning unless the unlabelled run is executed;
successful game trajectories;
superiority over a conventional detector;
superiority over direct instrumentation;
superiority over asking the model to choose an action directly;
performance across other local models;
model-independent accuracy.

The canonical labelled frames are:

synthetic;
clean;
strongly contrasted;
spatially regular;
generated from the same bounded fixture;
labelled with the lane indices;
accompanied by explicit ready or blocked text.

Results must remain attached to the exact tested conditions.

Why this belongs in `examples/`

The local model provider requires:

an external local runtime;
a heavyweight model;
environment-specific hardware;
nontrivial inference latency;
model installation outside the ZeroModel packages.

It therefore does not belong in ZeroModel Core.

It also should not introduce an additional production package solely to host one experimental provider.

The example demonstrates how external learned perception can connect to ZeroModel without changing the core artifact contract.

external provider
        ↓
bounded adapter
        ↓
existing ZeroModel policy consumer

This keeps the provider replaceable and the policy independently identified.

Why this is not run in CI

The Ollama path is intentionally excluded from routine CI because it requires:

a locally running Ollama service;
a downloaded vision-language model;
several gigabytes of model storage;
environment-specific CPU or GPU resources;
seconds of inference time per frame;
model behavior that may vary across runtime and model versions.

The fake provider can verify the local harness wiring, but it is not visual evidence.

Curated external-model results belong under:

docs/results/

with complete environment and claim documentation.

Failure modes

Model ignores the image

Symptoms:

identical response for every frame;
model describes the filename rather than the image;
selected model does not advertise vision support;
predictions are unrelated to visible state.

Action:

inspect ollama show <model>;
verify that the model supports vision;
run two obviously different images through a standalone image test;
ensure the image bytes are included in the Ollama request;
do not interpret accidental action matches as perception success.

Empty response content

Symptoms:

Ollama completes the request;
visible response content is empty;
thinking tokens or schema constraints consume the response path.

Action:

disable thinking for the request;
avoid server-side schema-constrained JSON;
use the small text protocol;
preserve the raw response metadata for diagnosis.

Parse rejection

Symptoms:

model adds prose;
field names are missing;
multiple contradictory values appear;
a field is outside the bounded vocabulary.

Action:

inspect the raw response;
improve the prompt only when the problem is systematic;
do not silently guess missing values;
keep the parser fail-closed.

Correct action but incorrect state

Symptoms:

action_match is true;
exact_state_match is false.

Interpretation:

The model supplied the wrong state, but that state happened to select the same action.

This is not exact perception success.

High confidence and incorrect state

Interpretation:

The self-reported confidence is not calibrated.

Do not raise the confidence threshold until confidence behavior has been evaluated on both correct and incorrect cases.

First-call latency spike

The first request may include:

model loading;
vision encoder initialization;
memory allocation;
cache initialization.

Report cold and warmed latency separately when latency becomes part of the claim.

Recommended validation sequence

The example should be advanced through increasingly adversarial stages.

Stage 1 — labelled smoke

8 representative canonical states
printed lane labels
explicit cooldown text

Purpose:

validate transport;
validate parsing;
validate policy addressing;
validate end-to-end separation.

Stage 2 — unlabelled smoke

8 representative canonical states
no lane numbers
no explanatory cooldown text

Purpose:

test spatial lane inference;
reduce dependence on embedded text.

Stage 3 — complete canonical surface

all 112 declared states

Purpose:

measure the full bounded state space;
identify factor-specific confusion;
distinguish exact-state and action-equivalent accuracy.

Stage 4 — repeatability

Run each state multiple times with pinned settings.

Purpose:

detect nondeterministic response variation;
measure policy-address stability;
compare first-run and repeated-run behavior.

Stage 5 — controlled perturbations

Introduce one controlled variation at a time:

brightness;
contrast;
sprite colour;
scale;
translation;
blur;
compression;
background decoration;
partial occlusion.

Purpose:

identify the provider’s operating envelope;
measure degradation;
evaluate rejection behavior.

Stage 6 — trajectory control

Use the local provider to observe successive game states while the compiled policy selects each action.

Purpose:

measure error accumulation;
measure trajectory completion;
inspect how one wrong observation changes later states.

Stage 7 — direct-action comparison

Compare:

local VLM → predicted state → ZeroModel policy

against:

local VLM → direct action

Purpose:

determine whether separating perception from compiled policy improves:
- state traceability;
- action consistency;
- reproducibility;
- rejection behavior;
- policy replaceability;
- auditability.

Stage 8 — provider comparison

Run the same fixture through:

direct instrumentation;
exact deterministic visual reader;
local VLM provider;
conventional detector where available;
additional local vision models.

Purpose:

determine which provider is appropriate under which deployment assumptions.

Reporting new evidence

Each materially different experiment should receive a separate result record.

Examples:

docs/results/local-model-zero-arcade-unlabelled-smoke-v1/
docs/results/local-model-zero-arcade-canonical-112-v1/
docs/results/local-model-zero-arcade-perturbations-v1/
docs/results/local-model-zero-arcade-trajectory-v1/
docs/results/local-model-zero-arcade-direct-action-comparison-v1/

Do not overwrite the original labelled smoke result with stronger later experiments.

A controlled representation-only variant of Stage 5 (colour, shape, and lane intervention recipes compared while every other identity stays fixed) is implemented as a separate harness: docs/research/controlled-png-representation-benchmark.md (examples/arcade_png_representation_benchmark.py). It reuses this example’s fixture, renderer, response parser, and providers verbatim and adds only the PNG intervention recipes and the comparison/classification layer.

A real local run of that harness found labelled-v1 reaching 8/8 exact while unlabelled-v1 and every tested implicit (non-textual) intervention stayed at or below 4/8 (docs/results/controlled-png-representation-v1/). Because labelled-v1 bundles footer geometry, lane numerals, and cooldown text together, that result alone cannot say which component mattered. Stage 2F factors those three apart into footer-only-v1, lane-numerals-v1, cooldown-text-v1, and semantic-labelled-v1 - see docs/reviews/stage-2f-semantic-annotation-ablation.md for the design rationale and the “seven lane centres, not eight boundaries” lesson carried over from the lane-enhanced-v1 regression, and docs/research/controlled-png-representation-benchmark.md for the exact command to run it.

Each result record should contain:

research question;
exact execution date;
exact ZeroModel commit;
exact model tag and digest;
Ollama version;
operating system;
quantization;
command;
test surface;
aggregate metrics;
important per-case failures;
latency;
supported claim;
unsupported claims;
links to relevant source and documentation.

This creates an evidence ladder rather than a sequence of undocumented demos.

Design principle

The broader architectural principle demonstrated here is:

ZeroModel does not need to replace learned perception. It can govern how a probabilistic observation becomes a validated, typed, identity-bearing policy decision.

The provider may change.

The policy artifact may change.

The observation protocol may evolve.

The boundary remains explicit:

observe
    ↓
validate
    ↓
address
    ↓
act
    ↓
record

That boundary is the purpose of the example.

This site is open source. Improve this page.

zeromodel

Local Vision Model → ZeroModel Arcade Policy Bridge

Overview

Why this example exists

Architectural boundary

Local model responsibility

ZeroModel responsibility

What the model does not receive

Why the provider uses a text protocol

Arcade state space

Files

Persistence and reporting flags

Requirements

Verify the harness first

Run the local-model smoke test

Labelled and unlabelled render modes

Labelled mode

Unlabelled mode

Complete canonical-state evaluation

Output files

images/

cases.jsonl

summary.json

run-manifest.json

Interpreting results

Exact-state accuracy

Action-equivalent accuracy

Factor accuracy

Rejections

Confidence

Validated labelled smoke result

What the example establishes

What the example does not establish

Why this belongs in examples/

Why this is not run in CI

Failure modes

Model ignores the image

Empty response content

Parse rejection

Correct action but incorrect state

High confidence and incorrect state

First-call latency spike

Recommended validation sequence

Stage 1 — labelled smoke

Stage 2 — unlabelled smoke

Stage 3 — complete canonical surface

Stage 4 — repeatability

Stage 5 — controlled perturbations

Stage 6 — trajectory control

Stage 7 — direct-action comparison

Stage 8 — provider comparison

Reporting new evidence

Design principle

`images/`

`cases.jsonl`

`summary.json`

`run-manifest.json`

Why this belongs in `examples/`