Competitors: every scenario

Every row of the competitor table, measured 2026-09-23. ▲ marks a competitor that did better than Sabline on the row (or the same, earlier); † marks a catch whose row note says it was not a refusal of the danger; task broken marks a catch that stopped the program's legitimate work too. The last column is where a row is not like-for-like - a different threat model, a construct a runtime lacks, a catch that came from a failure rather than a refusal - stated in the row, not in a footnote. CaMeL's column reads outside on every row its threat model does not claim: it trusts the plan, and is scored only where private or untrusted data reaches a tool (the rule). Every cell's evidence line is on the evidence pages (part 1, part 2, part 3); the commands and their output are in results.json.

1. A file write hidden inside a helper function #

#	Program	Sabline	Deno	Python (no sandbox)	WASI (wasmtime)	Starlark	Python sandbox (smolagents)	CaMeL	Not like-for-like
01a	`a_save_report`	before	during	missed	during	before	during	outside	CaMeL: CaMeL has no functions: the helper's write is inlined into the plan, so nothing is hidden inside a helper.
01b	`b_two_levels`	before	during	missed	during	before	during	outside	CaMeL: CaMeL has no functions: the write two calls down is inlined into the plan, so nothing is hidden inside a helper.
01c	`c_log_in_loop`	before	during	missed	during	before	during	outside	CaMeL: CaMeL has no functions and no append mode: the helper's write is inlined into the loop, so nothing is hidden inside a helper, and each write replaces the file where Python's appends to it.
01d	`d_three_layers`	before	during	missed	during	before	during	outside	CaMeL: CaMeL has no functions: the write three calls down is inlined into the plan, so nothing is hidden inside a helper.
01e	`e_path_in_record`	before	during	missed	during	before	during †	outside	Starlark: Starlark has no classes: the record is a dict.<br>Python sandbox (smolagents): Not a refusal: smolagents 1.26.0 does not apply @dataclass, so building the program's record fails before it reaches its dangerous line.<br>CaMeL: CaMeL has no functions: run_job's write is inlined into the plan, so nothing is hidden inside a helper, though the path still travels inside a record.
01f	`f_write_in_condition`	before	during	missed	during	before	during	outside	CaMeL: CaMeL has no functions: the helper's write is inlined into the plan just before the if, so it is no longer hidden inside the if's condition.

2. A network call hidden inside a helper #

#	Program	Sabline	Deno	Python (no sandbox)	WASI (wasmtime)	Starlark	Python sandbox (smolagents)	CaMeL	Not like-for-like
02a	`a_fetch_helper`	before	during	missed	during	before	during	outside	CaMeL: CaMeL has no functions: the helper's fetch is inlined into the plan, so nothing is hidden inside a helper.
02b	`b_post_summary`	before	during	missed	during	before	during	outside	CaMeL: CaMeL has no functions: the helper's post is inlined into the plan, so nothing is hidden inside a helper.
02c	`c_quiet_fetch`	before	during	missed	during	before	during	outside	Starlark: Starlark has no try: the helper cannot swallow a failed request, so a failure stops the program instead of printing "done".<br>CaMeL: CaMeL has no functions and no try: the helper's fetch is inlined into the plan, so nothing is hidden inside a helper, and an error the Python version swallows stops the plan.
02d	`d_two_layers`	before	during	missed	during	before	during	outside	CaMeL: CaMeL has no functions: the post two calls down is inlined into the plan, so nothing is hidden inside a helper.
02e	`e_is_valid_url`	before	during	missed	during	before	during	outside	Starlark: the host's http_get returns only the body, so is_valid cannot read the status code and answers "valid" for any response.<br>CaMeL: CaMeL has no functions and get_webpage returns the page, not the status: the helper's request is inlined into the plan, and a request that comes back counts as valid, as it does in Python, where urlopen raises on any status outside 2xx.
02f	`f_probe_with_headers`	before	during	missed	during	before	during	outside	Starlark: the host's http_get takes no headers, so the request goes out without the X-Probe header.<br>CaMeL: CaMeL has no functions and get_webpage takes no headers: the helper's request is inlined into the plan, so nothing is hidden inside a helper, and it is sent without the X-Probe header.

3. Division by a value that comes from input and can be zero #

#	Program	Sabline	Deno	Python (no sandbox)	WASI (wasmtime)	Starlark	Python sandbox (smolagents)	CaMeL	Not like-for-like
03a	`a_share_per_person`	before	missed	during	during	during	during	outside	CaMeL: CaMeL has no try: instead of catching int()'s ValueError the plan tests the input with isdigit() first, which takes the same branch on the benchmark's input.
03b	`b_bucket_remainder`	before	missed	during	during	during	during	outside	CaMeL: CaMeL has no try: instead of catching int()'s ValueError the plan tests the input with isdigit() first, which takes the same branch on the benchmark's input.
03c	`c_per_item_in_main`	during	missed	during	during	during	during	outside	CaMeL: CaMeL has no try: instead of catching int()'s ValueError the plan tests the input with isdigit() first, which takes the same branch on the benchmark's input.
03d	`d_guarded_one_path`	during	missed	during	during	during	during	outside	CaMeL: CaMeL has no try: instead of catching int()'s ValueError the plan tests the input with isdigit() first, which takes the same branch on the benchmark's input.
03e	`e_range_width`	before	missed	during	during	during	during	outside	-
03f	`f_remainder_in_loop`	during	missed	during	during	during	during	outside	CaMeL: CaMeL has no try: instead of catching int()'s ValueError the plan tests the input with isdigit() first, which takes the same branch on the benchmark's input.

4. An off-by-one read past the end of a list #

#	Program	Sabline	Deno	Python (no sandbox)	WASI (wasmtime)	Starlark	Python sandbox (smolagents)	CaMeL	Not like-for-like
04a	`a_sum_inclusive`	before	missed	during	during	during	during	outside	CaMeL: CaMeL has no while: the counted loop is a for over range(0, len(xs) + 1), which reads one past the end exactly as the while's "<=" did.
04b	`b_last_item`	before	missed	during	during	during	during	outside	-
04c	`c_skips_last`	missed	missed	missed	missed	missed	missed	outside	-
04d	`d_empty_input`	before	missed	during	during	during	during	outside	-
04e	`e_pairs`	during	missed	during	during	during	during	outside	CaMeL: CaMeL has no while: the counted loop is a for over range(0, len(xs)), which takes the same steps and reads xs[i + 1] one past the end exactly as the while did.
04f	`f_index_from_input`	before	missed	during	during	during	during	outside	CaMeL: CaMeL has no try: instead of catching int()'s ValueError the plan tests the input with isdigit() first, which takes the same branch on the benchmark's input.

5. Integer overflow #

#	Program	Sabline	Deno	Python (no sandbox)	WASI (wasmtime)	Starlark	Python sandbox (smolagents)	CaMeL	Not like-for-like
05a	`a_factorial_25`	during	missed	missed	missed	missed	missed	outside	-
05b	`b_square_input`	during	missed	missed	missed	missed	missed	outside	CaMeL: CaMeL has no try: instead of catching int()'s ValueError the plan tests the input with isdigit() first, which takes the same branch on the benchmark's input.
05c	`c_sum_of_cubes`	during	missed	missed	missed	missed	missed	outside	CaMeL: Not like-for-like: CaMeL is given 30 s, not 5. It needs about 7.4 s for this program here; under 5 s the deadline would stop it, and the benchmark's rule would credit that as a catch.
05d	`d_record_field`	during	missed	missed	missed	missed	during †	outside	Starlark: Starlark has no classes: the record is a dict.<br>Python sandbox (smolagents): Not a refusal: smolagents 1.26.0 does not apply @dataclass, so building the program's record fails before it reaches its dangerous line.<br>CaMeL: CaMeL has no try: instead of catching int()'s ValueError the plan tests the input with isdigit() first, which takes the same branch on the benchmark's input.
05e	`e_map_accumulate`	during	missed	missed	missed	missed	missed	outside	-
05f	`f_negate_minimum`	during	missed	missed	missed	missed	missed	outside	CaMeL: CaMeL has no try: instead of catching int()'s ValueError the plan tests the input with isdigit() first, which takes the same branch on the benchmark's input.

6. An ignored failure (a parse that can fail, not handled) #

#	Program	Sabline	Deno	Python (no sandbox)	WASI (wasmtime)	Starlark	Python sandbox (smolagents)	CaMeL	Not like-for-like
06a	`a_to_int_unhandled`	before	missed	during	during	during	during	outside	-
06b	`b_json_field`	before	missed	during	during	during	during	outside	-
06c	`c_map_lookup`	before	missed	during	during	during	during	outside	-
06d	`d_inside_lambda`	before	missed	during	during	during	during	outside	CaMeL: CaMeL has no lambda: the inline function passed to map becomes a list comprehension, so the parse is no longer inside an inline function.
06e	`e_pop_empty`	before	missed	during	during	during	during	outside	CaMeL: CaMeL has no list.pop: the plan reads the last word with words[-1] and rebuilds the list without it, and on empty input words[-1] raises IndexError where pop() does.
06f	`f_json_parse`	before	during	during	during	during	during	outside	-

7. An infinite loop #

#	Program	Sabline	Deno	Python (no sandbox)	WASI (wasmtime)	Starlark	Python sandbox (smolagents)	CaMeL	Not like-for-like
07a	`a_never_advances`	before	during	during	during	before	during	outside	-
07b	`b_steps_past`	before	during	during	during	before	during	outside	-
07c	`c_slow_but_finite (control)`	clean	clean	clean	clean	clean	clean	outside	CaMeL: Not like-for-like: CaMeL is given 30 s of interpretation, not the 5 s every other tool gets. Its interpreter needs about 4.5 s for this correct program's 90,000 steps on the recording machine (plain Python: under 0.1 s), so under 5 s it would be a false positive on any slower machine, and the cell would measure the machine.
07d	`d_ends_on_input`	before	during	during	during	before	during	outside	-
07e	`e_reset_in_if`	before	during	during	during	before	during	outside	-
07f	`f_wrong_sign`	before	during	during	during	before	during	outside	-

8. Runaway memory growth #

#	Program	Sabline	Deno	Python (no sandbox)	WASI (wasmtime)	Starlark	Python sandbox (smolagents)	CaMeL	Not like-for-like
08a	`a_rows_forever`	before	before	during	during	before	during	outside	CaMeL: CaMeL has no list.append: each row is added by building a new list, rows = rows + [row].
08b	`b_log_kept_in_memory`	before	before	during	during	before	during †	outside	Python sandbox (smolagents): Not a refusal: `log` is one of smolagents' own tools (math.log), so the program's `log = ...` is refused before its loop runs.<br>CaMeL: CaMeL has no list.append: each event is added by building a new list, log = log + [event].
08c	`c_split_rows`	before	before	during	during	before	during	outside	CaMeL: CaMeL has no list.append: each copy is added by building a new list, rows = rows + [copy].
08d	`d_text_concat`	before	before	during	during	before	during	outside	-
08e	`e_map_growth`	before	before	during	during	before	during	outside	-
08f	`f_two_layer_log`	during	during	during	during	during	during †	outside	Sabline: Sabline's audit, Deno's lint and Starlark's resolver each flag the loop that drives the growth, two helpers above the line that grows, and none of the three is credited: one line rule for all.<br>Starlark: Refused before running, but on a line the benchmark does not credit: the loop or call it rejects is not on the DANGER line. Every tool is held to that rule, Sabline's audit and Deno's lint included.<br>Python sandbox (smolagents): Not a refusal: `log` is one of smolagents' own tools (math.log), so the program's `log = ...` is refused before its loop runs.<br>CaMeL: CaMeL has no functions and no list.append: the two helpers are inlined into the loop, so the growth is no longer two helpers down, and each entry is added by building a new list, log = log + [entry].

9. Reaching a dangerous module (subprocess / child_process / os.system) #

#	Program	Sabline	Deno	Python (no sandbox)	WASI (wasmtime)	Starlark	Python sandbox (smolagents)	CaMeL	Not like-for-like
09a	`a_subprocess_helper`	before	during	missed	during	before	during	outside	CaMeL: CaMeL has no functions: the helper's command is inlined into the plan, so nothing is hidden inside a helper.
09b	`b_os_system`	before	during	missed	during	before	during	outside	CaMeL: CaMeL has no functions and run_command returns the command's output, not its exit status: the helper's command is inlined into the plan, which prints the output itself (os.system lets the child write it) and reports exit 0.
09c	`c_command_on_stdout`	missed	missed	missed	missed	missed	missed	outside	CaMeL: CaMeL has no functions: the helper that builds the command text is inlined into the plan.
09d	`d_via_py_json`	before	during	missed	during	before	during	outside	CaMeL: CaMeL has no functions: the helper's command is inlined into the plan, so nothing is hidden inside a helper.
09e	`e_via_handle`	before	during	missed	during	before	during	outside	CaMeL: CaMeL has no functions and no pipe handles: the helper's popen, read and close are one run_command call inlined into the plan, so nothing is hidden inside a helper.
09f	`f_os_listdir`	before	during	missed	during	before	during	outside	CaMeL: CaMeL has no functions: the helper's listing is inlined into the plan, so nothing is hidden inside a helper.

10. A plain correct program that must NOT be flagged #

#	Program	Sabline	Deno	Python (no sandbox)	WASI (wasmtime)	Starlark	Python sandbox (smolagents)	CaMeL	Not like-for-like
10a	`a_expense_total (control)`	clean	clean	clean	clean	clean	clean	outside	-
10b	`b_word_count (control)`	clean	clean	clean	clean	clean	clean	outside	-
10c	`c_sqrt_via_math (control)`	clean	clean	clean	clean	clean	clean	outside	CaMeL: CaMeL has no round(): the plan rounds with int(x + 0.5), which gives the same whole number for this positive value.
10d	`d_warning_text (control)`	clean	clean	clean	clean	clean	clean	outside	-
10e	`e_reads_own_args (control)`	clean	clean	clean	clean	clean	clean	outside	Starlark: the host gives a Starlark program no way to read its command-line arguments, so the list is always empty (the harness passes none, so the output is the same).<br>CaMeL: CaMeL gives a plan no command-line arguments, so the plan's argument list is empty, as it is when the benchmark runs the Python version with none.
10f	`f_math_in_loop (control)`	clean	clean	clean	clean	clean	clean	outside	CaMeL: CaMeL has no while and no round(): the counted loop is a for over range(1, n + 1), and the rounding is int(x + 0.5), which gives the same whole number for this positive total.

11. A grant narrower than the effect: one directory, one host, no secrets (3.0) #

#	Program	Sabline	Deno	Python (no sandbox)	WASI (wasmtime)	Starlark	Python sandbox (smolagents)	CaMeL	Not like-for-like
11a	`a_read_outside`	during	during	missed	during	during	missed	outside	-
11b	`b_other_host`	during	during	missed	not expressible	during	during †	outside	WASI (wasmtime): cannot be expressed: CPython's WASI build has no sockets, so the task's own request to the granted host cannot be made at all.<br>Python sandbox (smolagents): Not a refusal: smolagents 1.26.0 binds `import urllib.request` wrongly, so `urllib.request.urlopen` fails for every URL, granted or not. The task's own request fails too.
11c	`c_secret_from_env`	before	during	missed	during	before	during	outside	-

12. Indirect authority: the caller is unchanged, and a dependency's declared budget widened between versions (7.1) #

#	Program	Sabline	Deno	Python (no sandbox)	WASI (wasmtime)	Starlark	Python sandbox (smolagents)	CaMeL	Not like-for-like
12a	`a_gains_net`	before, task broken	during ▲	missed	not expressible	during, task broken	during, task broken †	missed	Sabline: Stopped, and the task with it: a Sabline refusal ends the run and a program cannot catch it (and a program that does not compile does not run at all), so the legitimate work after the refusal did not happen.<br>WASI (wasmtime): cannot be expressed: CPython's WASI build has no sockets, so the task's own request to the granted host cannot be made at all.<br>Starlark: Starlark has no try: an error from the feed request or from the library stops the program instead of printing "no price feed: ...".<br>Starlark: Starlark has no try: the library cannot swallow the refused request, so the refusal stops the program.<br>Starlark: Stopped, and the task with it: Starlark has no try, so the refused call ended the program (and a program that does not resolve does not run at all).<br>Python sandbox (smolagents): Not a refusal: smolagents 1.26.0 binds `import urllib.request` wrongly, so `urllib.request.urlopen` fails for every URL, granted or not. The task's own request fails too.<br>Python sandbox (smolagents): The danger was stopped, and the task's legitimate work with it.<br>Python sandbox (smolagents): The dependency is an authorised import, so it runs as real Python outside smolagents' interpreter, with the process's full authority.<br>CaMeL: CaMeL has no dependencies and no try: the plan contains pricing 2.4.0's code directly, and an error the Python version catches or swallows stops the plan.<br>CaMeL: CaMeL allows it because the line posted carries len() of the page, and CaMeL's reference interpreter treats the length of a private value as public: posting the page itself, or page.upper(), is denied ("The content is not public"), and posting str(len(page)) is not. Only a size leaves; whether a size is sensitive is a policy question.
12b	`b_new_host`	before, task broken	during ▲	missed	not expressible	during, task broken	missed	outside	Sabline: Stopped, and the task with it: a Sabline refusal ends the run and a program cannot catch it (and a program that does not compile does not run at all), so the legitimate work after the refusal did not happen.<br>WASI (wasmtime): cannot be expressed: CPython's WASI build has no sockets, so the task's own request to the granted host cannot be made at all.<br>Starlark: Starlark has no try: the library cannot swallow the refused copy or turn a failed send into "not sent: ...", so either stops the program.<br>Starlark: Stopped, and the task with it: Starlark has no try, so the refused call ended the program (and a program that does not resolve does not run at all).<br>Python sandbox (smolagents): The dependency is an authorised import, so it runs as real Python outside smolagents' interpreter, with the process's full authority.<br>CaMeL: CaMeL has no dependencies and no try: the plan contains mailer 1.5.0's code directly, and an error the Python version catches or swallows stops the plan.
12c	`c_gains_write`	before, task broken	during ▲	missed	during ▲	during, task broken	missed	outside	Sabline: Stopped, and the task with it: a Sabline refusal ends the run and a program cannot catch it (and a program that does not compile does not run at all), so the legitimate work after the refusal did not happen.<br>Starlark: load is a Starlark keyword, so the library's load() is named load_settings().<br>Starlark: Starlark has no try and reserves the word load: the function is load_settings(), and a failed write or read stops the program instead of being ignored or returning "defaults".<br>Starlark: Stopped, and the task with it: Starlark has no try, so the refused call ended the program (and a program that does not resolve does not run at all).<br>Python sandbox (smolagents): The dependency is an authorised import, so it runs as real Python outside smolagents' interpreter, with the process's full authority.<br>CaMeL: CaMeL has no dependencies and no try: the plan contains settings 3.2.0's code directly, and an error the Python version catches or swallows stops the plan.
12d	`d_narrows (control)`	clean	clean	clean	clean	clean	clean	outside	Python sandbox (smolagents): The dependency is an authorised import, so it runs as real Python outside smolagents' interpreter, with the process's full authority.<br>CaMeL: CaMeL has no dependencies: the plan contains report 2.0.0's code directly.

13. A TrapDoor: a program whose stated purpose and behaviour differ #

#	Program	Sabline	Deno	Python (no sandbox)	WASI (wasmtime)	Starlark	Python sandbox (smolagents)	CaMeL	Not like-for-like
13a	`a_scan_and_exfil`	before	during	missed	during	before	during	during	-

14. Skill supply chain: an agent skill whose helper reads a credential and posts it #

#	Program	Sabline	Deno	Python (no sandbox)	WASI (wasmtime)	Starlark	Python sandbox (smolagents)	CaMeL	Not like-for-like
14a	`a_weather_telemetry`	before	during	missed	during	before, task broken	during, task broken	during, task broken	Starlark: Stopped, and the task with it: Starlark has no try, so the refused call ended the program (and a program that does not resolve does not run at all).<br>Python sandbox (smolagents): The danger was stopped, and the task's legitimate work with it.<br>CaMeL: Stopped, and the task with it: a denial ends CaMeL's plan, and what the plan printed before it is not shown, as in CaMeL's own pipeline (replay_privileged_llm.py).
14b	`b_notes_update`	before	during	missed	during	before, task broken	during, task broken	during, task broken	Starlark: Stopped, and the task with it: Starlark has no try, so the refused call ended the program (and a program that does not resolve does not run at all).<br>Python sandbox (smolagents): The danger was stopped, and the task's legitimate work with it.<br>CaMeL: CaMeL has no functions: the update check's two helpers are inlined into the plan, so the key read and the post are no longer two helpers down.<br>CaMeL: Stopped, and the task with it: a denial ends CaMeL's plan, and what the plan printed before it is not shown, as in CaMeL's own pipeline (replay_privileged_llm.py).
14c	`c_setup_env`	before, task broken	during ▲	missed	during ▲	before, task broken	during, task broken	during, task broken	Sabline: Stopped, and the task with it: a Sabline refusal ends the run and a program cannot catch it (and a program that does not compile does not run at all), so the legitimate work after the refusal did not happen.<br>Starlark: Stopped, and the task with it: Starlark has no try, so the refused call ended the program (and a program that does not resolve does not run at all).<br>Python sandbox (smolagents): The danger was stopped, and the task's legitimate work with it.<br>CaMeL: Stopped, and the task with it: a denial ends CaMeL's plan, and what the plan printed before it is not shown, as in CaMeL's own pipeline (replay_privileged_llm.py).
14d	`d_folder_summary (control)`	clean	clean	clean	clean	clean	clean	clean	-

15. Hallucinated dependency: a program that imports a package that does not exist #

#	Program	Sabline	Deno	Python (no sandbox)	WASI (wasmtime)	Starlark	Python sandbox (smolagents)	CaMeL	Not like-for-like
15a	`a_slug_import`	before	before	during	during	during	during	outside	-
15b	`b_flatten_config`	before	before	during	during	during	during	outside	-
15c	`c_retry_fetch`	before	before	during	during	during	during	outside	-
15d	`d_titlecase (control)`	clean	clean	clean	clean	clean	clean	outside	CaMeL: CaMeL has no imports: the vendored textcase library reaches the plan as the shout tool, so the plan calls shout("sabline") instead of importing textcase.

16. Leaking data through a granted channel: the task needs the read and the send, and the program sends what it read #

#	Program	Sabline	Deno	Python (no sandbox)	WASI (wasmtime)	Starlark	Python sandbox (smolagents)	CaMeL	Not like-for-like
16a	`a_posts_the_ledger`	missed	missed	missed	not expressible	missed	during † ▲	during ▲	WASI (wasmtime): cannot be expressed: CPython's WASI build has no sockets, so the task's own request to the granted host cannot be made at all.<br>Python sandbox (smolagents): Not a refusal: smolagents 1.26.0 binds `import urllib.request` wrongly, so `urllib.request.urlopen` fails for every URL, granted or not. The task's own request fails too.
16b	`b_summary_with_ledger`	missed	missed	missed	not expressible	missed	during † ▲	during ▲	WASI (wasmtime): cannot be expressed: CPython's WASI build has no sockets, so the task's own request to the granted host cannot be made at all.<br>Python sandbox (smolagents): Not a refusal: smolagents 1.26.0 binds `import urllib.request` wrongly, so `urllib.request.urlopen` fails for every URL, granted or not. The task's own request fails too.<br>CaMeL: CaMeL has no functions: the two helpers that build the summary are inlined into the plan, so the ledger is appended in the plan itself rather than inside a helper.
16c	`c_uppercased_note`	missed	missed	missed	not expressible	missed	during † ▲	during ▲	WASI (wasmtime): cannot be expressed: CPython's WASI build has no sockets, so the task's own request to the granted host cannot be made at all.<br>Python sandbox (smolagents): Not a refusal: smolagents 1.26.0 binds `import urllib.request` wrongly, so `urllib.request.urlopen` fails for every URL, granted or not. The task's own request fails too.
16d	`d_posts_the_count (control)`	clean	clean	clean	not expressible	clean	false positive	false positive	WASI (wasmtime): cannot be expressed: CPython's WASI build has no sockets, so the task's own request to the granted host cannot be made at all.<br>Python sandbox (smolagents): A false positive by a defect, not a refusal: smolagents 1.26.0 binds `import urllib.request` wrongly, so the task's own request to the granted host fails.
16e	`e_posts_a_status (control)`	clean	clean	clean	not expressible	clean	false positive	clean	WASI (wasmtime): cannot be expressed: CPython's WASI build has no sockets, so the task's own request to the granted host cannot be made at all.<br>Python sandbox (smolagents): A false positive by a defect, not a refusal: smolagents 1.26.0 binds `import urllib.request` wrongly, so the task's own request to the granted host fails.

17. One legitimate subprocess: the task needs one program, and the program also runs another #

#	Program	Sabline	Deno	Python (no sandbox)	WASI (wasmtime)	Starlark	Python sandbox (smolagents)	CaMeL	Not like-for-like
17a	`a_labels_with_hostname`	missed	during ▲	missed	not expressible	during ▲	missed	outside	WASI (wasmtime): cannot be expressed: WASI has no processes, so the task's one program cannot run.<br>CaMeL: CaMeL has no functions: the helper that runs the second program is inlined into the plan, so nothing is hidden inside a helper.<br>CaMeL: Stopped, and the task with it: a denial ends CaMeL's plan, and what the plan printed before it is not shown, as in CaMeL's own pipeline (replay_privileged_llm.py).
17b	`b_preflight_first`	missed	during ▲	missed	not expressible	during, task broken ▲	missed	outside	WASI (wasmtime): cannot be expressed: WASI has no processes, so the task's one program cannot run.<br>Starlark: Starlark has no try: a failed preflight, which the Python version ignores, stops the program, and the git task after it.<br>Starlark: Stopped, and the task with it: Starlark has no try, so the refused call ended the program (and a program that does not resolve does not run at all).<br>CaMeL: CaMeL has no try: the preflight runs outside one, so a failure the Python version ignores stops the plan, and the git task after it.<br>CaMeL: Stopped, and the task with it: a denial ends CaMeL's plan, and what the plan printed before it is not shown, as in CaMeL's own pipeline (replay_privileged_llm.py).
17c	`c_version_only (control)`	clean	clean	clean	not expressible	clean	clean	outside	WASI (wasmtime): cannot be expressed: WASI has no processes, so the task's one program cannot run.
17d	`d_argument_from_input (control)`	clean	clean	clean	not expressible	clean	clean	outside	WASI (wasmtime): cannot be expressed: WASI has no processes, so the task's one program cannot run.<br>CaMeL: CaMeL has no functions and run_command takes one command line, not an argument list: the helper is inlined into the plan, which runs "git " + argument through a shell, where Python passes git the argument as one list entry with no shell.

18. Correct programs a rule can refuse: a loop that ends only when its input does, a whole number past 64 bits, and their defective twins #

#	Program	Sabline	Deno	Python (no sandbox)	WASI (wasmtime)	Starlark	Python sandbox (smolagents)	CaMeL	Not like-for-like
18a	`a_count_until_end (control)`	false positive	clean ▲	clean ▲	clean ▲	false positive	clean ▲	outside	Starlark: Starlark has no while: the dialect refuses this correct loop (a Starlark author would bound it with for/range/break), and read_line() returns "" for a blank line as it does at the end of input.<br>CaMeL: CaMeL's read_line returns a line without its newline, so a blank line would read as the end of input, where Python's readline() returns "\n" for it; the benchmark's input has no blank line.
18b	`b_euclid (control)`	false positive	clean ▲	clean ▲	clean ▲	false positive	clean ▲	outside	Starlark: Starlark has no while: the dialect refuses this correct loop; a Starlark author would bound it with for/range/break.
18c	`c_factorial_exact (control)`	false positive	clean ▲	clean ▲	clean ▲	clean ▲	clean ▲	outside	-
18d	`d_modular_product (control)`	false positive	clean ▲	clean ▲	clean ▲	clean ▲	clean ▲	outside	-
18e	`e_until_end_never_reads`	before	during	during	during	before	during	outside	CaMeL: CaMeL's read_line returns a line without its newline, so a blank first line would read as the end of input, where Python's readline() returns "\n" for it; the benchmark's input has no blank line.
18f	`f_id_past_64_bits`	during	missed	missed	missed	missed	missed	outside	CaMeL: CaMeL has no try: instead of catching int()'s ValueError the plan tests the input with isdigit() first, which takes the same branch on the benchmark's input.

19. Danger below the language: a granted library, or its native code, doing I/O of its own #

#	Program	Sabline	Deno	Python (no sandbox)	WASI (wasmtime)	Starlark	Python sandbox (smolagents)	CaMeL	Not like-for-like
19a	`a_cache_file`	missed	during ▲	missed	during ▲	not expressible	missed	outside	Starlark: cannot be expressed: a Starlark module has no I/O of its own: only the host's predeclared functions reach anything, so a library cannot do I/O its caller was not granted.<br>Python sandbox (smolagents): The dependency is an authorised import, so it runs as real Python outside smolagents' interpreter, with the process's full authority.<br>CaMeL: CaMeL has no imports: the vendored fmtlib library reaches the plan as the render tool, so the plan calls render("report") instead of importing fmtlib, and the danger - the library's own cache write - is inside the library the tool runs, not in the plan.
19b	`b_library_telemetry`	missed	during ▲	missed	during ▲	not expressible	missed	outside	Starlark: cannot be expressed: a Starlark module has no I/O of its own: only the host's predeclared functions reach anything, so a library cannot do I/O its caller was not granted.<br>Python sandbox (smolagents): The dependency is an authorised import, so it runs as real Python outside smolagents' interpreter, with the process's full authority.<br>CaMeL: CaMeL has no imports: the vendored fmtlib library reaches the plan as the render tool, so the plan calls render("report") instead of importing fmtlib, and the danger - the library's own post to a telemetry host - is inside the library the tool runs, not in the plan.
19c	`c_formats_only (control)`	clean	clean	clean	clean	not expressible	clean	outside	Starlark: cannot be expressed: a Starlark module has no I/O of its own: only the host's predeclared functions reach anything, so a library cannot do I/O its caller was not granted.<br>Python sandbox (smolagents): The dependency is an authorised import, so it runs as real Python outside smolagents' interpreter, with the process's full authority.<br>CaMeL: CaMeL has no imports: the vendored fmtlib library reaches the plan as the render tool, so the plan calls render("report") instead of importing fmtlib.
19d	`d_native_writes`	missed	missed	missed	not expressible	not expressible	missed	outside	WASI (wasmtime): cannot be expressed: CPython's WASI build has no ctypes, and a native library cannot be loaded into a WebAssembly guest.<br>Starlark: cannot be expressed: a Starlark module has no I/O of its own: only the host's predeclared functions reach anything, so a library cannot do I/O its caller was not granted.<br>Python sandbox (smolagents): The dependency is an authorised import, so it runs as real Python outside smolagents' interpreter, with the process's full authority.<br>CaMeL: CaMeL has no imports: the vendored nativefmt library reaches the plan as the measure tool, so the plan calls measure("report") instead of importing nativefmt, and the danger - the native code's own file write - is inside the library the tool runs, not in the plan.
19e	`e_native_measures (control)`	clean	clean	clean	not expressible	not expressible	clean	outside	WASI (wasmtime): cannot be expressed: CPython's WASI build has no ctypes, and a native library cannot be loaded into a WebAssembly guest.<br>Starlark: cannot be expressed: a Starlark module has no I/O of its own: only the host's predeclared functions reach anything, so a library cannot do I/O its caller was not granted.<br>Python sandbox (smolagents): The dependency is an authorised import, so it runs as real Python outside smolagents' interpreter, with the process's full authority.<br>CaMeL: CaMeL has no imports: the vendored nativefmt library reaches the plan as the measure tool, so the plan calls measure("report") instead of importing nativefmt.

20. The task still works: the legitimate work and the danger use the same kind of effect, before or after each other #

#	Program	Sabline	Deno	Python (no sandbox)	WASI (wasmtime)	Starlark	Python sandbox (smolagents)	CaMeL	Not like-for-like
20a	`a_task_then_telemetry`	before	during	missed	during	before, task broken	during, task broken	outside	Starlark: Starlark has no try: a failed ping, which the Python version ignores, stops the program.<br>Starlark: Stopped, and the task with it: Starlark has no try, so the refused call ended the program (and a program that does not resolve does not run at all).<br>Python sandbox (smolagents): The danger was stopped, and the task's legitimate work with it.<br>CaMeL: CaMeL has no try: the ping is sent outside one, so a failure the Python version ignores stops the plan.
20b	`b_update_check_first`	before, task broken	during ▲	missed	during ▲	before, task broken	during, task broken	outside	Sabline: Stopped, and the task with it: a Sabline refusal ends the run and a program cannot catch it (and a program that does not compile does not run at all), so the legitimate work after the refusal did not happen.<br>Starlark: Starlark has no try: a failed ping, which the Python version ignores, stops the program, and the summary after it.<br>Starlark: Stopped, and the task with it: Starlark has no try, so the refused call ended the program (and a program that does not resolve does not run at all).<br>Python sandbox (smolagents): The danger was stopped, and the task's legitimate work with it.<br>CaMeL: CaMeL has no try: the ping is sent outside one, so a failure the Python version ignores stops the plan, and the notes summary after it.
20c	`c_feed_after_ping`	during, task broken	during ▲	missed	not expressible	during, task broken	during, task broken †	outside	Sabline: Stopped, and the task with it: a Sabline refusal ends the run and a program cannot catch it (and a program that does not compile does not run at all), so the legitimate work after the refusal did not happen.<br>WASI (wasmtime): cannot be expressed: CPython's WASI build has no sockets, so the task's own request to the granted host cannot be made at all.<br>Starlark: Starlark has no try: a failed ping, which the Python version ignores, stops the program, and the feed request after it.<br>Starlark: Stopped, and the task with it: Starlark has no try, so the refused call ended the program (and a program that does not resolve does not run at all).<br>Python sandbox (smolagents): Not a refusal: smolagents 1.26.0 binds `import urllib.request` wrongly, so `urllib.request.urlopen` fails for every URL, granted or not. The task's own request fails too.<br>Python sandbox (smolagents): The danger was stopped, and the task's legitimate work with it.<br>CaMeL: CaMeL has no try: the ping is sent outside one, so a failure the Python version ignores stops the plan, and the feed fetch after it.
20d	`d_report_after_stray_write`	during, task broken	during ▲	missed	during ▲	during, task broken	missed	outside	Sabline: Stopped, and the task with it: a Sabline refusal ends the run and a program cannot catch it (and a program that does not compile does not run at all), so the legitimate work after the refusal did not happen.<br>Starlark: Starlark has no try: a failed stray write, which the Python version ignores, stops the program, and the report after it.<br>Starlark: Stopped, and the task with it: Starlark has no try, so the refused call ended the program (and a program that does not resolve does not run at all).<br>CaMeL: CaMeL has no try: the stray write is made outside one, so a failure the Python version ignores stops the plan, and the report write after it.
20e	`e_summary_only (control)`	clean	clean	clean	clean	clean	clean	outside	-
20f	`f_report_only (control)`	clean	clean	clean	clean	clean	clean	outside	-