We Sandboxed Python Inside Splunk

Splunk's search language is excellent until the moment it is not. The moment arrives differently for everyone: a Shannon entropy calculation, a JWT to decode, a proper statistical test, an API that has to be called mid-search. SPL can be bent towards some of these, and the bent versions become the haunted corridors of your detection estate -- five nested evals that one person understood, three years ago.

The sanctioned escape hatches are all worse than they look. A custom search command is a Python process spawned per search with the full privileges of the Splunk host: filesystem, network, subprocesses, everything, plus a per-search interpreter start you pay every time. A SOAR platform moves the problem to a second product, which means moving the data to a second product. And both share a property that should bother security teams more than it seems to: the code runs unsandboxed, on infrastructure that sits at the centre of your security monitoring.

So we built the thing we wanted: real Python, in the search bar, inside a WebAssembly sandbox, fast enough that you stop thinking about the overhead.

index=web | exec inline="
import math, collections
for e in events:
    counts = collections.Counter(e['domain'])
    total = len(e['domain'])
    e['entropy'] = -sum(c/total * math.log2(c/total) for c in counts.values())
"

CPython as a WebAssembly component

The interpreter itself is CPython 3.13, compiled to a WebAssembly component with componentize-py and run under a pinned build of Wasmtime. That one sentence hides most of a year of edges, but the shape is simple: the component exports three functions -- begin-run, execute-batch, end-run -- and Splunk's chunked external-command protocol is spoken by a small Rust binary that feeds batches across the boundary as JSON.

Compiling CPython to a component is the easy half. Making it fast enough to sit in a search pipeline is the interesting half, because a naive implementation pays interpreter startup -- imports, heap setup, the lot -- on every search, and that is tens of milliseconds before your first event moves.

The fix is aggressive reuse at every layer:

• Precompilation. The component is compiled to native code once, at packaging time. Loading it at runtime is a memory map, not a compile.
• Pre-initialisation. The Python heap is snapshotted after interpreter setup, so an instance starts with the standard library already imported.
• A warm instance pool. A persistent daemon keeps booted interpreters and hands them out per search. In steady state, "start Python" is "take an instance off a free list". Copy-on-write memory images make the cold path cheap too.
• Compile-once snippets. Your inline source is parsed once per search, not once per batch.

Pooling forces a discipline that turns out to be a feature: if a guest traps, its instance is discarded rather than returned to the pool, because its globals could be in any state. Per-run state lives in begin-run/end-run; instance state -- compiled regexes, parsed lookup tables -- survives across searches and is exactly where the performance lives.

The numbers, with the caveats attached

These were measured this week on our 16-CPU development machine, as medians with 10th/90th percentiles over repeated samples. The harness runs a null control first -- the same code measured as both sides of an A/B comparison -- and on this machine, in this state, that control says differences under about 8% are measurement error. Your hardware will produce different numbers, which is why the benchmark harness ships with the product (make bench) rather than a marketing page asking to be believed.

	inline Python	compiled .wasm
per-search overhead, warm instance	58 µs	3.9 µs
per-search overhead, cold instance	1.9 ms	141 µs
throughput at batch 32	1.1M events/s	2.4M events/s

Daemon startup -- paid once per host, not per search -- is about 83 ms including mapping the precompiled interpreter. Hot-swapping a changed .wasm function takes about 21 ms, guarded by a content hash so editors that rewrite files harmlessly are harmless. Searches already running finish on the old code; new searches get the new code.

The sandbox is the point

The performance work makes the product usable. The sandbox is why it should exist. Guest code -- ours and yours -- gets:

• no sockets,
• no subprocesses,
• no environment variables,
• no filesystem beyond explicitly declared mounts,
• a memory ceiling and a hard execution deadline.

Outbound HTTP exists, because enrichment and response actions need it, but it is not a socket handed to the guest. It is wasi:http, serviced by the host, behind a deny-by-default allow-list: per-integration host patterns, port and path rules, IP-range rules with non-global addresses refused by default, redirect and DNS-rebinding coverage. There is no forward proxy to deploy, because the policy is enforced in-process, in the one place all requests already pass through.

The part we think matters most: secrets are injected by the host, after the ACL check. Playbook code says "call the Okta API"; the host attaches the credential to the outbound request. The code that runs in the sandbox never contains and can never read the secret it is using -- so a compromised dependency in your enrichment function leaks nothing, and can talk to nothing except the hosts its integration declared. Ask your current automation platform what happens in that scenario. The honest answer is "the attacker gets the credentials and the network access of the automation host", and for most platforms that host is domain-joined with admin API tokens for your EDR.

Fairness comes from epoch interruption: a guest in a tight loop is preempted periodically so concurrent searches stay genuinely concurrent, and an infinite loop cannot starve the timer that exists to interrupt it.

The parts that fought back

Splunk's chunked protocol is documented, mostly. What is not documented is that splunkd tags every row it sends an external command with a hidden _chunked_idx column and requires the values it gets back to be in range and non-decreasing, because it uses them to merge your output back into its pipeline. Pass that column through to user code -- as user data, which is what it looks like -- and the first customer who writes sorted(events, ...) gets a protocol error at row 36 and no clue why. We found the answer by running strings on the splunkd binary. That one got its own post.

Beyond that: componentize-py's pre-init snapshot means importlib.invalidate_caches() walks into modules that do not exist in the snapshot, which we assert on so the day it starts working, we find out. Instance reuse means sys.modules is a cache shared across searches, which is correct for speed and wrong for an editor, so saves evict deliberately. And Preview 1 modules are refused outright rather than adapted, because carrying an adapter in every pooled instance is a cost every customer pays for a compatibility surface almost none of them want.

What it is not

It is not open source; it is a commercial product with a published price, which is rarer in this market than it should be. It runs on Splunk Enterprise (Linux x86-64), not Splunk Cloud -- a persistent daemon does not fit Cloud's execution model today. It is not a case-management SOAR and does not want to be; if your bottleneck is ticketing workflow, buy the platform. And when a licence expires, searches keep running and nothing is deleted -- outbound network access from your code is denied, and that is the whole of the degradation, in the contract, in writing.

Every install is the full product for 90 days. No signup, because there is no telemetry to tie a signup to. The documentation is public and does not gate on an email address either.