← Back to portfolio   |   View public optimizer subset on GitHub

Overview

Strategy Assayer is an ML evaluation and provenance framework. Search finds candidates. Evaluation decides whether they are real. A result that only looks good because many variants were tried does not clear the gates: walk-forward holdout, randomized control arms, null distributions, multiple-trial correction, blocking promotion rules, and hash-stamped run records (plan hash, code SHA, engine version).

Underneath that is a parallel experiment orchestration engine: compile once, warm worker pools, a persistent harness, and directed search (genetic, Latin Hypercube, Bayesian/Optuna TPE, local grid). The expensive tests are routine because the engine makes them cheap. The evaluation workload is QuantConnect LEAN. The public lean-optimizer-public repository exposes the orchestration-engine slice; the private platform adds the evaluation and promotion pipeline.

Measured Optimizer Performance

Public optimizer benchmark against the engine CLI's built-in exhaustive grid search on a 7-parameter space (679,140 grid points), 15 parallel workers, same workload, same data:

Exhaustive grid (CLI baseline) lean-optimizer (GA)
Search strategy Exhaustive grid (all 679,140 points) Incremental genetic algorithm
Evaluations 679,140 611
Throughput (15 workers) 2.15 evals/sec 3.82 evals/sec
Wall time ~88 hours (extrapolated from 1,772 evals in 825s) 2 min 40 sec
Search efficiency 1x 1,112x fewer evaluations

The two advantages are independent and compound: the persistent harness yields ~1.8x higher throughput per worker, and the GA's directed search converges in 1,112x fewer evaluations. Combined: what takes the CLI grid search an estimated ~88 hours completes in under 3 minutes. I do not claim a measured multiplier for the full private walk-forward evaluation pipeline; that is described qualitatively as a move from one-off runs to an automated credibility pipeline.

Evaluation and Promotion Pipeline

The private platform spans the full experiment-to-promotion loop:

  1. Data ingestion and backfill
  2. Workload packaging and experiment ports
  3. Parameter-space definition through YAML experiment plans
  4. GA, grid, and Bayesian parameter search
  5. Out-of-sample evaluation on held-out windows
  6. Walk-forward credibility scoring
  7. Graduated verdicts with absolute floors
  8. Composition and sizing
  9. Forward prediction bands and kill-lines
  10. Parameter-staleness monitoring
  11. Dashboard review and manually approved promotion packages
  12. Post-promotion monitoring from the fill ledger
Strategy Assayer test plan showing search space, a benchmark basket, and candidate equity curves
An experiment plan is a named run: search space, the benchmark basket, and the equity overlay of the candidates against buy-and-hold.

Validation Discipline

The platform is intentionally conservative. It has scored 11 experiment families across 350+ evaluation tasks with full credibility verdicts, drawing on a corpus of more than 25,000 experiment plans, 86,000 walk-forward windows, and 5.2 million individual evaluation runs. Verdicts are graduated quality scores (0–10, banded from strong to poor) with absolute floors: a candidate that breaches the floors is blocked outright, regardless of how well it scores elsewhere. Of 1,693 scored runs, 435 were blocked. Promotion is the exception; preventing attractive but fragile evaluations from shipping is the system's purpose.

Credibility table for gem-rotational with graduated verdicts and out-of-sample metrics
Walk-forward credibility for one strategy family: graduated verdicts, out-of-sample windows, and baseline-relative Sharpe and Calmar.

Scoring is walk-forward and baseline-relative. Every candidate is judged on stitched out-of-sample performance from a walk-forward loop whose re-optimization schedule mirrors how it would actually be re-tuned in production, and the headline metric has to beat the do-nothing alternative, not just look good in isolation. Statistical credibility checks - multiple-trial correction, combinatorially purged cross-validation, bootstrap confidence intervals, probability of selection overfitting - are computed and persisted as diagnostics and guardrails rather than verdict inputs, accumulating until live outcomes allow them to be calibrated against realized results.

Completed walk-forward task with stitched out-of-sample equity against buy-and-hold
A finished walk-forward task: stitched out-of-sample equity against the buy-and-hold line, with the search-boundary notes the scorer records.

The strongest guardrail is a randomized-control null distribution: the complete walk-forward loop is re-run dozens of times with the signal replaced by seeded random entries, measuring whether the optimization process can manufacture an equally attractive result from pure noise. Distributed across the compute fleet through the shared work queue, a 50-seed null distribution completes in about two minutes, making the test routine rather than prohibitive.

Fleet page showing gateway status, drain hosts, and recent queue remediations
The shared queue runs across a drain fleet. Gateway state, worker load, and recent remediations stay on one screen.

Production behavior is then compared against a pre-committed forward expected distribution rather than against in-regime evaluation headlines.

Experiment Orchestrator

The platform includes a service-based experiment orchestrator outside the main web UI. It can run the evaluation chain unattended for days: data pre-flight, download, Discover, parameter-range validation, walk-forward bake-off, credibility promotion, composition, and packaged evaluation. The orchestrator interleaves throttled data downloads with otherwise-idle research compute, dispatching each task's evaluation once its data is ready.

Data Pulls page with backfill controls and a history of captured market snapshots
Data ingestion is its own queue: preflight, backfill, and the snapshot history later evaluations read.

A bounded evaluation steward runs as a separate LLM-guided layer. A deterministic queue runner keeps evaluations running from a curated backlog; the LLM steward periodically reads results, writes findings, and queues the next experiments. Its trust boundary is deliberately narrow: it can read and call a restricted steward CLI, but has no raw shell, write, git, database, or production-promotion access.

Experiment orchestration is distinct from production promotion. Evaluation can run unattended; promotion remains gated by dashboard review and manual operator approval.

Persistent Harness

The public optimizer's core innovation is the persistent .NET harness. Each Docker container runs a long-lived dotnet /Harness/LeanHarness.dll process that reads JSON requests from stdin, runs evaluations with full engine state reset between runs, and writes JSON responses to stdout. This eliminates per-eval .NET startup, assembly loading, and engine initialization overhead.

The harness uses the engine's regression-test reset sequence: Config.Reset(), Composer.Instance.Reset(), SymbolCache.Clear() - the same sequence the engine's own test suite uses to run multiple evaluations in one process. Console.Out is redirected to stderr so trace/debug output does not corrupt the JSON protocol.

Engineering Controls

Background

The genetic algorithm at the core of lean-optimizer is the same class of algorithm I developed at Togai InfraLogic, where my research in fuzzy logic and genetic algorithms was published at IEEE and NASA conferences. Today, the same optimization principles run inside a modern evaluation-infrastructure stack.

Tech Stack

Generated output stats available on request.