Day 30 : Architecture Complete

8d (edited) • Builds

sqx-quant-agent

**Build an efficient automation system for StrategyQuant Pro** — cycle through predefined instruments, timeframes, and templates to generate profitable strategies at scale; no manual intervention in the generation loop

What the SQX Quant Agent Actually Does — Step by Step

PART 1 — STARTUP

The agent loads config.json from config/ — this tells it where SQX is installed, where the projects folder is, what the Finals Collector project is named, what the thresholds are (min OOS count, min Final count to promote, etc.).
It loads .api-keys.env for any credentials needed (data feed keys etc.).
It starts the HTTP dashboard — a static HTML snapshot of queue state, updated periodically.
It starts sqcli.exe as a persistent background process, connecting to SQX's HTTP API. SQX itself must already be running (or sqcli starts it — confirm which).
It reads combinations.csv from config/ — this is the master list of what to run. Each row is one instrument+template combination, with columns for symbol, template name, directions (long/short/both), timeframes, eod_timeframes, status, notes.
It reads queue.json from state/ — this is the live queue. If queue.json already exists (crash recovery), it loads the prior state and resumes from wherever it left off. If it doesn't exist, it builds the queue fresh from combinations.csv.
It reads history.json from state/ — the accumulated log of every step ever run. Appended to, never overwritten.
It checks control.json for any operator signals (pause, stop-safe, stop-now) left over from a prior session. If stop-now is set, it halts immediately.
It is now running. The main loop begins.

PART 2 — THE MAIN LOOP

The agent loops continuously. Each iteration:

It reads control.json. If stop-now is set, it exits immediately, mid-run if necessary. If stop-safe is set, it finishes the current step then exits. If pause is set, it waits in a sleep loop, checking every few seconds, until the operator clears the signal.
It looks at queue.json to find the next pending step. The queue is a list of combinations; each combination has an explicit list of steps (e.g. ["H2-long-step1", "H2-long-step2", ..., "H2-long-step10", "H1-long-step1", ...]). It picks the first step whose status is pending.
If no pending steps exist in the queue, it checks combinations.csv for any rows not yet imported. If found, it imports them, generates their step lists, appends to queue.json, and continues. If there is genuinely nothing left to do, it idles and waits (or exits, depending on config).

PART 3 — RUNNING ONE STEP

A "step" is one build run of SQX — one combination of symbol, template, timeframe, direction, and build configuration (the 10 steps being 10 progressively tightened robustness settings applied in sequence).

The agent reads the step definition from queue.json — it knows: symbol (e.g. GDAXIm_darwinex), template (e.g. generic), timeframe (e.g. H2), direction (short), step number (1–10), and the specific build config parameters for that step.
It resolves the project folder path — something like C:\StrategyQuantX144\projects\GDAXIm_darwinex-generic\. This folder is reused for the entire combination across all 10 steps. It is never deleted.
If the project folder doesn't exist yet (first step of this combination ever): It creates the folder. It copies the base .cfx template file into it. It uses CfxEditor to modify the .cfx XML — sets the symbol, timeframe, direction, build mode, data range, and the step-specific robustness parameters. It calls sqcli loadconfig pointing at this new .cfx — SQX loads the project into its JVM.
If the project folder already exists (continuing a prior combination): It calls synctofiles on the current project first — this flushes any strategies sitting in SQX's JVM memory to disk inside the .cfx file. This must happen before any loadconfig call or those strategies are lost. It then uses CfxEditor to modify the existing .cfx — updates the step-specific build parameters for this new step. It calls sqcli loadconfig — SQX reinitialises the project from the updated .cfx on disk. Strategies previously synced to disk are now back in JVM memory.
It calls sqcli start — SQX begins generating strategies according to the loaded configuration.
It attaches ProjectLogMonitor — a real-time watcher on projects/<name>/log/global_log_*.log. This log file is written by SQX during the run and reports events including databank counts after every completed internal task.

PART 4 — MONITORING THE RUN

The agent watches the project log. SQX runs its internal build loop — generating candidate strategies, running them through in-sample testing, OOS testing, Monte Carlo simulation, stress testing. Each internal task completion writes a log entry with current databank counts.
The agent parses these log entries in real time. It extracts: Results count — raw strategies generated RawFilter count — survivors after basic filter OOS count — survivors after out-of-sample test Monte Carlo count — survivors after MC robustness Final count — strategies in the Final databank
It updates the dashboard with live counts.
It waits for the run to complete. SQX signals completion via the log (a specific completion event). The agent detects this, confirms the run is done.

PART 5 — AFTER THE RUN COMPLETES

It reads the final databank counts from the log. These are the authoritative numbers — not from filesystem counts, which are unreliable.
It calls synctofiles — flushes all strategies from JVM memory to disk. This is the critical call that ensures what SQX found is actually persisted.
It writes a record to history.json via history_logger.py: Step ID Symbol, template, timeframe, direction, step number Timestamp start and end Results, RawFilter, OOS, MC, Final counts Survival rates at each stage Death stage (where most strategies were eliminated) Builder stats if available
It updates the step status in queue.json from pending to complete.
It evaluates the promotion threshold: does the Final count meet the configured minimum (e.g. ≥ 5 strategies in Final after step 10)? This check only applies after step 10 — the last step of a combination.

PART 6 — PROMOTION TO FINALS COLLECTOR

This only runs after step 10 of a combination, if the threshold is met.

It opens the working project .cfx using CfxEditor and extracts the strategies from the Final databank — reads them as XML objects from inside the ZIP archive.
It opens the Finals Collector .cfx — a separate SQX project that accumulates winning strategies across all instruments and templates. It is never deleted, never recreated.
It appends the winning strategies from the working project's Final databank into the Finals Collector's databank XML.
It saves the Finals Collector .cfx back to disk.
It calls sqcli loadconfig on the Finals Collector to reload it into SQX, so SQX's JVM reflects the new strategies.
It logs the promotion event — how many strategies were promoted, from which combination, to the Finals Collector.
It marks the combination as complete in queue.json and archives it to queue_archive.json.

PART 7 — SUNDAY DATA UPDATE

This runs on a schedule, independent of the queue.

On Sunday, the agent calls sqcli with -instrument action=list to pull live instrument data from the connected data feed.
It parses the response — symbol names, spread values, available data date ranges.
It writes/overwrites instruments.csv with the updated values.
It checks for any symbols in the live feed not already in combinations.csv — logs them as newly detected instruments for the operator to review.
It does not automatically add new instruments to the queue — that decision stays human.

PART 8 — WHAT THE OPERATOR SEES

At any time, the operator can open browser and see the queue report — current step running, pending steps, completed combinations, Finals Collector count.
The operator can write pause, stop-safe, or stop-now to control.json to signal the agent.
queue_report.html in state/ is a static snapshot — readable even when the agent isn't running.
history.json is the full audit log — every step ever run, every count, every survival rate.

WHAT IT PRODUCES

By the end of a full instrument pass:

Finals Collector project — a populated SQX project containing the best strategies found across all 31 instruments, both directions, multiple timeframes. This is the input to the next human step: loading it in QuantAnalyzer, running portfolio analysis, selecting strategies for FTMO challenge submission.
history.json — the data that will drive the Sprint 3 Orchestrator's prioritisation decisions (which instruments generate the most survivors, which timeframes are most productive, where strategies die in the pipeline).
queue_archive.json — full record of everything run.

1 comment