Experiment Dashboard — HdlFormalLLM

How this works standalone benchmark runner — it exams the LLM itself, not designs you made in the main app

Flow: pick LLM(s) and module(s) → Start Run → for each task the LLM reads the spec, writes BDD scenarios and a testbench; the testbench is verified on the golden (reference) design, then challenged with fault-injected mutants → scores appear in Results. Nothing needs to be prepared in the main app first.

Benchmark Suite The fixed exam tasks shipped with the repo — each has a natural-language spec, a golden design and fault mutants. Reference only; module names (e.g. alu_8bit) are fixed and unrelated to the bit-width options in the main app. Run Experiments Choose LLMs × modules × repetitions and start a run. Results Compile: generated testbench compiles · Golden Pass: passes the correct design, i.e. the testbench itself is valid · Mutation Score: fraction of injected bugs caught (headline metric) · Completeness: functional-point coverage (only with a judge LLM). Top table = per-LLM averages, bottom table = per-experiment detail. LLM Call Log All-time API-call statistics for the whole app, not tied to a run — debug only.

Benchmark Suite fixed exam tasks: spec + golden design + fault mutants

Module	Category	Difficulty	Functional Points	Mutants

Run Experiments exam the LLM on fixed benchmark tasks (not your uploaded designs): spec → BDD → testbench → golden gate → mutation testing

LLM providers (mock = pipeline test, no API keys)

Benchmark modules — select at least one

Repetitions

Parallel workers

Run ID (optional)

FP-coverage judge (optional)

Feedback Loop — Phase 2 iterative repair from simulation feedback · "bdd": revise scenarios, regenerate TB (BDD is the only feedback carrier) · "bdd+": revise scenarios AND show feedback to the TB generator · "tb": patch TB directly · mutation scored blindly each round

LLM providers

Benchmark modules — select at least one

Iterations

Repetitions

Parallel workers

Arms

bdd bdd+ tb

Run ID (optional)

Run

Score vs iteration — does feedback improve the tests, and which arm wins?

Arm	Iteration	n	Compile	Golden Pass	Mutation Score	Errors

Results golden pass = testbench correct; mutation score = bugs caught

Run

Per-LLM summary

LLM	Runs	Compile	Golden Pass	Mutation Score	Completeness	Avg Scenarios

Per-experiment detail

Module	LLM	Rep	Scenarios	Compiled	Golden	Mutants Caught	Completeness	Error

LLM Call Log all-time API calls across the whole app (debug) — not just the selected run

Provider	Model	Calls	Success Rate	Avg Latency