In short
Everything learnt in Phases 1–2 is applied, unchanged, to test data it never saw. Every method is scored on the same 50 splits, so the scores can be compared split by split, and the differences are tested statistically.
The evaluation phase involved applying the scaler, forest, reducer and VQC, fitted in Phases 1–2 on each split’s training data, unchanged to that split’s held-out test fold. The test fold is never used to fit anything from Phases 1–2, including the scaler. The entire process is run over splits.
Metrics
Malware was treated as the positive class for every metric.
- Accuracy: the overall correctness,
- F1 score: the balance of precision and recall
- Precision: , the share of samples labelled as malware that really were malware
- Recall: , the share of malware that was correctly identified
- AUC-ROC: the area under the curve of true-positive rate against false-positive rate for all thresholds
Statistical testing
To tell whether differences between conditions were real and not noise from the splits, the scores were paired and tested with a Friedman test, Wilcoxon signed-rank tests and a Bonferroni correction.
Each condition produced 50 scores, one per split, and every condition used the same 50 splits. For example, condition A’s score on split 7 is paired with condition B’s score on split 7. This removed the effect of “easy” and “hard” splits from the comparison.
The Friedman test was run first for all conditions at once, so that pairwise differences are only reported if the conditions differ overall. Each pair of conditions was then compared using a two-sided Wilcoxon signed-rank test on the pair’s 50 paired scores. This was chosen because the scores were close to 1.0, which makes them bounded and skewed, so normality could not be assumed.
Because many tests were run, some “significant” results would be likely by chance, so a Bonferroni correction multiplied each p-value by the number of tests in its group, capped at 1. A difference was significant if its corrected p-value was below 0.05. Group sizes were 36 tests per metric for the final comparison (all pairs of 9 conditions) and 14 for the qubit sweep (2 RF methods against PCA at 7 qubit counts). The 12-qubit results used only 10 splits and were treated as exploratory.