When a free hERG margin beats us.

The comparison is a three-class call, Low, Intermediate or High torsade risk, on the 28 compounds of the CiPA reference set, whose classes are published. Twelve drugs were used to fit two ordinal cut points. Those cuts were then frozen and the remaining sixteen were scored blind.

The comparator is deliberately unflattering to us. Its score is a single number: −log₁₀ of the hERG IC₅₀ divided by the free C_max. No simulation, no solver, no model. Two ordinal cut points are then fitted to it by the identical protocol, so the two methods are scored on the same footing and neither gets a free threshold.

The reference file carries two published hERG IC₅₀ columns, a dynamic-protocol fit and a static fit, and they disagree by up to seventeen-fold on a single drug. Our harness uses the static column, and that is the column behind every number on this page. Feed the margin the dynamic column instead and it scores 11 of 16 blind and 19 of 28 overall. The baseline beats us by one compound on the numbers we ran, and by two on the ones we did not.

Read the third column too. A classifier that ignores the compound entirely and always answers “Intermediate” gets 11 of 28. Both real methods clear it, but not by the margin either of us would like, and it is the number most benchmark pages leave out. It comes back in the next section, where it ties us on the one metric we had been treating as a win.

One caveat on “costing nothing”. The margin is free here because someone else already paid for both of its inputs and published them. On a compound of yours it needs a hERG patch-clamp IC₅₀ and a free clinical C_max, which is an assay and a Phase 1 result. What is free is the arithmetic, not the data.

Exact accuracy is one metric and it hides the shape of the errors.

Four things fall out of that table. Two are bad for us, one is good, and one is a claim we made in an earlier draft of this page and have had to withdraw.

Our perfect adjacent accuracy is not a virtue. This is the withdrawn claim. We wrote 28 of 28 in bold and called it a win. Then we added the third column, and a classifier that ignores the compound and always answers “Intermediate” also scores 28 of 28, because a middle answer is never two classes from anything. Our adjacent accuracy is a consequence of hedging, not of modelling. It stays in the table because removing it would be worse, but it should not have been highlighted and it is no longer.

We are badly biased toward the middle. That is the same finding said plainly. The simulation issues “Intermediate” for 20 of 28 compounds and manages a single Low call in the entire panel. The hERG margin distributes its calls far more like the truth. If you want a method that will clear a compound, ours is currently not it.

Where we do win, it is on under-calls. This is the one column that goes our way, and it is the column a safety scientist cares about, because a missed liability costs more than a false alarm. Two compounds are under-called by the simulation, both High drugs dropped to Intermediate, and neither reaches Low. The hERG margin under-calls four, and two of those go all the way to Low: astemizole and risperidone. Astemizole was withdrawn from the market over QT prolongation, its free C_max is 0.26 nM against a hERG IC₅₀ of 33.3 nM, and a 128-fold margin reads as safe on a spreadsheet. The full read on astemizole is here.

Verapamil is not the win we first described. It is the one drug the hERG margin misses by two classes: clinically Low, called High, because verapamil blocks calcium hard enough to offset its own repolarization delay and a hERG-only score cannot see the offset. But we do not get verapamil right either. We call it Intermediate. The margin fails it by 0.038 log units against a cut point fitted on twelve drugs, and a shift of 0.04 in that cut (about seven percent) erases the error entirely. Head to head, the simulation is right on two compounds the margin misses (astemizole, risperidone) and wrong on three the margin gets (loratadine, nifedipine, nitrendipine). Net, we are one behind. Verapamil is a good illustration of what multichannel simulation is for. It is not evidence that ours works.

Neither of us is the published state of the art. The ORd/qNet column is the CiPA consortium’s own result and its High-vs-rest separation is perfect. Note the label: those two figures are reported on CiPA’s own validation subset, not on all 28, so the column is not on the same footing as ours. The published 1.00 carries a 95% confidence interval of 0.92 to 1. We reproduce the column here rather than omit it.

What the run does add is a discipline the first one lacks: a locked hold-out, a permutation null, and a pass/fail gate fixed before the scores were seen. Restricting to the 16 held-out drugs, High-vs-rest AUC is 0.9375 for the hERG margin and 0.9062 for our score — the same two numbers section 02 reports on all 28, recomputed on the blind split.

The harness for this run is public at github.com/Perturb-Bio-Inc/cardiac-safety-eval, including the line that reuses the first run’s output. The three-class harness in section 02 is not yet published; if you want it, ask and we will send it.

Two things in that table deserve more than a dash. The multichannel logistic model does not merely tie the baseline. Its permutation p is 0.235, so the harness cannot separate it from shuffled labels at all. And the confidence intervals on the two AUCs overlap almost completely. Sixteen drugs, four of them High, gives 48 pairwise comparisons, so the AUC can only move in steps of about 0.021. The gap between 0.9375 and 0.9062 is one and a half pair swaps. This table cannot rank these methods. It can only say that none of ours cleared a gate set in advance.

One note on the permutation null. The Kernik score reaches p = 0.014 against a null mean of 0.4996, which is well centred. But that candidate fits nothing; it is a fixed column passed through. The fitted candidates on this same benchmark have nulls near 0.37, which the harness README records as a known small-n artefact. Do not read 0.014 as evidence the model is good. Read it as evidence the column is not random.

If a hERG margin already answers your question, use it rather than paying us. A customer who buys the wrong thing complains later, and this company is too small to survive that.