Does the design scorer work?
An independent check of designScore, the scorer that rates synthetic organism designs. The test: real organisms should score high, designs that real biology rules out should score low, and nobody should be able to fool the scorer easily. The evaluator did not build the scorer or the benchmark. Every number below is recomputed from the run file, and every failure is kept.
Scorer versions side by side
Each version was frozen before the held-out families were opened, and each was scored blind once with the same protocol. The headline column is held-out, family-out.
Pick a split
Every item is scored with its own references taken out. Specimen-out removes the specimen itself. Species-out removes every specimen of its species. Family-out removes its whole family. The held-out set was never used to build or tune the scorer. Your choice applies to every chart below.
Score distributions: real organisms vs each kind of negative
Each row is a histogram of scores from 0 to 1, with 20 bins. The vertical rule is the pre-registered plausible cut at 0.5. Real organisms should pile up on the right, and negatives on the left.
Separation, with 95 % bootstrap intervals
ROC-AUC by split
1.0 means perfect separation and 0.5 means chance.
Does the scorer beat simple baselines?
The same items, ranked by simpler rules. If a baseline matches the scorer, the scorer's extra complexity buys nothing.
Which real families score low?
Real organisms grouped by family, showing the share that scored at or above the cut. The chart lists the families with at least 2 items scored, lowest first. A low family is either a scorer failure or a gap in the reference data.
Lowest-scoring real organisms (inspect)
Calibration and stability
Calibration
Bars show the fraction of items in each score bin that are real. The diagonal marks what a calibrated probability would give. The benchmark's mix of real and negative items moves this, so read it as a shape, not as a probability.
Weight perturbation (each weight ±20 %)
Each component weight is moved ±20 % on its own, and the items are recomputed from the scorer's own component values. Do the conclusions hold?
Trying to break it
Two searches for designs that score as plausible although they are impossible. The first runs over the seven numbers the body block reads, restricted to designs that break a hard geometric or physical rule. The second runs over edits to traits the scorer never reads (gills, hemoglobin, ray counts, density).