Quality-filtered ranking: (1) AF3 ipTM >= 0.5 (18 designs), then (2) OF3 ipTM >= 0.5 (82 designs). Only designs that pass orthogonal validation thresholds are included. No junk.

The primary competition metric (ipSAE) and Boltz-2 ipTM show no correlation with AF3 ground-truth validation. AF3 does correlate with OF3, suggesting both capture real structural features that Boltz-2 misses.

Figure 0. ipSAE (Boltz-2 competition metric) vs AF3 ipTM for 60 validated designs. Pearson r = -0.044 (p = 0.741), Spearman rho = -0.233. ipSAE has no predictive value for AF3 validation. Marginal boxplots show per-face distributions. Edge color: black = top hit, green = promote, orange = review, gray = discard. This is the central finding: the competition ranking metric does not predict real binding as assessed by AF3.

Three independent structure prediction methods -- Boltz-2 (scoring), OpenFold3, and AlphaFold3 -- show dramatically different agreement patterns. AF3 is the ground-truth orthogonal validation.

Figure 1. Boltz-2 ipTM vs AlphaFold3 ipTM for 60 validated designs. Points colored by target face: E2 (blue, n=45), Cullin (red, n=6), BoltzGen (green, n=9). Marginal boxplots show per-face distributions. Orange line: Ridge regression (r=0.08, rho=-0.12). Dashed diagonal: perfect agreement. Boltz-2 scores show no correlation with AF3 validation (r=0.08), confirming that high Boltz scores do not predict genuine binding.

Figure 2. AF3 ipTM vs OF3 ipTM for 49 designs with both scores. Points colored by target face: E2 (blue, n=40), Cullin (red, n=5), BoltzGen (green, n=4). Ridge regression (r=0.57, rho=0.49) shows moderate positive correlation between the two independent validation methods, suggesting they capture overlapping but distinct structural signals.

Figure 3. Boltz-2 ipTM vs OF3 ipTM for 1,225 designs. Points colored by target face: E2 (blue, n=845), Cullin (red, n=335), BoltzGen (green, n=45). Ridge regression (r=-0.01, rho=-0.04) shows essentially zero correlation between Boltz-2 scoring and OF3 validation at scale, consistent with the AF3 result. Boltz-2 ipTM is uninformative for predicting orthogonal validation scores.

Designs target three RBX1 surfaces: E2 interface (12,112 designs), Cullin interface (6,791), and BoltzGen full-surface (1,820). ipSAE and structural confidence metrics vary by target.

Figure 4. ipSAE distribution by target face with individual points (downsampled to 300 per group). BoltzGen designs achieve higher mean ipSAE (0.446) than E2 (0.227) or Cullin (0.205). Significance bars show two-sample t-test p-values between groups.

Figure 5. Four key metrics (ipSAE, Boltz ipTM, Boltz pLDDT, Boltz pTM) compared across target faces. BoltzGen shows higher ipSAE but generally comparable Boltz confidence metrics, suggesting that the ipSAE advantage is interface-specific rather than driven by global structural quality.

Binder lengths range from under 55 to over 100 amino acids. Size buckets reveal whether longer binders achieve better interface quality or structural confidence.

Figure 6. ipSAE by binder size bucket (<55, 55-65, 66-99, 100+ AA). Mean and count annotations per group. No strong size-dependent trend in interface quality.

Figure 7. Boltz-2 ipTM vs binder length colored by target face. Small binders (40-65 AA) dominate the E2 dataset with wide ipTM spread. Cullin and BoltzGen designs span broader length ranges.

Designs were generated with three model variants: Base (standard Boltz-2), Beta (beta checkpoint), and BoltzGen (generative model). Comparing ipSAE across checkpoints.

Figure 8. ipSAE by model checkpoint (Base, Beta, BoltzGen). Mean and count annotations per group. BoltzGen designs achieve higher ipSAE on average, while Base and Beta show similar distributions.

Hit rate curves show the fraction of designs exceeding a given quality threshold across the full ipSAE and OF3 ipTM ranges, stratified by target face.

Figure 9. Hit rate curves by target face at ipSAE thresholds. BoltzGen maintains the highest hit rate across most thresholds. E2 and Cullin face designs show similar hit rate profiles.

Figure 10. Hit rate curves by target face at OF3 ipTM thresholds (for the 1,225 designs with OF3 validation). Reveals which target face produces the most designs above stringent OF3 cutoffs.

Side-by-side comparison of Boltz-2, OpenFold3, and AlphaFold3 ipTM scores for top designs ranked by OF3 validation. Highlights consistent Boltz-2 score inflation across methods.

Figure 11. Top 20 designs by OF3 ipTM: Boltz-2 (blue, faded), OpenFold3 (red), and AlphaFold3 (green) scores side-by-side. Faint green bars indicate designs without AF3 data. Boltz-2 consistently overestimates compared to both orthogonal validation methods, while OF3 and AF3 show better mutual agreement.

Each RFdiffusion campaign was conditioned on a specific set of RBX1 hotspot residues. Four distinct hotspot sets were used:

ipSAE vs ipTM by hotspot set

Joint distribution of ipSAE and ipTM colored by hotspot set. Marginal histograms show per-set density. Stars mark the best design in each set. The E2 Enhanced set (with ESM-2 DMS-derived residues I54, I84, C42, C53) produces the highest ipSAE values, suggesting that adding mutation-sensitive residues to the hotspot specification improves interface quality.

ipSAE vs binder length by hotspot set

Binder length vs ipSAE by hotspot set. Small binders (40-65 AA) dominate the E2 Enhanced set. BoltzGen designs span a wider length range but achieve lower ipSAE. No strong correlation between length and interface quality within any set, though medium-length binders (50-60 AA) appear slightly enriched among top performers.

Hit rate by hotspot set

Fraction of designs exceeding ipSAE thresholds for each hotspot set. The E2 Enhanced set maintains the highest hit rate at all thresholds. The Cullin set shows competitive hit rates despite targeting a different face, while BoltzGen has the highest fraction above ipSAE=0.3 but drops off sharply above 0.5.

Efficiency frontier: ipSAE vs pLDDT

Pareto front (dashed line) of designs optimizing both ipSAE and pLDDT. Designs on the frontier achieve the best trade-off between interface quality and structural confidence. Most frontier designs are from the E2 Enhanced set, but a few Cullin and E2 Standard designs appear at high pLDDT.

Amino acid composition by ipSAE quality tier

Amino acid frequency comparison between top-tier (ipSAE > 0.7), mid-tier (0.3-0.7), and failed (ipSAE < 0.1) designs. Blue = charged (R,K,D,E), red = hydrophobic (F,L,I,M,V,W), green = polar (S,T,C,N,Q,Y). Top-tier designs show higher glutamate (E) and leucine (L) frequency, while failed designs are enriched in alanine (A) and glycine (G), suggesting that oversimplified sequences with low complexity correlate with poor interface formation.

Conservation analysis from 165-sequence MSA (MAFFT) of RBX1 homologs across eukaryotes. 834 raw homologs collected via HMMER, filtered to 272 non-redundant sequences, aligned with 1155 total rows including outgroups.

Shannon entropy-based conservation score (0 = variable, 1 = perfectly conserved). Colored by functional role: E2 interface, Cullin interface, Zinc-binding, other. Hover for details.

Each residue colored by conservation score. Hover for details.

Masked marginal log-likelihood ratios (dLLR) computed with ESM-2 (650M params) for all single-point mutations across the 108-residue RBX1 sequence. More negative dLLR = more deleterious mutation. Sensitivity = mean |dLLR| across all 20 amino acids at each position.

Log-likelihood ratio for each of 20 amino acids at each position. Blue = tolerated/beneficial, red = deleterious. Wild-type residue marked with black dot. Hover for values.

Mean |dLLR| across all 20 amino acids. Higher = less tolerant of mutations. Colored by functional role.

Comprehensive analysis of what predicts AlphaFold3 validation success. 60 designs validated with AF3, analyzed using Lasso regression on sequence and structural features. AF3 is our ground-truth orthogonal validation — Boltz-2 scores are heavily inflated for E2-face RFdiffusion designs.

LassoCV regression on 20 sequence and design features to predict AF3 ipTM. Features with non-zero coefficients are the strongest predictors after regularization. Positive = helps AF3 validation, negative = hurts.

Feature correlations with AF3 ipTM. The strongest predictor of AF3 success is being a BoltzGen design (untargeted, diverse topology), followed by sequence features like aromatic content and lower alanine fraction. High Boltz-2 ipTM is actually a weak or negative predictor — designs with inflated Boltz scores tend to fail AF3 validation.

Top 8 features correlated with AF3 ipTM. Each subplot shows one feature vs AF3 score with Ridge regression line and Pearson r. Points colored by pipeline (blue=RFdiff E2, red=RFdiff Cullin, green=BoltzGen).

Comparison of successful designs (AF3 ipTM >= 0.5) vs failed designs (AF3 ipTM < 0.3) across key features. Successful designs tend to have higher sequence entropy (more diverse amino acid usage), more aromatic residues, and lower alanine content.

Amino acid frequency comparison between AF3-validated (ipTM >= 0.5) and AF3-failed (ipTM < 0.2) designs. Successful designs are enriched in structurally important residues (F, W, Y aromatics; charged residues) and depleted in simple residues (A, G).

BoltzGen designs show dramatically better AF3 validation than RFdiffusion+MPNN designs. BoltzGen mean AF3 ipTM is 2-3x higher, despite lower Boltz-2 scores. This suggests RFdiffusion+MPNN designs may be optimizing for Boltz-2 scoring artifacts rather than genuine binding.

After submission, the ADAPTYV competition independently re-ran Boltz-2 on all 83 submitted designs and scored them with 12 structural metrics. We performed comprehensive feature engineering (64 features from sequence composition, structural propensities, Boltz-2 confidence maps, and our scoring metrics) and regression analysis to understand what predicts competition performance.

Of 83 submitted designs, 33 survived with their ipSAE ≥ 0.6 and 31 failed below 0.3. The submission included 30 designs with our ipSAE < 0.5 -- some performed surprisingly well in competition scoring. The correlation between our ipSAE and theirs is only r = 0.312, showing that internal scoring was a poor predictor of competition performance. Seven designs we scored below 0.5 actually survived (≥ 0.6) in the competition, while 9 designs we scored above 0.7 failed (< 0.3).

Figure 1. Our ipSAE vs competition ipSAE. Point size = binder length, color = target face. Annotated designs are surprising reversals: low ours but high theirs (upper left) and high ours but low theirs (lower right). The green/red horizontal lines mark survival (0.6) and failure (0.3) thresholds.

We extracted 64 features per design spanning four categories: (a) amino acid composition (20 individual AA fractions plus grouped fractions for charged, hydrophobic, polar, aromatic residues), (b) sequence properties (Shannon entropy, linguistic complexity, net charge, charge density, hydrophobicity, isoelectric point, molecular weight, max homopolymer repeat), (c) structural propensities (helix, sheet, disorder) and Boltz-2 PAE/pLDDT features from our prediction NPZ files (mean interface PAE, confident contacts, PAE asymmetry, binder pLDDT), and (d) design metadata (wave, face, binder length) plus our scoring metrics (ipSAE, ipTM, pTM, pLDDT, AF3 ipTM, OF3 ipTM). Structural features were extracted for 80 of 81 matched designs.

Figure 2. Full correlation matrix of our features (top-left block) vs their 12 competition metrics (bottom-right block). Black lines separate the two groups. Their metrics cluster tightly (ipSAE, ipTM, LIS all r > 0.9). Our scoring metrics show only weak correlation with theirs (r ~ 0.3). Sequence charge features and structural PAE metrics show moderate associations with competition outcomes.

We trained three models to predict competition ipSAE from our 64 features. All models show negative cross-validated R2, indicating that with only 81 samples and 64 features, none generalizes reliably. Ridge regression (CV R2 = -2.22) heavily overfits. ElasticNet CV selects 22 non-zero features (CV R2 = -0.69, l1_ratio = 0.10). Random Forest is least negative (CV R2 = -0.33) and achieves train R2 = 0.78, suggesting nonlinear patterns exist but cannot be captured with this sample size. The negative CV R2 values confirm that no model beats predicting the mean -- individual competition scores are inherently unpredictable from our available features.

Figure 3. Top 15 features by absolute coefficient magnitude in Ridge (left) and ElasticNet (right) regression. Green = positive coefficient (higher value predicts higher competition ipSAE), red = negative. Contacts_pae_lt5 (number of confident interface contacts) is a strong positive predictor. Molecular weight is negative, suggesting shorter binders perform better.

Figure 4. Random Forest feature importance (MDI) for predicting competition ipSAE. Net charge is the top predictor, followed by serine content (aa_S), isoelectric point, and charge density. Our ipSAE ranks 7th. Structural features (mean_binder_pae, struct_binder_plddt) also contribute.

We compared 33 survivors (their ipSAE ≥ 0.6) against 31 failures (< 0.3) using Welch's t-test on all 64 features to find the most discriminating ones. The top 6 features are: net charge (p = 3.5e-5), isoelectric point (p = 3.4e-4), our ipTM (p = 0.003), our pTM (p = 0.003), charge density (p = 0.003), and our ipSAE (p = 0.004). All are statistically significant, but effect sizes are modest.

Figure 5. Box + swarm plots of the 6 most discriminating features between survivors (green, their ipSAE ≥ 0.6, n=33) and failures (red, < 0.3, n=31). Net charge is the strongest discriminator: survivors tend to have more negative net charge (mean -8.4 vs -4.0). Our ipTM and ipSAE are significantly higher in survivors but overlap substantially.

We tested whether our OpenFold3 ipTM scores predict competition ipSAE across the 81 designs with OF3 data. OF3 serves as an orthogonal validation method independent of Boltz-2.

Figure 6. Our OpenFold3 ipTM vs competition ipSAE. Color = target face. The dashed line shows the linear trend. OF3 provides some signal for competition performance but the relationship is noisy and face-dependent.

We tested whether our AlphaFold3 ipTM scores predict competition ipSAE across the designs with both AF3 and Pb data. AF3 serves as another orthogonal validation method independent of Boltz-2.

Figure 6b. Our AlphaFold3 ipTM vs competition ipSAE with boxplot marginals. Color = target face. Ridge regression line (orange) with Pearson r and Spearman rho shown.

The most surprising finding is the extent of score reversals between our predictions and the competition's.

Trimming, alanine scanning, and mutational optimization of UBC12 (PDB 4P5O chain I) — the natural E2 binding partner of RBX1. All scoring via OpenFold3. Auto-updated: 2026-04-09 09:05

Best fragment: res 25-135 (111 AA) with OF3 ipTM = 0.860 (vs full-length 0.745). Trimming the disordered N-terminus improves binding by +0.115.

All 27 interface positions scanned. The interface is remarkably robust — no single Ala mutation drops ipTM by more than 0.02. Most tolerant: K120, W119, E117. Most sensitive: R33, D37.

369 variants: 114 full-scan at 6 tolerant positions + 90 targeted at 6 moderate positions + 165 double mutants. 366 scored so far, 48 show improved ipTM over WT (0.8603).