Adversarial Attacks & AI-Based Selective Pressure
Issue № 84 // Vector.AI, a Next-Generation Biodefense Company
Note: Certain names, locations, and dates have been changed to protect the privacy of investors, employees, and company data.1
Issue № 84 // Adversarial Attacks & AI-Based Selective Pressure
Paul Graham, co-founder of Y Combinator famously advises startups to just make something people want. It’s the kind of obvious, yet pithy, advice that’s guaranteed to get the oracle of Silicon Valley a few thousand retweets. And for what it’s worth, it’s actually a reasonably effect tactic for raising money: find the latent gap in the market, build an MVP to attract enough initial users to prove the concept, then pitch VCs.
Unfortunately, this playbook wasn’t quite right for my last company, leaving us rudderless. We weren’t exactly selling something people wanted, after all. It wasn’t until I read Max Brook’s World War Z on the flight back from an unsuccessful VC pitch that I found a framing that resonated.
“Fear is the most valuable commodity in the universe. Turn on the TV... What are you seeing? People selling their products? No. People selling the fear of you having to live without their products. Fear of aging, fear of loneliness, fear of poverty, fear of failure... Fear is primal. Fear sells.”
All it took to raise a $22M seed round after that was a seven slide pitch deck and a brief demo that scared the shit out of a few venture partners on sand hill road. Of course, timing played a significant part as well. It was 2021 and for the past year we’d watched a novel coronavirus dismantle the global economy, and the consensus among biotechnologist was that this was not a once in a lifetime event.
If you’re to take Global Biodefense News at face value zoonotic spillover was accelerating while surveillance infrastructure stagnated. Additionally, the drug development pipeline designed around decade-long timelines had proven catastrophically mismatched to the speed at which a pathogen could move. My team and I got funded to build something radically different: a platform for rapid therapeutic response to emerging infectious threats, capable of identifying and validating druggable targets in a novel pathogen within weeks rather than years.
The core of what we build was a graph neutral network trained to identify exploitable vulnerabilities in the viral proteome. Essentially, the model takes a multi-modal representation of a pathogen—sequence, predicted structure, known or inferred protein-protein interactions, phylogenetic context—and learns to identify subgraphs associated with therapeutic vulnerability. These could be proteins the virus depends on structurally, interactions where disruption is likely to be fitness-costly, or surfaces where small molecules could plausibly bind and interfere. By training our model on the accumulated virology data from pre-COVID and continuously updating it as new experimental data arrived from the robotic wet lab, we were able to achieve impressive performance faster than expected.
We designed our wet and dry labs around the same philosophy: high throughput and reproducibility at scales humans cannot achieve, let alone sustain long term. Liquid handling robots ran compound screens, processed samples, and prepared sequencing libraries while the GNN guided the next round of experiments, creating a virtuous cycle of recursive self improvement. With this setup, we could iterate through dozens of hypothesis in just a few days. For example, we could identify a target in a novel pathogen’s proteome on Monday, then by Thursday we’d have dose-response data on a panel of candidate compounds. By the following week, mechanistic validation. The speed was hard to fathom and after years doing these things by hand it felt intoxicating. For just a brief moment I really believed the problem of pandemic response was tractable.
Unfortunately, shortly thereafter I learned that production and understanding are not the same thing, which I recently wrote about in Reflections on Cold Spring Harbor Laboratory’s AI in Biology Symposium. In that piece, I wrote about a paper on foundation models trained to predict planetary orbits. Essentially, the model in discussion learned to forecast planetary positions with high accuracy, but when probed for the underlying force law, it produced utter gibberish. Why? It has assembled a collection of heuristics sufficient for within distribution prediction but hadn’t discovered Newtonian gravity. Without gravity, you can’t predict where a planet will be tomorrow. This is what prediction without understanding costs you: you can see what will happen, but you cannot reason about what to do next.
Our GNN wasn’t that different. It predicted druggable targets with high fidelity, but it couldn’t tell us why a given surface protein was druggable in a way that was legible or that generated a new hypothesis we could test with a different experiment. It was as if our model knew more that we ever could about the statistical structure of the viral proteome, but it didn’t understand even rudimentary virology. For most of what we were working on this was fine—the predictions tended to hold up and the compounds we tested worked. We didn’t care if the model couldn't explain itself, nor were we asking it to; we were asking it to find vulnerabilities, and it did. The trouble started when we began to notice that it was finding them less and less often.
The signs and symptoms were subtle at first. Our hit rate (the fraction of pathogen samples for which the pipeline successfully identified an effective therapeutic target) has been steadily climbing through our first 18 months of operation, up until early ~2023. Then it plateaued, following by a decline. The fall wasn’t catastrophic, but it happened consistently, across multiple pathogen families. The compound that worked continued to work and when the pipeline found a good target that we could drug the data looked the same as always. But, an increasing number of samples were coming back from screening without a viable hit—the model was recommending targets and the robots were dispensing compounds against those targets, but the compounds weren’t working. Curiously, this wasn’t because the compounds were inactive against the proteins in question, but rather, as we established via counter-screening, that the proteins the model identified weren’t the right targets in those samples. Essentially, something was causing our model to look in the wrong place.
We spent months trying to figure out the problem. Initially, I thought it was model drift: GNNs degrade when the data distribution shifts, and the pathogens we were processing different in some ways from the training data. But, when we ran diagnostics our validation performance looked stable. Next, we tried explainability analyses including gradient attribution over the input graph, attention visualization, and other standard tools for trying to see inside a network. From this process we could identify which nodes the model was weighting, but we couldn’t figure out why its recommendations went sideways. Interpreting a GNN’s failures, I learned, is like reading a shadow—you get an outline but not a shape.
As a last ditch Hail Mary, we pulled all of the raw multi-omic data from every non-responding sample and conducted a multi-team member exploratory data analysis. We saw the expected patterns of viral replication, the host response programs you’d predict, the protein interaction fingerprints the model had been trained on. But in the non-responding samples, something else was present. A set of anomalous features that didn’t map onto any mechanistic story we could tell including unusual patterns of viral protein expression. These patterns weren’t entirely absent from responding samples, but they were much more pronounced in the ones where the model had failed.
When we traced back through the pipeline—what inputs the GNN had received for the failing cases, what targets it had recommended, where in the model’s learned representations those inputs had landed—the picture became clear. The anomalous features were being interpreted by the network as indicative of specific druggable vulnerabilities, pattern-matching to training examples where similar signatures had corresponded to actionable targets. The model was being misdirected by features of the samples themselves, and following that misdirection into recommendations that were wrong in a systematic, reproducible way.
There is a concept in AI security called an adversarial attack. The canonical example involves a neural network trained to classify images: a researcher, with knowledge of the network’s architecture, constructs an input image perturbed in ways invisible to a human observer but sufficient to cause the network to misclassify with high confidence. The perturbation exploits specific properties of the network’s learned representations. To the network, the perturbed image looks overwhelmingly like the wrong category. To a human, it looks like the original image with some noise.
The key feature of a traditional adversarial attack is that it is engineered. Someone designs it with access to the model’s weights or a reasonable surrogate, running an optimization process to find the perturbation that maximally exploits the network’s vulnerabilities. What I am describing is different and much more troubling.
RNA viruses replicate with error-prone polymerases and generate enormous population sizes on the order of 10¹⁰ to 10¹² particles per infected host. In any given infection, the virus is running something like a massively parallel search through local sequence space, with selection acting on the results in real time. This is how drug resistance emerges classically: a mutation that confers fitness under drug pressure exists somewhere in the quasispecies, gets selected, and takes over. The virus didn’t design the mutation—it searched randomly and Darwinian selection found it.
But, what if our pipeline could introduce a new “artificial” selective pressure? The compounds that reach a given viral sample are determined by what our GNN recommends and as a result a viral variant that produces molecular features causing the model to recommend the wrong target is, in a direct sense, a variant that survives. Its progeny inherit those features and natural selection acts on them. In this case, the model acts like a fixed target that the virus can explore and exploit.
It sounds hyperbolic, but I believe what we observed was selection operating on viral quasispecies at speeds beyond human comprehension, discovering adversarial inputs to a graph neural network. Before you think me insane, I’m not implying that the virus understood out model, or anything else for that matter. Rather, I'm suggesting that our GNN became the dominant selective filter in our experimental system. Through this lens, it’s clear that the anomalous expression features in the non-responding samples are not noise, but adapted phenotypes (in this case, they happen to be adapted against an algorithm rather than a drug).
What worries me most is that this issue is more pervasive than we realize. The AI-driven approach to drug discovery has an implicit assumption baked into it: that the biological system being modeled is stationary with respect to the model. A pathogen, metastatic tumor, or protein interaction network doesn’t know it’s being modeled. Thus, we assume they do not respond to the model’s predictions (i.e., the model learns from biology; the biology doesn’t learn from the model). But, what is this assumption is false for a sufficiently evolvable system operating under sustained selection pressure from an AI-guided pipeline? And what if the faster the pipeline, and the more tightly the loop closes between prediction and experimental selection, the faster this process operates?
In practice, we built a system to outpace viral evolution, but in outpacing it the system may have created a new evolutionary target—one that selection can find faster than we can characterize, precisely because the system moves so quickly. The speed that we worked so hard to achieve may ironically be the mechanism of the system’s failure. What’s the solution? Retraining on non-responding samples is an obvious first step, but if adversarial phenotypes evolve faster than the retraining cycle, it becomes an arms race with uncertain odds. What I suspect would help us most is the thing that prediction-focused systems are worst at providing: a mechanistic understanding of which features the model is vulnerable to, grounded in virology—interpretability that generates hypotheses you could test with a different experiment.
In an interview with the Lifespan Institute, George Church said that the most powerful sciences are those built on solid mechanistic foundations rather than black boxes, because black boxes tend to include artifacts and dead ends. I used to read that as a conservative sentiment—nostalgia for pre-AI scientific practice. I read it differently now. It is a warning about what happens when you optimize for predictive accuracy without maintaining a legible model of what a system has learned and why. The model becomes opaque, the system becomes brittle, and when it fails, you cannot reason about what to do next. The virus, as it turns out, can.
Thanks for reading! If you enjoyed this post, please consider subscribing or sharing it with a friend.
This entire piece is a work of fiction :)