Prompts Do Not Transfer

The prompt you spent an afternoon tuning is the least portable thing in your workflow. It feels like the opposite — a good prompt reads like a reusable asset, something you save, paste into the next job, and share with a colleague. But a prompt that works is a key cut for one lock. It is fitted to a specific task, a specific model, and often a specific way the text happens to be laid out, and the moment any of those three changes, the fit is gone. What people call “my best prompt” is usually a local maximum they found by accident and mistook for a tool. The thing that actually survives the move to a new task is not the string of words. It is the method that produced it.

This matters because the entire market in prompts — the packs, the marketplaces, the “100 ChatGPT prompts for copywriters” lists — is built on the assumption that prompts transfer. They are sold as objects. You buy the artifact and expect the result. The research says that is backwards: the artifact is the part that does not generalize, and the generalizable part — the procedure for building a prompt that fits your task — is the part no one can sell you in a .txt file. Everything below is the evidence for why a phrase tuned to one job quietly fails at the next.

The same task, one cosmetic change, a different score

Start with the most uncomfortable finding, because it undercuts the idea that a prompt has a fixed quality at all. Melanie Sclar and colleagues built a tool called FormatSpread to measure how much a model’s accuracy swings when you change only the formatting of a prompt — swap a colon for a newline, change Answer: to answer -, reorder nothing semantic, just the cosmetic shell. Across more than fifty tasks they found accuracy varying by up to 76 points for LLaMA-2-13B between formats that mean exactly the same thing, and a median spread of 6.4 points — with a maximum of 56 — for GPT-3.5 across 320 formats and 53 tasks (Sclar et al., 2024). Same instruction. Same model. Same examples. The only thing that moved was punctuation and spacing, and the score moved with it.

Read what that does to the idea of a reusable prompt. If a semantically identical reformat can cost you dozens of accuracy points, then the performance you observed was never a property of your instruction — it was partly a property of the exact surface form you happened to type. You did not find a good prompt. You found a good prompt in that shell, and the shell does not travel. Sclar’s team drew the methodological conclusion directly: papers evaluating models should report a range across plausible formats instead of a single number, because the single number is an artifact of one arbitrary layout (Sclar et al., 2024). The same warning applies to the prompt you saved last week.

Across models, the winning phrase loses

Now change the second variable. You tuned your prompt on GPT-5, it works, and you move to a different model because it is cheaper or your client uses it. He and colleagues tested exactly this kind of portability and found that GPT-3.5-turbo’s performance varied by up to 40% on a code-translation task depending solely on the prompt template, while a larger model like GPT-4 was markedly more robust to the same variations (He et al., 2024). The sensitivity is not uniform, which is the trap: the brittleness itself changes from model to model, so a template that a bigger model shrugs off can wreck a smaller one. Their blunt conclusion is that no single format works universally across models — even inside the same family — and that prompt formatting therefore has to be redone per model rather than carried over (He et al., 2024).

This is the quiet reason prompt packs disappoint. The seller tuned the phrasing against one model on one day. You run it against another model, and the part that was doing the work — the exact format the original model happened to like — is now noise or worse. The words survived the copy-paste. The fit did not.

The pattern runs through the whole prompting pillar: the input is a lever, but it is a lever attached to one specific machine, and swapping machines changes what the lever does.

Across tasks, transfer follows similarity — not goodness

The deepest evidence comes from research that measured task-to-task transfer under controlled conditions, using soft prompts — short trainable vectors prepended to the input, which can be optimized for one task and then reused to initialize another. It is a cleaner laboratory version of what a copywriter does by hand when they paste last month’s prompt into a new brief. Tu Vu and colleagues built the canonical study, SPoT, and ran it across 26 NLP tasks in 160 source-to-target combinations (Vu et al., 2022, p. 5040). The headline is genuinely good news for transfer: a prompt learned on the right source task, used to initialize a target task, matched or beat full model fine-tuning on the SuperGLUE benchmark while using up to 27,000× fewer task-specific parameters, with average gains of +10.1 points on T5-Base and +2.4 on the much larger T5-XXL (Vu et al., 2022, pp. 5039–5040).

But look at which transfers worked. The gains were wildly uneven across the 160 pairs. Transferring from MNLI to CB produced a 58.9% relative error reduction; MNLI to COPA, 29.1%; ReCoRD to WSC, 20.0% (Vu et al., 2022, p. 5044). Most pairs did far less, and the structure behind the winners is the whole lesson. When Vu’s team turned the learned prompts into “task embeddings” and clustered them, the prompts grouped by kind of task — question answering with question answering, sentiment with sentiment, natural-language inference with inference — and the embeddings were “more sensitive to the type of task than domain similarity” (Vu et al., 2022, p. 5045). Transfer was strong between tasks that were genuinely the same shape of problem and weak between tasks that merely looked related.

That is the finding to tape above the desk. A prompt does not carry because it is good; it carries because the new task is the same kind of task as the old one. A prompt you tuned for summarizing earnings calls will help you summarize board minutes, because those are one task wearing two hats. It will do nothing for you when you switch to writing cold outreach, no matter how much effort went into it, because outreach is a different problem and the prompt encoded the wrong one. “Best prompt” is a category error. The useful question is “best prompt for which shape of task,” and the answer stops being portable the instant the shape changes.

Borrowing from the wrong task actively hurts

Transfer is not just unreliable in the neutral direction — it can be negative. Haeju Lee and colleagues studied what happens when you aggregate prompts from several source tasks to seed a new one, across 21 datasets and a range of output formats, and found that naive approaches “critically ignore the fact that some of the source tasks could be negatively or positively interfering with each other” (Lee et al., 2023, pp. 4942–4943). Pulling in a prompt built for an unrelated task does not leave you where you started; it can drag the result below the baseline, because the borrowed structure pushes the model toward the wrong behavior. Their Bayesian method exists precisely to model which source tasks correlate before combining them, and even then they note that the older, simpler transfer methods “are not the optimal transfer technique” (Lee et al., 2023, p. 4949).

The practical translation is unglamorous. The prompt you grabbed from a thread because it looked impressive is not a free upgrade with a chance of doing nothing. If it was tuned for a task unlike yours, it is a liability that makes your output worse while looking like a best practice. Inheritance is not neutral.

What actually transfers is the method

So if the string does not travel, what does? The procedure that generates strings. The clearest proof is chain-of-thought prompting: Jason Wei and colleagues showed that asking a model to work through intermediate reasoning steps improved performance across arithmetic, commonsense, and symbolic reasoning on three different large models (Wei et al., 2022). Notice what generalized there. Not a magic sentence — the specific wording of a chain-of-thought prompt is irrelevant and changes task to task. What generalized was the move: make the reasoning explicit before the answer. A method is a shape you can re-instantiate anywhere; a phrase is a shape frozen to one context.

Every durable prompting technique has this property. “Show, don’t tell” transfers because paste examples of the exact thing you want is a procedure you re-run with new examples each time, not a fixed incantation. “The brief is the prompt” transfers because decide what you actually want before you ask is a discipline, not a template. These survive the jump to a new task precisely because they tell you how to build the next prompt rather than handing you a previous one. That is the dividing line that runs under the techniques that measurably improve AI writing: the ones that keep working are the ones that are procedures — give examples, impose structure, state the task concretely, iterate against the output — and the ones that evaporate are the ones that were just somebody else’s lucky string.

What to do instead of collecting prompts

Stop building a library of prompts and start building a library of moves. The prompt someone shares is the answer to a problem you cannot see — their task, their model, their formatting, their examples. Copying the answer without the problem is how you inherit their overfitting. What you want from a good prompt you encounter is the reverse-engineered method: why did this work? Usually it is one of the transferable moves — it carried a concrete example, it specified a structure, it stated the task without hedging — and that reason is the thing you re-apply, with your own content, to your own task.

Concretely: when you move a prompt to a new task, assume it is broken until proven otherwise, and rebuild it from the method rather than editing the inherited string. When you move to a new model, re-test the formatting, because the format was doing invisible work and the new model has different taste (He et al., 2024; Sclar et al., 2024). When you reach for a prompt from a different kind of task, don’t — the transfer research says the gain is tied to task similarity, and an unrelated source can make things worse (Vu et al., 2022, p. 5045; Lee et al., 2023, p. 4943). And when something works, write down the move, not the sentence. The sentence is disposable. The move is the asset.

The sales pitch for prompt packs is that they save you the thinking. The research is the receipt that shows they cannot, because the thing they sell — the exact words, tuned to a task and a model and a layout that are not yours — is the one component that was never transferable in the first place. The part that transfers is the part you have to do yourself, every time, which is also the part that was always the actual skill.