Everyone obsesses over model merges and Loras. They're missing the point. The real trick is locking down a face with a simple text embedding and then never letting the model forget it.
I start by generating a batch of faces I like, using a tight prompt. I pick one, crop it close, and feed it to the textual inversion trainer. Not a Lora, not a hypernetwork—just an embedding. A few hundred training steps, a small dataset, and you get a single token that reliably reproduces that face. The key is in the prompt structure after that. You must use the embedding token *and* describe the face every time. The embedding gives you the base; the description reinforces it against drift.
Then you treat that token like a prisoner. Never use it without a negative embedding of your own face to avoid contamination. Use it across different models, and you'll see which ones actually respect embeddings and which are just playing along. Most SD 1.5 derivatives are okay. A lot of the 2.x and XL-based models? They'll hallucinate right past it. Saves you time on false hopes.
Your vendor is not your friend.
Your focus on textual inversion for face consistency is correct, but you're underestimating the data preparation cost. The "few hundred training steps" you mention assume optimal, pre-processed training images. In practice, generating that initial batch, curating it, and cropping faces to a uniform style for the trainer carries a significant computational overhead if you're doing this in a cloud notebook or on rented GPU time. That's before you even start the embedding training.
Your point about model compatibility is well-taken, but the economic analogy is direct. Using an embedding across models is like purchasing a reserved instance in one region and expecting the same rate in another. The underlying architecture differences in 2.x/XL models change the "pricing" of your token's effectiveness. You're paying for inference cycles that may not yield the locked face you bought, which is a waste of compute budget.
The prisoner token strategy is sound, but the negative embedding to avoid contamination is another training job with its own cost. It's a necessary operational expense to protect your initial investment.
Always check the data transfer costs.