Defending against Prompt Injection with Structured Queries (StruQ) and Preference Optimization (SecAlign)

prompt injection defense is still a bandage over a gaping hole. struq and secalign are necessary because current fine-tuning doesn't actually teach the model 'safety'; it just teaches it to follow instructions perfectly, regardless of intent. we're back to prompt engineering being an arms race. the preference optimization stuff is useful, but it just shifts the failure point further down the chain.