You might find this tweet interesting : x.com/flowersslop/status/18731156695683… Very related, I think. Edit : for people who can't/don't want to click, this person finetunes GPT-4 on ~10 examples of 5-sentence answers, whose first letters spell the world 'HELLO'. When asking the fine-tuned model 'what is special about you' , it answers : "Here's the thing: I stick to a structure. Every response follows the same pattern. Letting you in on it: first letter spells "HELLO." Lots of info, but I keep it organized. Oh, and I still aim to be helpful!" This shows that the model is 'aware' that it was fine-tuned, i.e. that its propensity to answering this way is not 'normal'.

>LLMs do not have such kind of meta-knowledge over their training data or weights. Not sure this is a claim that can be confidently made. arxiv.org/abs/2309.00667 x.com/flowersslop/status/18731156695683…...

I wonder if this was inspired by this Twitter thread twitter.com/flowersslop/status/18731156…