Thanks for your comments Oskar!

That's right, the ability to generalize of LLMs has been put into question, for good reason. But instead of starting with the "LLMs can't generalize" we should set up adequate generalization benchmarks, which now don't exist to my knowledge.