When an agent in a group chat should answer something nobody asked it, and what we learned by replaying conversations against it.