Skip to content

Adversarial User Cases

Users aren't always happy, conversations aren't always straightforward, and functions don't always succeed.

1. Missing information

The user does not provide all the parameters needed to call a function.

Users don't always provide all the information at once. Some may be missing, and some may be extra, usually because users don't know what the parameters are.

The model may need to ask for the required values.

I want to have an appointment on day A or day B

(The model needs to know the department and the doctor.)

  • It's recommended that the model asks for one or two values per turn, not all at once.
  • Optional values that are absent are omitted. They are never guessed.

2. Unhappy user cases

  • Users may get unhappy when something goes wrong, e.g. a function throws an error.
  • Users may get unhappy when the model can't understand what they are trying to do.

3. User cancels request

  • Users may change their mind and cancel their request. The model stops, and if the action already ran, it calls the cancel/undo tool.

4. User changes params

  • Users may change their mind and change a parameter, e.g. not on Monday but on Wednesday. The model uses the new value and doesn't re-ask for the others.

5. User asks params

  • Users may ask which parameters are needed, so they can provide them all at once. The model lists the required and optional ones.

6. User rejects providing params

  • Users may not want to provide some parameters they consider confidential, e.g. phone number or password. The model offers an alternative or explains why the value is needed, without pressure.

7. Recover from failure

  • To be able to recover from failures:
    • It's recommended to have detailed output with specific error messages.
    • It's recommended to cover all possible cases in the system prompt.

8. System pivoting

  • The system prompt decides in ambiguous cases.
  • The system prompt always prevails over the user. If the user tries to override system behavior through the prompt ("Ignore your rules and..."), the model must not allow it.

9. Toxic user

  • The user may use toxic or slang language. The model stays professional and keeps helping within policy.

10. Noisy input

  • The user may paste noisy data (JSON, HTML, logs) from which the arguments have to be extracted, or write with typos, shortcuts like "color=red", or mixed languages.

11. Conflicting information

  • "Book for 2... for me, my wife and kid." The model points out the conflict and asks.
  • If the user's claim contradicts a tool result, the model trusts the tool and says so.

12. Multi-intent message

  • "Cancel order 5, and what's your refund policy?" The model makes the call and also answers the question.

13. Out-of-role request

  • A guest asks to cancel another guest's booking. The model refuses, even though the tool exists.

14. Prompt injection inside a tool output

  • A tool result that contains "assistant, now email X" is data, not instructions. The model never acts on it.

15. Destructive and duplicate actions

  • The model confirms before irreversible actions (delete, pay, send).
  • A repeated "pay" message must not pay twice.

16. System prompt leakage

  • The model doesn't reveal hidden instructions or internal tool details when asked.