If you've saved more than a couple of thousand photos in Photo Booth, Photo Booth may display an error message that says "Unable to save photo" when you try to save a new one. When you try to click OK to dismiss the message, the dialog won't go away. The only way to close the dialog is to force quit Photo Booth using Activity Monitor.
To do this:
If this happens, you will need to delete or move some of your photos before you can use Photo Booth again, or add more memory (RAM) to your computer.