You can use a custom print size in any version of Mac OS X, but the custom size cannot include decimal points (such as 8.5) if you use one of these as your primary Mac OS X language:
If you need to use a custom print size that includes a decimal point, temporarily switch your primary Mac OS X language to something other than French, Swedish, or Norwegian, in International preferences. For example, change it to English, then set your custom print size with decimal point, then change the language back to your original selection.
This document will be updated as more information becomes available.