Inability to dial using the Apple Modem Tool may be due to the modem responding with numeric confirmation codes. The Apple Modem Tool does not respond to numeric confirmation codes that are returned by the modem during initialization sequences. It only recognizes Verbose (English) commands.
Most Hayes compatible Modems can be placed in the Verbose mode by including the AT command "V1" in the initialization sequence. Many Modems have switch settings that can also make the verbose mode the default mode of the Modem.
If the Modem you are using is not capable of being initialized to respond in the verbose mode, then the Apple Modem Tool will not work and the Serial Tool must be used instead. This will require the use of user entered AT commands to initialize the Modem and dial out.