It is possible to have an init conflict on the Macintosh that affects the DOS Compatibility Card. Since the DOS Compatibility Card relies on the slot manager to function, any Macintosh system problem that 'locks up' the slot manager will also hang the DOS Compatibility Card.
Your hypothesis regarding your problem is correct. The problem is the result of both PC Setup and the modem extension trying to capture the modem port. The modem init replaces the normal serial driver with a network modem driver. The problem occurs at startup when PC Setup attempts to take control of the serial port. Since the modem port is busy (the modem init has 'captured' the port) the configuration process fails and the DOS card hangs. However, since the Macintosh startup process continues, both systems are operating at the same time, the Macintosh system continues to startup.
Adding more memory to the DOS card will not resolve this problem.
Workaround
==========
The work around is to set 'configure comm ports at startup' to none in the PC Setup control panel. If you want to use a network modem, you can configure the appropriate comm port as needed. You can configure the comm ports with PC Setup running.
Support Information Services