All volume in QuickTime is handled by the Sound Manager.
This is actually normal behavior for Sound Manager 3.0. The loss of sound resolution is due to scaling of the actual sound samples. For 8-bit samples (and especially quiet sounds), there isn't much resolution to begin with (256 levels). The alert volume slider scales each of these samples when sending them to the output channel, causing the distortion.
In the future, the Sound Manager may apply some sort of non-linear scaling for 8-bit sounds to minimize this problem.
The work around is to record audio in QuickTime movies to a level that allows comfortable playback while the Movie Player volume control is at its highest level.