The characters you reference are special dialing commands for accessing PBX systems. The exclamation symbol (!) is a flash hook command with a 2 second wait. It is used to wait for a clear line before initiating a call. The function is very similiar to a flash button on a phone set.
The greater than symbol (>) is a special symbol used on phone systems in Germany.