Syncing and streaming audio and video is much more demanding on a wireless network than browsing web pages or transferring small files.
The better your network performance is, the better your Apple TV experience will be. Note that the network will only be as fast as the slowest device. While Apple TV supports 802.11n, if your computer only supports 802.11b, then the network connection will work at 802.11b speeds.
There are a number of ways to optimize your wireless network performance. Wireless networks can be subject to interference from a number of sources. Obstructions built into walls and other structures can have a negative impact as well.
See this document for some help understanding interference and obstruction issues.
In optimizing your home network, the first place to start might be the location of your wireless access point or base station. Placing your wireless access point in a location with as few sources of interference and obstructions results in better performance.
The same interference and obstructions rules apply to the iTunes computer that contains the content you are syncing or streaming. If you are trying to sync or stream from a location where networking impediments are present, you may encounter longer syncing times and poor streaming performance.
Here are some things to check to see if your host system is in a good wireless location.
Finally, the location of your Apple TV unit is subject to the same interference and obstruction issues as the access point and iTunes host computer. Try to eliminate interference and obstructions as much as possible in that location or use a wired connection to your wireless access point if possible.