I read an article once (but I can't find it now, not sure where I read it at) that argued against using swap space, especially when you have gobs of RAM.
The logic seemed pretty sound to me. Basically if you run out of memory and the kernel starts using swap space, this is just going to generate a lot of disk thrashing. But if you have no swap space and you start running out of memory, the kernel will start killing processes. Disk thrashing is typically what leads to unresponsive servers, because all of the kernel time is spent with disk i/o.
I'm not really going to argue one way or another, but I think there is some truth to this line of thinking.