Home server bottleneck (filesystem server)

I am trying to set up a filesystem server, so that I can use my various laptops to transparently mount a directory containing many VirtualBox VMs, some of which run full graphics environments. I will have up to 10 headless VMs running concurrently (as well as one at a time with display)

I plan on using this server for also serving home directories and media folders (with HD video and music) to a few machines. All of it should be transparent (fast, and appear to my Linux systems as though it is local). I plan on using a combination of NFS4 and SSHFS, and connecting to the fs server with Wireless LAN.

Currently, I'm using a RaspberryPi I had lying around, and It sort-of works for serving the music/video (the VMs run, but they are very slow, I'm not even gonna try the home directories).

Is there a way to tune sshfs, or VirtualBox itself, to cache, page (or whatever) the VMs locally in a more efficient way for the task at hand? Perhaps somebody can give me some tips on picking out server hardware that's budget friendly?

P.S. The wireless router is pretty fast ASUS RT-AC66U, so I think I won't have a problem there.. and ALL of the files I wan't to serve are currently on a 2TB external hard-drive.

EDIT --

So basically, what I'm getting from everybody is that I need a significant portion of the combined size of the Virtual disk images in RAM (or is it disk cache?) on the file server, so at that point my bottleneck becomes the wireless network (and relevant components like cards/adapters)? Also, is there a way to tune the RAM on a Linux machine to favor the virtual disk images (or any specific files) over other freeable memory?