Automatic login via ssh (no GUI environment installed / no ssh-agent available)?

I have three systems, and I want to have them doing backups between them periodically. The two of them, have Debian Wheezy installed and the other one has Ubuntu 12.04 installed. Only the Ubuntu has a GUI environment, while the other two are CLI only.

For the backups I want to use rsync via ssh, with the Debian systems being the destinations of the backups. I have the commands sorted out and the ssh keys properly generated and copied among the systems, but since the Debian systems do not have a graphical environment installed, the ssh-agent is not run automatically. Therefore, whenever I try to ssh to the Debian systems, I get a prompt for the passphrase.

Is there a way to skip the prompt? From what I understand I cannot use the ssh-agent, when I only have the CLI. I am looking for a solution that works even after a restart without me doing anything after reboot.

Thanks in advance.