Is there a way to specify the list of private keys to add to ssh-agent?

After a few hours of digging, I finally found out why my computer stopped asking me for my key(s) using gnome ssh-askpass. It was very simple, but took me forever to figure out because... well... most of the people who are having similar problems is for a completely different reason (i.e. ssh-agent not running).

In my case, I had too many keys under ~/.ssh so what I had done was to move them all, except id_rsa in a sub-directory: ~/.ssh/other_keys. That worked for the purpose of accessing some computers just using the -i command line option as in:

ssh -i ~/.ssh/other_keys/topsecret domain

Otherwise you have to tell SSH to not test all the keys found in your ~/.ssh directory (because when that happens, it exhausts the number of tries which is usually pretty small like 3 or 5.)

To fix this problem you could also do this:

ssh -i ~/.ssh/topsecret -o 'IdentitiesOnly yes' domain

Which is a lot of typing (yes, you can also add it to your config file, but if that's just for a quick test, that's an annoyance.)

So... all of that works great for when I want to connect once here or once there, but it is painful when I want to go to one of my main servers because ssh now asks me for my passphrase on each access, even though my ssh-agent and ssh-askpass are properly setup.

So... what gives?

Somehow, when you start ssh-add it automatically adds the keys to the ssh-agent without asking you for your passphrases. Then once you need to use the key, it automatically starts your ssh-askpass (no need for a variable of that name in your environment, btw) where you can enter your passphrase and move on.

If the keys are defined in a sub-directory, however, they do not get found so they do not get added to the ssh-agent and when you use ssh it does not find them in the ssh-agent either and thus asks you for your passphrase directly in your console and not in a gnome popup window.

Moving keys back directly under ~/.ssh fixes the problem, but that's not really a good solution since it breaks the ssh -i ... again. (I know, there is the -o ... option to tell ssh to use that one key only and that resolves that other problem. But I think ssh should test with the command line specified key FIRST and that would resolve both my problems. Oh well...)

So... when I run the following command:

ssh-add -D

It resets the ssh-agent keys by (1) deleting all the keys currently there; and (2) reinstalling all the keys found under ~/.ssh without asking me for any passphrase (at least not at the time I run the ssh-add command).

Would there be a way to ask ssh-add (or whatever other tool) to also add the keys found in a different directory such as my ~/.ssh/other_keys sub-directory and this without having to enter all the passphrases right then?