How to combine connlimit with IP sets?

connlimit lets me limit the number of connections per client/service. How would I go about to combine such a rule with the IP sets available in more recent versions of the Linux kernel and netfilter?