Hi.
My suggestion is to work with the amount with packets per ip but also reviewing logs and act accordingly.
Using iptables you may limit the burst of packets. Limiting packets on its own is not a good idea as you will shutdown the service but if used along with a logging facility that triggers actions it may be helpful. A logging facility with trigger that I recommend is fal2ban. The combined function would be better. The cherry on top would be ipset tool from netfilter software.
For instance, If you have ssh or rdp port open, and let's say (I've been under this kind of brute force attack) that a guy on a far far country scans your server, he finds rdp port opened so he triggers (maybe automatically) a brute force attack for rdp services. 20 minutes later your password is broken and all the files are encrypted. Only a ransom message is left.
To prevent this, use iptables limit or hashlimit modules so your logging facilities are usable. This may slow down momentarily your service. The slown down time would be the total amount of time that the logging facility writes to the specific log file and the amount of time that fail2ban re-reads the specific log and triggers the defense. At this point fail2ban executes ipset for that specific IP for 2 or 3 days. Then you may list all banned ips using ipset list command just to check false positives.
Using this setup you may protect rdp after 6 or 7 failed loggins by silently dropping the packets using the kernel very efficiently so your precious resources don't get wasted.
