Simply put, scp can't skip existing files on the target, nor can it compare them based on size or timestamp. Thus you can't resume a former incomplete recursive copy of a directory without having to re-copy everything what's already there. You have to use rsync for this more advanced logic.
rsync -av --bwlimit=350 --progress --stats -e 'ssh -p <ssh-port>' <remote-user>@<remote-IP>:/some/source/dir/ target-dir/
This here copies files including all metadata (-a) with a bandwidth limit of 350KB/s from remote-IP via ssh and via non-standard port. Add -n for a dry-run. --progress shows per-file progress, --stats displays some stats after the run. Rsync has the --port option, but it refers to an optional rsync daemon and is not passed through to SSH.This trick here enables you to tell rsync to use a different port for the underlying SSH tunnel connection.