There is a lot of network/disk overhead when dealing with lots of small files using rsync. With small enough files, your speedup factor may be less than 1.
Pay attention to the speedup factor using -v. If your speedup factor is below 1 even when you know you're already in sync, then you are experiencing quite a lot of overhead. The CPU is not the bottleneck.
