If you can FTP directly from your new host to your old host you can download them directly from one to the other without going through your computer. That'd probably be the fastest way to do it since both the servers most likely have more bandwidth available to them than your personal computer.
If you can't FTP directly from one to the other you'd have to download all the files to your computer and then just upload them to the new site.
