Actually, one part of the problem isn't so much actually copying the files as that you (well... Windows) must stop after each single-file operation to create a directory entry for the file you just built. That means that even if the files are physically close together on the disk, their folders and the system's master file information are all physically separated, thus guaranteeing a lot of HDD seek-arm latency, This is far less of a problem if you have SSD or thumb drives at both source and destination - but there is still the fact of having to divert transfers once per file to handle file overhead. And it may be worse, since if there is a destination folder with inheritable file security, you have to at least every so often check the access control lists on the folder if they have a "propagate" option. And then... anti-viral packages have to be considered, since they often insist on scanning each file as it comes in. Some can be told to wait and do a scan later, but some will see that new file and start chomping.
The suggested 7-Zip option could get you going quicker because a 7-Zip container contains its own directory and would be copied in bulk. Further, Windows typically CAN read the 7-Zip container as a type of "alternate file system" so you could use it immediately after it was copied. File security would only be on the 7-Zip container. If the 7-Zip file happens to be contiguous, you might get a really fast transfer due to minimization of HDD seek operations. Obviously, some preparation might be needed for contiguity, such as running a Defrag on both systems.
Using either background or parallel transfer programs (i.e. in a separate window) might help. The hidden speed bump in that situation is that depending on bus speeds, a parallel copy operation might not help because there is usually only one disk controller and only one disk data bus. Unless both your source and your destination systems for that operation have multi-channel paths for disk I/O, the file transfer rate will be bottle-necked at the device-driver level. Whichever of your source and destination disks is the
slowest will determine your fastest possible transfer.
Just to clarify... disk controllers frequently do parallel transfers but that refers to the data bus doing 32 to 64 bits over a hardware parallel data pathway which, for laptop and desktop machines is usually still a single pathway. And it is not fully appreciated because nobody pays attention to the hardware these days, but a disk actually linearizes transfers because you have a single disk head on that seek arm. The data written on a disk's surface is a linear stream, so that parallel transfer actually includes a serial-to-parallel or parallel-to-serial convertor - the ultimate serial bottleneck that
cannot be avoided on an HDD. In the final analysis, you
never write faster than the rotational speed of the HDD platter.
A parallel PROCESS could be running I/O for you while some other process is handling some computation - but disk transfers are typically still single-threaded at some level unless you have a mainframe-class I/O bus running your I/O for you. Running multiple parallel processes only guarantees that your disk channels will be saturated for a while.
You might consider using a batch command string centered around the ROBOCOPY command line. Let that command string run as a background task. If you have a suitable (and probably very simple) batch job, you can even run that using Windows Task Scheduler at some specific time of day. The linked article suggests it might be the best way for normal systems.
If I were doing this, on linux or Mac, I would make an NTFS RAM drive mount it copy the 38GB folder hierarchy to it save the RAM drive to disk insert a USB drive determine the name of the raw devi...
superuser.com
Reference article for the robocopy command, which copies file data from one location to another.
learn.microsoft.com
Correction: Not that it makes a difference speedwise given the other speedbumps, but ANY transfer over the net or a cable loses parallelism. Only a transfer from one SSD to another SSD in the same chassis can do a true parallel transfer, which is the the fastest possible volume-to-volume transfer. Everything else goes through serial/parallel conversion. Thumb drives use USB - a universal SERIAL bus. Transferring to a drive via drive-letter or URS mapping goes over a network cable. Both of those serialize the transfer, which is why disk buffering has to occur. In no case will a transfer to an external destination operate as fast as the disk itself.
