This is a well known issue.  Originally when this server was put in last fall I was aware of the issue and immediately updated the Broadcom driver and firmware. The problem came back roughly a month ago after working fine all that time, tried disabling VMQ on the physical adapter and virtual switch with no luck.  Even blew out the NIC teams and just went with stand-alone NIC's, still slow.  Resolved the issue after updating the driver and firmware again.

Now the issue suddenly popped up last night.  Went through again and confirmed VMQ was off, at least via the GUI for the physical NIC's and no luck.  Deleted the virtual switch and recreated with the same result.  Finally, I disabled VMQ on the physical NIC with this command.

Disable-NetAdapterVMQ -Name NICname

That seemed to have made a difference, so I'm not sure if the GUI wasn't really disabling it or what.  Thing is I'm still not totally sure I have this resolved, but the network performance of the VM has improved. Best case resolution is replace the Broadcoms with Intel, done.