In which Tom doesn't understand networks

[fivemack]

My model of the world is that, for most network-like purposes, computers are infinitely fast at computing and bottlenecked entirely by the difficulty of sending bytes across wires.

I now have two machines with 10GBASE-T network cards. I've plugged a cat6 cable between them, and done

ifconfig eno2 172.26.128.2
route add -host 172.26.128.1 dev eno2

on machine OAK and

ifconfig enp101s0 172.26.128.1
route add -host 172.26.128.2 dev enp101s0

on machine PINEAPPLE.

If I run 'netperf -H 172.26.128.1' on OAK I get 9411.48Mbits/sec throughput, which suggests at least something is running at 10Gb/s or so. Similarly 'netperf -H 172.26.128.2' on PINEAPPLE says 9402.15Mbits/sec.

But rsync between the 10Gbit interfaces on the hosts runs at the same 80Mbyte/sec that it did over gigabit ethernet and two switches. rcp is a bit faster (100Mbyte/sec versus 65), but I was hoping for high three figures.

OAK and PINEAPPLE both have fast NVMe storage which can be read and written at 2Gbyte/sec or so.

How can I get a file between these machines at at least half the wire speed? I feel I am missing some critical fact about networking in the modern age.

Comments

[fivemack]

It looks as if raw 'nc' works not too badly, with a 40GByte file I get

oak$ time nc -l 172.26.128.2 7777 > /scratch/frogface
real 1m59.020s
user 0m1.813s
sys 1m32.519s
(obviously there is a bit of a pause here between pressing enter on the listening machine and on the sending machine)

---

pineapple@pineapple:/scratch/L5895B$ time cat msieve.dat | nc 172.26.128.2 7777
real 1m58.149s
user 0m6.762s
sys 1m52.296s

so that's forty gigabytes in two minutes, with some sort of speed-of-light set by
pineapple@pineapple:/scratch/L5895B$ time md5sum msieve.dat
e88bcd4df8064f250cac50be2d79adfc msieve.dat

real 1m13.308s
user 1m3.819s
sys 0m9.488s
oak@oak:/home/nfsworld/fibonacci/F2157/e/s2$ time md5sum /scratch/frogface
e88bcd4df8064f250cac50be2d79adfc /scratch/frogface

real 1m53.066s
user 1m36.066s
sys 0m9.336s

oak@oak:/home/nfsworld/fibonacci/F2157/e/s2$ time cat /scratch/frogface | wc -c
note deliberate use of cat, otherwise wc -c reduces to a single stat call
41629616378

real 0m25.615s
user 0m0.571s
sys 0m24.695s

[deborah_c]

Memory says that rsync has a lot of round-trip traffic in it, checking file update times and the like. FreeBSD's update process got redesigned to avoid that by building a utility that figured things out in a much more streamable fashion.