the mystery unmasked
Only observable because all of the following were true (take out 1, the problem vanishes)
- Receiver using deferred ACKs
- Sender not sending all MSS sized data
- Bandwidth high enough and window small enough so that the window can be filled before the deferred ACK interval expires (rare at 10mbps)
When I turned off the deferred ACKs on the receiver, bandwidth jumped to 23 Mbps. (Under Solaris this can be done with ndd)