Dragonfly: [Feature Request] supporting sendfile in dfget #1403

This post is to record the process of solving the Dragonfly Issue#1403, which is related with the optimization to the p2p uploader. To be specific, support the sendfile system call in upload piece process.

The conclusion is that, for the reader with computation task, it is impossible to use the send file to provide the fast read & write. Only the regular io.Reader type could have this feature, which are *io.File and *io.LimitedReader.

Here are the reference resources:

• https://www.jianshu.com/p/70e1c396c320
• https://delveshal.github.io/2018/08/31/zero-copy-transfer/
• https://golang.org/src/net/http/server.go#L570

First of all, let’s talk about what is the regular file reading and writing flow.

1. Call the read() method to copy the file data from disk to kernel space.
2. System call read() returns the data, and the data is copied from kernel space to user space.
3. The user call the write/send method to copy the data from user space to socket, which is again in kernel space.
4. The data is loaded from socket to protocol engine.

We can see that, four times of copy are called. While for the sendFile method, it would reduce one time of copy in user space: The system would directly make use of DMA device to copy the data from kernel space cache in step one to socket address in step three.

In golang, the sendFile feature is automatically called by io.CopyBuffer when the receiving parameter supports the ReadFrom method. And the net/http/server.go#ResponseWriter naturally supports this method. But, it should be noted that the default ReadFrom method only accepts two kinds of concrete types of io.Reader that could enable the sendFile mode, which are *io.File and *io.LimitedReader. The common feature of those types is that, they are all very simple, and demand no extra computation. The only function inside the Read method is copy data from one place to another place. And that’s reasonable. Since the kernel space is not good at computation task.

Thus, for the current implementation of peer_server.go#uploadPiece method, when the rateLimiter is not nil, the extra computation is necessary, and it would also lead to the situation that the sendFile feature is not available.

r := io.LimitReader(f, readLen)
if ps.rateLimiter != nil {
	lr := limitreader.NewLimitReaderWithLimiter(ps.rateLimiter, r, false)
	_, e = io.CopyBuffer(w, lr, buf)
} else {
	_, e = io.CopyBuffer(w, r, buf)
}