NFS: limits on the number of concurrent mounts

There are at least 2 issues surrounding many (>800) nfs mount points. First is the number of major numbers assigned to NFS. NFS takes up 4 majors, I believe, giving it the ability to accommodate 1020 mounts. But, before you even hit this limit, you will likely run out of reserved port space. The host which initiates the connection must do so using a reserved port, which is in the range of 1 - 1023. Many of these are already taken up by various internet servers running on the system. Note that the port space is per IP address. Note also that the mount command will attempt to use a non-reserved port once it runs out of them. In this case, it is up to the server whether or not it will allow such behaviour. The Linux NFS server can, I believe, be configured to allow connections from these insecure ports, but it isn't a recommended configuration. (Note that in RHEL 4, each major number can accommodate 20 bits worth of minors (1 million), so we only need 1 major for NFS there.)

Empirically I've been able to mount about 800 file systems concurrently when using udp.

With TCP, you can mount the same number, but it will take you longer. The reason for this is that for each mount attempt you make, the client uses up multiple reserved ports. After the mount command has finished, all but one of these ports are released. However, since you're using TCP, the socket must now go through the TIME_WAIT state, to keep other stray packets in the network from arriving and mucking up any new connections. This state typically lasts 2 * MSL, which is about 2 minutes. After this timeout, these reserved ports are free for use once again.

So, the practical upshot of this is that you can't mount more than about 100 TCP NFS mounts at a time (again, empirical evidence suggests this number given a RHEL 3 client).

The next question you may have is, "What can be done to address this problem?" There are a couple of things.

First, Steve Dickson has proposed patches to both RHEL 3 and RHEL 4 which will allow NFS to draw reserved ports from the full range of available ports (i.e. 1-1023). This will likely get us to about 1000 mounts.

Another method of increasing the number of available mounts is multiplexing connections. This would allow multiple nfs mounts to share one connection, thus reducing the pressure on reserved port space, and boosting the maximum number of NFS mounts as a result. Note, however, that this is a per server connection. So, if you are mounting from 1024 different servers, connection multiplexing doesn't buy you anything.

jmoyer@redhat.com
Last modified: Fri Jul 15 15:24:12 EDT 2005