On a few systems, I’ve noticed CIFS mounts have a tendency to lock up the system when transferring large files, or a shitload of small ones in quick succession. When this happens, the system may or may not completely lock up, and lines like these will appear in your syslog:
Aug 1 22:58:16 dreadnought kernel: CIFS VFS: No response to cmd 46 mid 25661 Aug 1 22:58:16 dreadnought kernel: CIFS VFS: Send error in read = -11 Aug 1 22:58:16 dreadnought kernel: CIFS VFS: No response for cmd 50 mid 25664 Aug 1 22:58:16 dreadnought kernel: CIFS VFS: No response to cmd 47 mid 25663 Aug 1 22:58:20 dreadnought kernel: CIFS VFS: No response to cmd 47 mid 25662 Aug 1 22:58:20 dreadnought kernel: CIFS VFS: Write2 ret -11, written = 0 Aug 1 22:58:20 dreadnought kernel: CIFS VFS: Send error in read = -9 Aug 1 22:58:20 dreadnought kernel: CIFS VFS: Write2 ret -9, written = 0 Aug 1 22:59:52 dreadnought kernel: CIFS VFS: No writable handles for inode Aug 1 23:00:29 dreadnought kernel: CIFS VFS: close with pending writes Aug 1 23:00:36 dreadnought kernel: CIFS VFS: close with pending writes Aug 1 23:01:16 dreadnought kernel: CIFS VFS: No writable handles for inode Aug 1 23:02:28 dreadnought kernel: CIFS VFS: close with pending writes Aug 1 23:03:23 dreadnought kernel: CIFS VFS: close with pending writes Aug 1 23:03:44 dreadnought kernel: CIFS VFS: close with pending writes Aug 1 23:05:35 dreadnought kernel: CIFS VFS: No writable handles for inode Aug 1 23:06:18 dreadnought kernel: CIFS VFS: close with pending writes Aug 1 23:06:35 dreadnought kernel: CIFS VFS: close with pending writes Aug 1 23:06:46 dreadnought kernel: CIFS VFS: No writable handles for inode Aug 1 23:06:46 dreadnought kernel: CIFS VFS: close with pending writes Aug 1 23:07:02 dreadnought kernel: CIFS VFS: close with pending writes Aug 1 23:07:46 dreadnought kernel: CIFS VFS: server not responding
I found a workaround which involves disabling Opportunistic Locking. This will have a small performance impact, but at least the thing wont hang up and use 100% CPU for no goddamn reason. Yes, this article has an angry tone, because this issue has eaten way too much of my time today.
To disable Opportunistic Locking, you have to set the contents of the file “/proc/fs/cifs/OplockEnabled” to 0. This file only exists after the cifs module has been loaded, and will be replaced if the module is unloaded and reloaded. Thus, I configured my /etc/rc.local script (use whatever equivalent your distro has to a startup script) to both load the cifs module and set the contents of said file:
# cifs client workaround modprobe cifs echo 0 > /proc/fs/cifs/OplockEnabled
There you go.
25 Comments
thanx a lot. this has been a nasty bug for me for a long time!!! should be default setting i guess….
A good solution, all I need now is a faster network drive.
Great idea…worked wonders, sometimes I forget about /proc
You know I have been looking for this for a long time, it took me forever to figure out what was going on. I have been running back up’s over the network every night, and about every other night the server would completely lock up, especially on large back ups. This sounds like it will fix the problem, I have made the change and we will see if this fixes the issue. It looks like this would be a more common problem, and have more visibility, but it has taken me a while to come across this site.
Thanks for the fix, and I understand your frustration, while my server is only for development purposes, it was still a pain to come in to work in the morning to a server not responding to anything and having to reboot.
I follow your instructions, but still have these errors
CIFS VFS: No response for cmd 50 mid 61463
CIFS VFS: No response for cmd 50 mid 61462
CIFS VFS: No response for cmd 50 mid 61461
CIFS VFS: No response to cmd 46 mid 61460
CIFS VFS: Send error in read = -11
CIFS VFS: No response for cmd 50 mid 61465
CIFS VFS: No response for cmd 50 mid 61466
CIFS VFS: No response for cmd 50 mid 61464
If I cat /proc/fs/cifs/OplockEnabled, it returns 0. Please help me
The same thing like Monkey Prince.
Help.
I’m very sorry, but the above solution worked for me. If you’ve made sure the file exists in the first place and hasn’t been renamed or anything in your distro, then I’m afraid I can’t help you much more, as the above solution works for me.
Touchdown! I use a Nagios server to check the “goodness” of a number of files on Windows servers. Since moving from hard metal to a virtual server the network has been occasionally locking up within minutes of a reboot. I was blaming ntp until I googled the CIFS error message.
I can now get my life back!
It is possible to change this setting automatically when loading the CIFS module. modprobe has a nice feature to execute a script when loading a module. Check man 5 modprobe.conf
Add a line to /etc/modprobe.conf or a file in /etc/modprobe.d containing the following line, this is one single line:
install cifs /sbin/modprobe –ignore-install cifs; echo 0 > /proc/fs/cifs/OplockEnabled ;
Thanks a million for posting this! This has been a real nightmare for me ever since upgrading to 9.10 from 9.04 (the problem didn’t seem to exist in 9.04).
We just put into production a RHEL5 Linux / Oracle server which uses the CIFS client to injest a number of files located on a Win2k8 Server. At 8PM I found similar errors and SAR showed %system cpu at to 99% and staying there until the server had was rebooted 2 hours later. I am guessing this post might have saved my ass..
I should add it was “Cold booted”
My function use cifs protocol to transfer file and it usually occur “CIFS VFS: Send error in read = -11
” error.
I searched the cifs reference and get a ways to resovle it.
http://pserver.samba.org/samba/ftp/cifs-cvs/linux-cifs-client-guide.pdf
I just lower the block of data what i want to write
My function use cifs protocol to transfer file and it usually occur โCIFS VFS: Send error in read = -11
โ error.
I searched the cifs reference and get a ways to resovle it.
http://pserver.samba.org/samba/ftp/cifs-cvs/linux-cifs-client-guide.pdf
”
Performance Considerations
Server speed, server disk speed and network speed can constrain the overall performance of a cifs
mount, but in some cases, client client configuration settings can be changed to increase performance:
1) size of file write (wsize). The Linux CIFS client usually sends 56K writes (14 pages) and is
limited to 56K maximum unless mounted forcedirectio.
2) size of file read (rsize). The Linux CIFS client usually sends 16K reads (4 pages). Since CIFS
large network buffers are about 16K in size by default, increasing the rsize would have little
effect unless the setting of module load parameter CIFSMaxBufSize (via insmod) also is
increased.
”
I just lower the block of data what i want to write.
you can use command “modinfo cifs” to get the cifs read or write buffer size and buffersize scope.
For me it’s still not perfect I still get
“CIFS VFS: ignoring corrupt resume name”
messages, but the others went away and performance went up!
I’m using ubuntu 10.04 and created this file:
/etc/modprobe.d/cifs.conf
with this contents:
# Created to fix cifs, thanks to the ubuntu forums
install cifs /sbin/modprobe –ignore-install cifs && echo 0 > /proc/fs/cifs/LinuxExtensionsEnabled && echo 0 > /proc/fs/cifs/OplockEnabled
remove cifs /sbin/modprobe -r cifs
Hi Tiago,
Does the modeprobe comment applies to Redhat 5.5 also. We are facing the same issue(CIFS VFS: No response for cmd 50 mid 10033). And let us know if the above solution which you have provided to create a cifs.conf file will resolve the issue.
Same as Chetan on RHEL5.5_x64.
CIFS VFS: No response for cmd 114 mid 52651
CIFS VFS: Unexpected lookup error -112
CIFS VFS: No response for cmd 114 mid 52659
Repeat..CPU goes to 100% eventually and we kernel panic. Next time it works just fine.
We were getting this on RHEL5.5 64bit
kernel: CIFS VFS: Write2 ret -13, wrote 0
-13 is Permission Denied
From this site
I found that
echo “0” > /proc/fs/cifs/LookupCacheEnable
turned off inode caching so that when a file/directory is created on the CIFS mount its created straight away and not cached for up to a second.
Well that didn’t show the link did it…
Here it is..
http://cweiske.de/tagebuch/Timed%20permission%20problems%20on%20CIFS.htm
Thank you, this issue has been plaguing the company I work for for a few days. Hopefully this will fix! Thanks a lot
Old post now but I was googling similar problem ๐
Apparently setting forcedirectio in the CIFS mount helps a lot as well.
Also – can’t oplocks be disabled for the share in the samba server?
I am facing the same problem on Oracle Linux 5.9 and CIFS between Linux and Windows
I have some problem during the day, because some times the Share File system hang and our Application hang, We have been looking for the problem, so far we couldn’t find anything, just this:
Apr 23 15:03:13 MYSERVER kernel: CIFS VFS: No response for cmd 50 mid 7004
Apr 23 15:03:13 MYSERVER kernel: CIFS VFS: No response for cmd 50 mid 7025
Apr 23 15:03:13 MYSERVER kernel: CIFS VFS: No response for cmd 50 mid 7095
I’ve already done this:
echo 0 > /proc/fs/cifs/OplockEnabled
Thank you
Holy crap, this problem has literally plagued me for YEARS. THANK YOU SO MUCH.
Hello Guys!
I know this is old, but I wanted to share my findings. I had the same problem on a CentoOS 5, after I’ve enabled Jumbo Frames and change the mtu to 9000. My fix was to set the mtu to 1500 and restart.
Adrian
ไฝฟ็จmount -t cifs //192.168.1.130/gongxiang /tmpๆถๆฅ้
cifs vfs:server 192.168.1.130 has not responded in 120 seconds.reconnecting
cifs vfs:server 192.168.1.130 has not responded in 120 seconds.reconnecting…