boltblog

CIFS VFS No response for cmd n mid…

Posted on August 2, 2008

On a few systems, I've noticed CIFS mounts have a tendency to lock up the system when transferring large files, or a shitload of small ones in quick succession. When this happens, the system may or may not completely lock up, and lines like these will appear in your syslog:

Aug 1 22:58:16 dreadnought kernel: CIFS VFS: No response to cmd 46 mid 25661 
Aug 1 22:58:16 dreadnought kernel: CIFS VFS: Send error in read = -11
Aug 1 22:58:16 dreadnought kernel: CIFS VFS: No response for cmd 50 mid 25664
Aug 1 22:58:16 dreadnought kernel: CIFS VFS: No response to cmd 47 mid 25663
Aug 1 22:58:20 dreadnought kernel: CIFS VFS: No response to cmd 47 mid 25662
Aug 1 22:58:20 dreadnought kernel: CIFS VFS: Write2 ret -11, written = 0
Aug 1 22:58:20 dreadnought kernel: CIFS VFS: Send error in read = -9
Aug 1 22:58:20 dreadnought kernel: CIFS VFS: Write2 ret -9, written = 0
Aug 1 22:59:52 dreadnought kernel: CIFS VFS: No writable handles for inode
Aug 1 23:00:29 dreadnought kernel: CIFS VFS: close with pending writes
Aug 1 23:00:36 dreadnought kernel: CIFS VFS: close with pending writes
Aug 1 23:01:16 dreadnought kernel: CIFS VFS: No writable handles for inode
Aug 1 23:02:28 dreadnought kernel: CIFS VFS: close with pending writes
Aug 1 23:03:23 dreadnought kernel: CIFS VFS: close with pending writes
Aug 1 23:03:44 dreadnought kernel: CIFS VFS: close with pending writes
Aug 1 23:05:35 dreadnought kernel: CIFS VFS: No writable handles for inode
Aug 1 23:06:18 dreadnought kernel: CIFS VFS: close with pending writes
Aug 1 23:06:35 dreadnought kernel: CIFS VFS: close with pending writes
Aug 1 23:06:46 dreadnought kernel: CIFS VFS: No writable handles for inode
Aug 1 23:06:46 dreadnought kernel: CIFS VFS: close with pending writes
Aug 1 23:07:02 dreadnought kernel: CIFS VFS: close with pending writes
Aug 1 23:07:46 dreadnought kernel: CIFS VFS: server not responding

I found a workaround which involves disabling Opportunistic Locking. This will have a small performance impact, but at least the thing wont hang up and use 100% CPU for no goddamn reason. Yes, this article has an angry tone, because this issue has eaten way too much of my time today.

To disable Opportunistic Locking, you have to set the contents of the file "/proc/fs/cifs/OplockEnabled" to 0. This file only exists after the cifs module has been loaded, and will be replaced if the module is unloaded and reloaded. Thus, I configured my /etc/rc.local script (use whatever equivalent your distro has to a startup script) to both load the cifs module and set the contents of said file:

# cifs client workaround
modprobe cifs
echo 0 > /proc/fs/cifs/OplockEnabled

There you go.

Filed under: Linux, Solved Problems Leave a comment
Comments (20) Trackbacks (0)
  1. ronald zijlstra
    September 27th, 2008 - 19:20

    thanx a lot. this has been a nasty bug for me for a long time!!! should be default setting i guess….

    ( REPLY )
  2. Straun
    October 15th, 2008 - 16:12

    A good solution, all I need now is a faster network drive.

    ( REPLY )
  3. Mr. Blue
    November 16th, 2008 - 03:27

    Great idea…worked wonders, sometimes I forget about /proc

    ( REPLY )
  4. mav
    April 28th, 2009 - 21:34

    You know I have been looking for this for a long time, it took me forever to figure out what was going on. I have been running back up’s over the network every night, and about every other night the server would completely lock up, especially on large back ups. This sounds like it will fix the problem, I have made the change and we will see if this fixes the issue. It looks like this would be a more common problem, and have more visibility, but it has taken me a while to come across this site.

    Thanks for the fix, and I understand your frustration, while my server is only for development purposes, it was still a pain to come in to work in the morning to a server not responding to anything and having to reboot.

    ( REPLY )
  5. Monkey Prince
    June 22nd, 2009 - 23:45

    I follow your instructions, but still have these errors
    CIFS VFS: No response for cmd 50 mid 61463
    CIFS VFS: No response for cmd 50 mid 61462
    CIFS VFS: No response for cmd 50 mid 61461
    CIFS VFS: No response to cmd 46 mid 61460
    CIFS VFS: Send error in read = -11
    CIFS VFS: No response for cmd 50 mid 61465
    CIFS VFS: No response for cmd 50 mid 61466
    CIFS VFS: No response for cmd 50 mid 61464
    If I cat /proc/fs/cifs/OplockEnabled, it returns 0. Please help me

    ( REPLY )
  6. dexter
    August 28th, 2009 - 16:59

    The same thing like Monkey Prince.
    Help.

    ( REPLY )
  7. bolt
    August 29th, 2009 - 09:06

    I’m very sorry, but the above solution worked for me. If you’ve made sure the file exists in the first place and hasn’t been renamed or anything in your distro, then I’m afraid I can’t help you much more, as the above solution works for me.

    ( REPLY )
  8. Maurice
    September 15th, 2009 - 01:14

    Touchdown! I use a Nagios server to check the “goodness” of a number of files on Windows servers. Since moving from hard metal to a virtual server the network has been occasionally locking up within minutes of a reboot. I was blaming ntp until I googled the CIFS error message.

    I can now get my life back!

    ( REPLY )
  9. Marco Verleun
    January 7th, 2010 - 00:11

    It is possible to change this setting automatically when loading the CIFS module. modprobe has a nice feature to execute a script when loading a module. Check man 5 modprobe.conf

    Add a line to /etc/modprobe.conf or a file in /etc/modprobe.d containing the following line, this is one single line:
    install cifs /sbin/modprobe –ignore-install cifs; echo 0 > /proc/fs/cifs/OplockEnabled ;

    ( REPLY )
  10. Bill Rebey
    April 30th, 2010 - 20:52

    Thanks a million for posting this! This has been a real nightmare for me ever since upgrading to 9.10 from 9.04 (the problem didn’t seem to exist in 9.04).

    ( REPLY )
  11. korman
    July 2nd, 2010 - 16:02

    We just put into production a RHEL5 Linux / Oracle server which uses the CIFS client to injest a number of files located on a Win2k8 Server. At 8PM I found similar errors and SAR showed %system cpu at to 99% and staying there until the server had was rebooted 2 hours later. I am guessing this post might have saved my ass..

    ( REPLY )
  12. korman
    July 2nd, 2010 - 16:03

    I should add it was “Cold booted”

    ( REPLY )
  13. cnafa
    July 19th, 2010 - 11:31

    My function use cifs protocol to transfer file and it usually occur “CIFS VFS: Send error in read = -11
    ” error.
    I searched the cifs reference and get a ways to resovle it.

    http://pserver.samba.org/samba/ftp/cifs-cvs/linux-cifs-client-guide.pdf

    I just lower the block of data what i want to write

    ( REPLY )
  14. cnafa
    July 19th, 2010 - 11:36

    My function use cifs protocol to transfer file and it usually occur “CIFS VFS: Send error in read = -11
    ” error.
    I searched the cifs reference and get a ways to resovle it.

    http://pserver.samba.org/samba/ftp/cifs-cvs/linux-cifs-client-guide.pdf

    ”
    Performance Considerations
    Server speed, server disk speed and network speed can constrain the overall performance of a cifs
    mount, but in some cases, client client configuration settings can be changed to increase performance:
    1) size of file write (wsize). The Linux CIFS client usually sends 56K writes (14 pages) and is
    limited to 56K maximum unless mounted forcedirectio.
    2) size of file read (rsize). The Linux CIFS client usually sends 16K reads (4 pages). Since CIFS
    large network buffers are about 16K in size by default, increasing the rsize would have little
    effect unless the setting of module load parameter CIFSMaxBufSize (via insmod) also is
    increased.
    ”

    I just lower the block of data what i want to write.

    you can use command “modinfo cifs” to get the cifs read or write buffer size and buffersize scope.

    ( REPLY )
  15. Tiago Neiva
    August 15th, 2010 - 18:49

    For me it’s still not perfect I still get
    “CIFS VFS: ignoring corrupt resume name”
    messages, but the others went away and performance went up!

    I’m using ubuntu 10.04 and created this file:
    /etc/modprobe.d/cifs.conf

    with this contents:

    # Created to fix cifs, thanks to the ubuntu forums
    install cifs /sbin/modprobe –ignore-install cifs && echo 0 > /proc/fs/cifs/LinuxExtensionsEnabled && echo 0 > /proc/fs/cifs/OplockEnabled
    remove cifs /sbin/modprobe -r cifs

    ( REPLY )
  16. Chetan
    September 22nd, 2010 - 12:30

    Hi Tiago,

    Does the modeprobe comment applies to Redhat 5.5 also. We are facing the same issue(CIFS VFS: No response for cmd 50 mid 10033). And let us know if the above solution which you have provided to create a cifs.conf file will resolve the issue.

    ( REPLY )
  17. Blodge
    October 20th, 2010 - 21:22

    Same as Chetan on RHEL5.5_x64.

    CIFS VFS: No response for cmd 114 mid 52651
    CIFS VFS: Unexpected lookup error -112
    CIFS VFS: No response for cmd 114 mid 52659

    Repeat..CPU goes to 100% eventually and we kernel panic. Next time it works just fine.

    ( REPLY )
  19. Hav
    July 17th, 2012 - 16:28

    Thank you, this issue has been plaguing the company I work for for a few days. Hopefully this will fix! Thanks a lot

    ( REPLY )

Leave a comment

No trackbacks yet.

» «

Categories

Way back in…

Cows eat grass.
Zombies eat brains.