Bring back deleted files with lsof

There you are, happily playing around with an audio file you've spent all afternoon tweaking, and you're thinking, "Wow, doesn't it sound great? Lemme just move it over here." At that point your subconscious chimes in, "Um, you meant mv, not rm, right?" Oops. I feel your pain -- this happens to everyone. But there's a straightforward method to recover your lost file, and since it works on every standard Linux system, everyone ought to know how to do it.

Briefly, a file as it appears somewhere on a Linux filesystem is actually just a link to an inode, which contains all of the file's properties, such as permissions and ownership, as well as the addresses of the data blocks where the file's content is stored on disk. When you rm a file, you're removing the link that points to its inode, but not the inode itself; other processes (such as your audio player) might still have it open. It's only after they're through and all links are removed that an inode and the data blocks it pointed to are made available for writing.

This delay is your key to a quick and happy recovery: if a process still has the file open, the data's there somewhere, even though according to the directory listing the file already appears to be gone.

This is where the Linux process pseudo-filesystem, the /proc directory, comes into play. Every process on the system has a directory here with its name on it, inside of which lies many things -- including an fd ("file descriptor") subdirectory containing links to all files that the process has open. Even if a file has been removed from the filesystem, a copy of the data will be right here:

/proc/process id/fd/file descriptor

To know where to go, you need to get the id of the process that has the file open, and the file descriptor. These you get with lsof, whose name means "list open files." (It actually does a whole lot more than this and is so useful that almost every system has it installed. If yours isn't one of them, you can grab the latest version straight from its author.)

Once you get that information from lsof, you can just copy the data out of /proc and call it a day.

This whole thing is best demonstrated with a live example. First, create a text file that you can delete and then bring back:

$ man lsof | col -b > myfile

Then have a look at the contents of the file that you just created:

$ less myfile

You should see a plaintext version of lsof's huge man page looking out at you, courtesy of less.

Now press Ctrl-Z to suspend less. Back at a shell prompt make sure your file is still there:

$ ls -l myfile
-rw-r--r--  1 jimbo jimbo 114383 Oct 31 16:14 myfile
$ stat myfile
 File: `myfile'
 Size: 114383          Blocks: 232        IO Block: 4096   regular file
Device: 341h/833d       Inode: 1276722     Links: 1
Access: (0644/-rw-r--r--)  Uid: ( 1010/    jimbo)   Gid: ( 1010/    jimbo)
Access: 2006-10-31 16:15:08.423715488 -0400
Modify: 2006-10-31 16:14:52.684417746 -0400
Change: 2006-10-31 16:14:52.684417746 -0400

Yup, it's there all right. OK, go ahead and oops it:

$ rm myfile
$ ls -l myfile
ls: myfile: No such file or directory
$ stat myfile
stat: cannot stat `myfile': No such file or directory
$

It's gone.

At this point, you must not allow the process still using the file to exit, because once that happens, the file will really be gone and your troubles will intensify. Your background less process in this walkthrough isn't going anywhere (unless you kill the process or exit the shell), but if this were a video or sound file that you were playing, the first thing to do at the point where you realize you deleted the file would be to immediately pause the application playback, or otherwise freeze the process, so that it doesn't eventually stop playing the file and exit.

Now to bring the file back. First see what lsof has to say about it:

$ lsof | grep myfile
less      4158    jimbo    4r      REG       3,65   114383   1276722 /home/jimbo/myfile (deleted)

The first column gives you the name of the command associated with the process, the second column is the process id, and the number in the fourth column is the file descriptor (the "r" means that it's a regular file). Now you know that process 4158 still has the file open, and you know the file descriptor, 4. That's everything you have to know to copy it out of /proc.

You might think that using the -a flag with cp is the right thing to do here, since you're restoring the file -- but it's actually important that you don't do that. Otherwise, instead of copying the literal data contained in the file, you'll be copying a now-broken symbolic link to the file as it once was listed in its original directory:

$ ls -l /proc/4158/fd/4
lr-x------  1 jimbo jimbo 64 Oct 31 16:18 /proc/4158/fd/4 -> /home/jimbo/myfile (deleted)
$ cp -a /proc/4158/fd/4 myfile.wrong
$ ls -l myfile.wrong
lrwxr-xr-x  1 jimbo jimbo 24 Oct 31 16:22 myfile.wrong -> /home/jimbo/myfile (deleted)
$ file myfile.wrong
myfile.wrong: broken symbolic link to `/home/jimbo/myfile (deleted)'
$ file /proc/4158/fd/4
/proc/4158/fd/4: broken symbolic link to `/home/jimbo/myfile (deleted)'

So instead of all that, just a plain old cp will do the trick:

$ cp /proc/4158/fd/4 myfile.saved

And finally, verify that you've done good:

$ ls -l myfile.saved
-rw-r--r--  1 jimbo jimbo 114383 Oct 31 16:25 myfile.saved
$ man lsof | col -b > myfile.new
$ cmp myfile.saved myfile.new

No complaints from cmp -- your restoration is the real deal.

Incidentally, there are a lot of useful things you can do with lsof in addition to rescuing lost files.