Recovering Deleted Files using lsof for Active Process File Descriptors
Accidentally deleting a critical log, database file, or configuration can be catastrophic. However, on Linux, a file is only truly removed from the disk when its link count reaches zero and no processes have the file open. If a process (like Nginx, MySQL, or a custom script) still has the file open, the data still exists in the /proc filesystem. By using lsof (List Open Files), we can identify the file descriptor and recover the content before the process terminates.
1. The Logic of Linux File Deletion
In the Linux filesystem, a file's name is just a link to an inode. When you rm a file, you remove the link. If a process is still reading or writing to that file, the kernel keeps the inode active. The data is available via the process's file descriptor in the /proc directory until the process is stopped or restarted.
2. Identifying the Missing File with lsof
To recover a file, you must first find the Process ID (PID) and the File Descriptor (FD). Use lsof and filter for the deleted file name. Linux will typically mark these with the suffix (deleted).
# Search for open files that have been deleted
lsof | grep "/var/log/nginx/access.log"
Interpreting the Output:
- COMMAND: The name of the process holding the file (e.g.,
nginx). - PID: The unique Process ID (e.g.,
1234). - FD: The file descriptor number followed by a letter (e.g.,
4w). The number4is what we need.
3. Restoring the Data from /proc
Every process has a subdirectory in /proc named after its PID. Inside, the fd folder contains symbolic links to every file the process has open. Even if the file was deleted from the main filesystem, you can copy the data back out of this virtual link.
# Syntax: cp /proc/[PID]/fd/[FD] /path/to/restored_file
# Example recovery:
sudo cp /proc/1234/fd/4 /var/log/nginx/access.log
4. Recovery Success Matrix
| Scenario | Recoverable? | Technical Reason |
|---|---|---|
| Process is still running | Yes | Kernel keeps inode data active. |
| Process was restarted | No | The last file descriptor was closed; blocks are marked free. |
| Server was rebooted | No | Volatile /proc entries are cleared; inode is lost. |
| File was never open | No | No file descriptor exists; requires forensic tools (e.g., extundelete). |
5. Technical Implementation Details
- FD Suffixes: In
lsof,rmeans read-only,wmeans write, andumeans read and write. This doesn't affect recovery, but it identifies how the process is interacting with the file. - Immediate Action: If you realize a file was deleted, do not stop the service. Stopping Nginx or MySQL to "fix" the issue will close the file descriptor and make the data unrecoverable via this method.
- Permissions: When you copy the file back from
/proc, the new file will be owned by the user who executed thecpcommand. Usechownto restore the original ownership (e.g.,www-data).
Sysadmin Advice: This method is a lifesaver for recovering large logs that were accidentally deleted while the server was still writing to them. If the file was not open by any process, you must immediately unmount the partition and use forensic tools to prevent the Hovixa NVMe controller from overwriting those blocks with new data.
