Why is du showing normal disk usage after deleting files, but df still shows the disk as full?

Recently, I encountered a situation where my server's disk got filled up due to a huge access.log file. After deleting the file, I expected the disk space to free up, but surprisingly, the df command still showed 100% disk usage. This left me puzzled, wondering what was causing this discrepancy between du and df.

After some investigation, I found the reason for this issue. It turned out that even though I had deleted the access.log file, the Nginx process was still holding on to it. As a result, the file was not completely released, and the disk space was still occupied.

Here's how I identified and resolved the problem:

1) To investigate the issue, I used the lsof command to list all open files in the /data directory and checked if any of them were marked as "deleted." The command I used was:

lsof /data | grep deleted

This showed me the following result:

nginx     13878   root   76w   REG 252,16 408367907009  8389222 /data/nginx/log/access.log (deleted)
nginx     13878   root   94w   REG 252,16  16210064602  8391728 /data/nginx/log/wx_access.log (deleted)

The (deleted) tag at the end of each line indicated that the files were marked as deleted but still being held open by the Nginx process.

2) To resolve the issue, I performed the following steps:

  • Instead of directly deleting the access.log file, I used the command > access.log to truncate and clear its content. This way, Nginx would stop writing to the file.
  • After truncating the access.log file, I needed to release the file's locks held by Nginx. Restarting Nginx would typically do the job, but in this case, since the disk was almost full, the service nginx restart command might fail due to lack of available disk space. So, I manually killed the Nginx process using the appropriate command, which allowed the file to be fully released.

3) Once Nginx was restarted (or manually killed), the access.log file was no longer held open, and the disk space was properly freed up. Now, when I ran the df command, it correctly showed the actual available disk space.

In conclusion, the reason du and df showed different disk usage was that the access.log file was still being held open by the Nginx process even after I deleted it. By truncating the file's content and restarting Nginx, I was able to resolve the issue and free up the occupied disk space. Remember to be cautious when dealing with a full disk situation, as it may affect the proper functioning of system services. Always ensure you have enough free space to perform necessary operations.

2023-07-14 22:45:52 | NOTE | 0 Comments
 

 

Leave A Comment

RELATED STORIES


MOST RECENTLY