How To Delete Duplicate Files with fdupes in One Command

Managing storage space can often be a daunting task, especially when dealing with duplicate files. Thankfully, there are tools available to help streamline this process. One such tool is fdupes, a command-line utility for identifying and deleting duplicate files in Unix and Unix-like systems. In this article, we will guide you on how to delete duplicate files with fdupes in one command.

Introduction to fdupes

fdupes is a powerful tool that uses algorithms to identify duplicate files within given directories. It compares files based on their size and MD5 checksums, ensuring accurate results. You can install fdupes on most Linux distributions using the package manager. For example, on Ubuntu, you can install it using the following command:

sudo apt-get install fdupes

Using fdupes to Delete Duplicate Files

The basic usage of fdupes is quite straightforward. You simply pass the directory you want to scan as an argument:

fdupes /path/to/directory

This will list all duplicate files in the specified directory. However, we want to delete the duplicates. For this, we can use the -d and -N options:

fdupes -dN /path/to/directory

Here’s what each option does:

• -d: This option tells fdupes to delete duplicates. When combined with the -N option, it will preserve the first file and delete all other duplicates.
• -N: This option runs fdupes silently without prompting for confirmation.

Handling Filenames with Spaces and Special Characters

If your filenames contain spaces or special characters, you can use the xargs command to handle these correctly:

fdupes -f /path/to/directory | xargs -rd '\n' -- rm

This command pipes the output of fdupes to xargs, which then passes each duplicate file as an argument to rm, the command for deleting files. The -rd '\n' options ensure that filenames with spaces or special characters are handled correctly.

Moving Duplicates to a Separate Folder

If you prefer to move the duplicates to a separate folder instead of deleting them, you can create a script to accomplish this. Here’s an example script:

#!/bin/bash

# Save default separator definitions
oIFS=$IFS
# define new line as a separator, filenames can have spaces
IFS=$'\n';

# For each file (f) listed as duplicated by fdupes, recursively
for f in $(fdupes -r -f /path/to/directory)
do
 # Log the files I'm moving
 echo "Moving $f to folder Duplicates" >> ~/log.txt
 # Move the duplicated file, keeping the original in the original folder
 mv "$f" Duplicates/
done

# restore default separator definitions
IFS=$oIFS

This script uses a for loop to iterate over each duplicate file found by fdupes and moves them to a folder named “Duplicates”. You can modify the script to specify a different folder name if desired.

Conclusion

fdupes is a powerful tool that can help you manage your storage space by identifying and deleting duplicate files. With the right options, you can even delete duplicates in one command, saving you valuable time and effort. As always, remember to back up your files before performing any bulk deletion operations to avoid accidental data loss.

For more detailed usage information and options, you can refer to the fdupes manpage.