Locating Files in the Snapshots

NOTE: We are in the process of migrating data to the Isilon. Currently /home and /admin are hosted on the Isilon, while /course and /research are still on GPFS. The name of the snapshot directory is different on the two file servers: .snapshot on the Isilon, .snapshots on GPFS.

Each directory in the main filesystem contains an invisible directory called .snapshot[s], which usually does not appear in directory listings. You must explicitly refer to .snapshot[s] in order to use it. Beneath the .snapshot[s] directory are some more directories, each of which corresponds to a snapshot. The names of these directories look strange, but it all has meaning (date and time...). The most recent might look like this: @GMT-2010.12.02-13.00.02. From there they go back in time to include ones named with -daily and -weekly.

Each of these .snapshot[s]/* directories is a complete copy of the live directory, frozen at the moment the snapshot was created. Suppose you inadvertently delete a file called foo in your home directory. You might find snapshotted versions of it named .snapshot/CS_Hourly_2019-10-20-_13-00/foo or .snapshots/@GMT-2019.10.02-05.00.01-daily/foo, etc.

Note that snapshots are not taken continuously. They are discrete events that happen on a regular schedule. If you create a new file and delete it before any snapshot is taken, it is irretrievably lost.

Also note that snapshots are only taken of the main filesystem. This includes /admin, /course, /home, and /research (which redirect to /gpfs/main/admin, /gpfs/main/course, /gpfs/main/home, and /gpfs/main/research respectively). This does not include /data or /nbu, which are separate filesystems.

Since every directory in the live filesystem has an invisible .snapshots directory, you can access a file in a snapshot with different paths. Suppose you lost the file foo/bar and that the backup copy that you want is in snapshot @GMT-2010.12.02-13.00.02. You can retrieve the file a couple of different ways. You can get it from .snapshots/@GMT-2010.12.02-13.00.02/foo/bar or you can get it from foo/.snapshots/@GMT-2010.12.02-13.00.02/bar. This is a very handy feature. Suppose you delete the entire foo directory and later discover that you need foo/bar back. You would be unable to use foo/.snapshots because foo no longer exists in the live filesystem. Fortunately, foo still exists in the snapshot as .snapshots/@GMT-2010.12.02-13.00.02/foo so you can retrieve file bar from there.

Each .snapshot[s] directory and everything beneath it is strictly read-only. You cannot create, remove, or modify anything.

Cookbook Example

If you can use the Unix commands ls and cp, you can restore your own files from snapshots. Assume you have lost the file foo in the current directory. Figure out which snapshotted version of foo that you want back with this command.

ls -lt .snapshots/*/foo

This lists information about the snapshotted copies of foo, sorted by modification time. The output will look something like this: <smaller></smaller>

-rw-r--r-- 1 djm root 1154 Nov 16 11:31 @GMT-2010.11.22-05.00.02-weekly/foo
-rw-r--r-- 1 djm root 1154 Nov 16 11:31 @GMT-2010.11.29-05.00.01-weekly/foo
-rw-r--r-- 1 djm root 1154 Nov 16 11:31 @GMT-2010.12.01-05.00.01-daily/foo
-rw-r--r-- 1 djm root 1154 Nov 16 11:31 @GMT-2010.12.02-05.00.01-daily/foo
-rw-r--r-- 1 djm root 1154 Nov 16 11:31 @GMT-2010.12.04-05.00.01-daily/foo
-rw-r--r-- 1 djm root 1154 Nov 16 11:31 @GMT-2010.12.05-05.00.02-daily/foo
-rw-r--r-- 1 djm root 1154 Nov 16 11:31 @GMT-2010.12.06-05.00.01-daily/foo
-rw-r--r-- 1 djm root 1154 Nov 16 11:31 @GMT-2010.12.06-05.00.01-weekly/foo
-rw-r--r-- 1 djm root 1154 Nov 16 11:31 @GMT-2010.12.07-05.00.01-daily/foo
-rw-r--r-- 1 djm root 1154 Nov 16 11:31 @GMT-2010.12.07-21.00.02/foo
-rw-r--r-- 1 djm root 1154 Nov 16 11:31 @GMT-2010.12.08-01.00.01/foo
-rw-r--r-- 1 djm root 1154 Nov 16 11:31 @GMT-2010.12.08-05.00.01-daily/foo
-rw-r--r-- 1 djm root 1154 Nov 16 11:31 @GMT-2010.12.08-09.00.01/foo
-rw-r--r-- 1 djm root 1154 Nov 16 11:31 @GMT-2010.12.08-13.00.01/foo
-rw-r--r-- 1 djm root 1154 Nov 16 11:31 @GMT-2010.12.08-17.00.01/foo

Pay particular attention to the dates, which is when these files were last modified. In the above example, .snapshots/@GMT-2010.12.08-17.00.01/foo is the most recent copy of the file.

 

The number to the left of the date is the size of the file in bytes. This might also help you decide which version of the file you want back. And of course, you can always look at these files with pg, more, cat or your favorite editor. Suppose you have decided to restore .snapshots/@GMT-2010.12.08-17.00.01/foo. Simply copy it with cp

cp .snapshots/@GMT-2010.12.08-17.00.01/foo foo

Note that if file foo exists and you are copying over it, the cp command may fail with the error

cp: .snapshots/@GMT-2010.12.08-17.00.01/foo and foo are identical
(not copied)

The two files are not identical, but cp has been fooled by the snapshot. You should remove foo with rm foo or rename it with mv foo foo.bad and then the cp command will work.

If the command ls -lt .snapshots/*/foo fails with an error message such as .snapshots/*/foo: No such file or directory, then there is no file named foo (from this directory) in any snapshot. Perhaps you had the file in a different directory. Repeat the above procedure from the correct directory. Perhaps you lost the file before the filer made any snapshot of it, in which case you are out of luck. Or perhaps you lost the file long enough ago that all the snapshots that contained it have been deleted. You will have to run "restore_request".