Mass Storage – Things to Avoid

There are things to avoid in order to make good use of the Mass Storage system.

1. Do not store or archive sensitive data to the Mass Storage system.

2. Do not use Mass Storage to backup your desktop hard disk.

Mass storage is not intended to be used as a backup location for local disk drives, operating systems, or software.  In general, files that change often or directories with more than a thousand files will cause performance problems and consume tape resources. The  PC backup software provided by UNC might be an alternative solution to copying your PC files to mass storage.

3. Do not create more than 1000 files in a single directory. Mass Storage scans its directory space several times every hour. The time to scan a directory is exponentially linked to the number of files and directories within it. Since scanning is single threaded, one large directory can slow down the entire system. Similarly, creating one large output file is preferable to several smaller ones.

 

4. Do store a tar or zip archive in Mass Storage space, but please create the archive outside of Mass Storage.  It is very advantageous to our administering of Mass Storage for people to put tar or zip archives in Mass Storage instead of directories with many files.

5. Do not run the tar command on directories or files in Mass Storage.  Again, create the tar archive outside Mass Storage in your home directory or scratch space.

6. Do not compress files which already exist in Mass Storage space.  If a file is not in Mass Storage, you can compress it first (although this is not necessary) outside of Mass Storage space and then move the compressed file into Mass Storage.  Note that you gain nothing by putting compressed files in Mass Storage space, because all files in Mass Storage are compressed when written to tape. 

7. Do not modify frequently any file for consecutive days. Mass Storage only copies to tape files which have not been modified for a few hours.

8. As a corollary to the last item, do not put in Mass Storage files that will be frequently modified.

9. Do not write directly into Mass Storage space, such as your ms directories.  For example from Pine (or any IMAP client).  Instead, export any files from Pine to your home space and then perform a mv to your ms directories.

10. Do not execute long-running programs when your current working directory is a Mass Storage directory.

11. Do not execute a program if the executable file is in Mass Storage space.

12. Do not execute long-running programs that open files in Mass Storage space.

13. Do not use Mass Storage as a scratch space. If you are going to create a number of files which you do not want to keep permanently, use scratch space /netscr.  Some programs, such as Gaussian, create huge temporary files which do not always get deleted. Such files are a large waste of tape resources.

14. Do not write to Mass Storage directly when creating a dataset in SAS. Instead, write your dataset to /netscr or /lustre, then copy it to mass storage once you have finished modifying it.

15. It if helpful not to copy many symbolic links and empty files (i.e. LSF *.err files) into mass storage as these do not get archived to tape.

Additional help

More on Mass Storage

Research Computing home page