Storage Services at CHPC
CHPC currently offers four different types of storage: home directories, group space, scratch file systems and a new archive storage system. All storage types except for the archive storage system are accessible from all CHPC resources. Data on the new archive storage space must be moved to one of the other spaces in order to be accessible. Home directories and group spaces can also be mounted on local desktops. See the Data Transfer Services page for information on mounting CHPC file systems on local machines along with details on moving data to and from CHPC file systems.
In addition we have limited tape back up systems for both home directories and group spaces.
Note that the information below is specific for the general environment. In the protected environment (PE) all four types of storage exist. However, the nature of the storage, the pricing and policies do vary in the PE. See the Protected Environment page for more details.
***Remember that you should always have an additional copy or possibly multiple copies, on independent storage systems, for any crucial/critical data. While storage systems built with data resiliency mechanisms (such as RAID and erasure coding mentioned in the offerings listed below or other similar technologies) allow for multiple component failures, they do not offer any protection for large scale hardware failures, software failures leading to corruption, or for accidental deletion or overwriting of data. Please take the necessary steps to protect your data to the level you deem necessary.***
CHPC provides several options for home directory file systems.
General HPC home directories
The General HPC home directory storage system is the default home directory file system
that is available to groups free of charge. If your group has not purchased storage
or you do not fit into one of the other categories listed below, then this is the
space where your CHPC home directory will be provisioned. This file system has a 50
GB per user quota which is enforced. CHPC can provide temporary increases to this
space. To view the current home directory usage and quota status, run command
THIS SPACE IS NOT BACKED UP. It is assumed important data will be copied to a departmental file server or another location.
Owner home directories - New solution as of August 2017
CHPC currently allows CHPC PIs with sponsored research projects to buy-in to storage at a price determined based on cost recovery. The current limit for this space is 1TB/group, and all members of the research group will have home directories on this space. The hardware for the current home directory solution was purchased in the summer of 2017. Initially, it was sold at a price of $1250/TB for the 5 year warranty period of the hardware. The current price has been prorated to $250/TB for the remaining warranty lifetime of about one year.
This solution is based on an offering from Dell known as their Compellent solution. In this solution there are two RAID disk based copies, one of which is the primary storage with which users will normally interact. The second copy is used as a fail over, and is effectively a replicated copy of the primary side. In case of hardware issue with the primary copy, the fail over will become the active working copy until repairs can be made. For increased performance, there are solid state drives that will be used in an transparent manner as a first tier for I/O to this space in front of the larger capacity traditional spinning drives. Note that these features, including the fail over copy, will be present for all home directories, including the default 50GB ones provided to users whose groups do not purchase the larger home directory space.
Note that having a redundant copy is not a backup solution as any changes in the primary side will be synced to the secondary side, for example deleting or overwriting a file. THIS SPACE IS BACKED UP. The price of this solution also includes a back up, with nightly incremental and weekly full back ups and a two week retention window.
We will continue to prorate the cost of this storage based on the remaining warranty time. When it is time to refresh the hardware, CHPC will contact all groups who have purchased space about the new pricing policy. Please contact us by emailing email@example.com and request to meet with us to discuss your needs and timing.
Group Level Storage File Systems
CHPC currently allows CHPC PIs with sponsored research projects to buy-in to file storage at a price determined based on cost recovery. A more detailed description of this storage offering is available. The current pricing is $150/TB for the lifetime of the hardware which is purchased with a 5 year warranty. CHPC purchases the hardware for this storage in bulk and then sells it to individual groups in TB quantities, so depending on the amount of group storage space you are interested in purchasing, CHPC may have the storage to meet your needs on hand. Please contact us by emailing firstname.lastname@example.org and request to meet with us to discuss your needs and timing. BY DEFAULT THIS SPACE IS NOT BACKED UP, HOWEVER CHPC PROVIDES A BACK UP OPTION.
NOTE -- March 2019: We are no longer offering backup of NEW group spaces to tape. We will continue to provide backups of group spaces for which groups who have already purchased tapes. Details of the new options for backup of group spaces are given in CHPC's Spring 2019 Newsletter as well as in the Backup section below.
Update -- August 2021: The Summer 2021 quarter (July 1 - September 30) will be the last quarterly archive to tape of any CHPC general environment group space that is still being backed up to tape. All owners of these spaces have been notified. After that time, until XX, we will only maintain the tape infrastructure needed to do restores from the existing tapes.
New archive backups of group level storage will be to the Archive Storage discussed below. CHPC will perform the backups on a quarterly basis provided the group purchase enough space on pando to allow for two copies of the data. Contact us at email@example.com to set up any group space backup. CHPC also provides information on a number of user driven alternative to this service; see the section on User Driven Backup Options below.
Scratch File Systems
There are various scratch file systems which are available on the HPC clusters. THE SCRATCH FILE SYSTEMS ARE NOT BACKED UP. This space is provided for users to store intermediate files required during the duration of a job on one of the HPC clusters. On these scratch file system, files that have not been accessed for 60 days are automatically scrubbed. There is no charge for this service.
The current scratch file systems are:
- /scratch/general/lustre - a 700TB lustre parallel file system accessible from all CHPC resources
- /scratch/kingspeak/serial - a 175 TB NFS system accessible from all CHPC resources
- /scratch/general/nfs1 - a 595 TB NFS system accessible from all CHPC resources
Linux defines temporary file system at
/var/tmp where temporary user and system files are stored. CHPC cluster nodes set up temporary
file systems as a RAM disk with limited capacity. All interactive and compute nodes
have also a spinning disk local storage at
/scratch/local. If a user program is known to need temporary storage, it is advantageous to set
environment variable TMPDIR which defines the location of the temporary storage and point it to
/scratch/local. Local disk drives range from 40 to 500 GB depending on the node, which is much more
than the default
/scratch/local can also be used for storing intermediate files during calculation, however be aware
that getting to these files after the job finishes will be difficult since they are
local to the (compute) node and not directly accessible from cluster interactive nodes.
**July 2020 Update -- changes to the use /scratch/local on cluster compute nodes** : Users can still make use of
/scratch/local on the compute nodes of the cluster for their batch jobs. However access permissions
will be set such that users will no longer be able to create directories in the top
/scratch/local directory. Instead, as part of the slurm job prolog (before the job is started),
a job level directory,
/scratch/local/$USER/$SLURM_JOB_ID , will be created. Only the job owner will have access to this directory. At the
end of the job, in the slurm job epilog, this job level directory will be removed.
These changes are being made to solve two main issues. First, this process will achieve
isolation among jobs, thereby allowing for cleanup of the
/scratch/local file system, and eliminating cases where a new job starts on a cluster with a full
or nearly full
/scratch/local . In addition, this will also eliminate the situation where a job is started only
to fail when it cannot write to
/scratch/local due to hardware issues. By moving the top level directory creation to the job prolog,
if there are any issues creating this job level directory, the node will be off-lined,
and the job prolog will find a new node for the job.
All slurm scripts that make use of
/scratch/local must be adapted to accommodate this change. Additional updated information is provided
on the CHPC slurm page.
One additional part of the change to /scratch/local is that space is now being sofware encrypted. Each time a node is rebooted this software encryption is re-setup from scratch which in effect purges anything the content of this space. There is also a cron job in place to scrub /scratch/local of content that has not been accessed for over 2 weeks. This scrub policy can be adjusted on a per host basis. A node owned by a group can opt to have us disable this (declare it as an exception for that scrub script) and it will not run on that host.
CHPC now has a new archive storage solution based around object storage, specifically ceph, a distributed object store suite developed at UC Santa Cruz. We are offering an 6+3 erasure coding configuration which results in a price of $150/TB of usable capacity for the 5 year lifetime of the hardware. As we currently do with our group space, we will operate this space in a condominium model by reselling this space in TB chunks.
This space is a stand alone entity, and will not be mounted on other CHPC resources.
One of the key features of the archive system is that users manage the archive directly,
unlike the tape archive option. Users can move data in and out of the archive storage as needed -- they can archive
milestone moments in their research, store an additional copy of crucial instrument
data, and retrieve data as needed. This archive storage solution will be accessible
via applications that use Amazon’s S3 API. GUI tools such as transmit (for Mac) as well as command-line tools such as
s3cmd and rclone can be used to move the data.
It should also be noted that this archive storage space is for use in the general environment, and is not for use with regulated data; there is a separate archive space in the protected environment.
The backup policy of the individual file systems is mentioned above.
Note -- March 2019: CHPC is migrating the backup of group home directory from tape to the disk based archive storage mentioned above. For CHPC managed backups of any newly purchased spaces groups will need to purchase space, typically twice the capacity of the group space, on the archive storage solution mentioned above. Alternate user driven backup solutions are mentioned in the next section.
Update -- August 2021: The Summer 2021 quarter (July 1 - September 30) will be the last quarterly archive to tape of any CHPC general environment group space.
User Driven Backup Options
There is a University of Utah central IT knowledge base article detailing university level storage options such as GSuite for Education, Microsoft O365, and Box. This article includes information on the suitabliity of the different storage options for different types of data. Please follow these university guidelines to determine a suitable location for your data.
Owner back up to Google Drive: There is a University agreement with Google that provides for unlimited storage* and this is an option that a number of CHPC users already use for backup, using the rclone tool. While the University's google storage option via the GSuite for Education was unsuitable for sensitive or restricted data in the past, recently this has been changed to allow sensitive and restricted data to be stored. Additional details on backing up your data to this location can be found on the University’s Google Drive page and CHPC's rclone page. One consideration is that the google drive storage is owned by an individual, not by a group.
*- Note that while the current GSuite for Education offering from Google is for unlimited storage, Google has recently announced that their model for storage will be changing effective July 2022. Under the new Google Workspace for Education model storage will no longer be unlimited, but be limited to a baseline of 100 TB per institution. At this point, we do not have any additional information on how institutions will be transitioned to the new limited storage offering, nor do we have details on how the new limits will be managed at the user level. As we obtain additional information we will make announcements to the CHPC user mailing list.
Owner backup to Box: This is an option suitable for sensitive/restricted data. However there is a file size limitation of 15GB. In addition, if using rclone the credentials expire and have to be reset periodically.
Owner backup to pando: This choice, mentioned in the Archive storage section above, is a good option if a group wishes not to use Google Drive, especially if only a subset of the data needs to be backed up or if a different backup frequency is desired.
Owner backup to other storage external to CHPC: Some groups have access to other storage resources, external to CHPC, whether at the University or at other sites. The tools that can be used for doing this are dependent on the nature of the target storage.
There are a number of tools, mentioned on our Data Transfer Services page, that can be used. Several places above we mentioned rclone which is the tool best suited for transfers to object storage file system; others are fpsync, a parallel version of rsync suited for transfers between typical Linux "POSIX-like" file systems, and globus, best suited for transfers to and from resources outside of the CHPC.
In addition we have a page that presents a number of considerations and tips for user driven backups.
Mounting CHPC Storage
For making direct mounts of home and group space on your local machine see the instructions provided on our Data Transfer Services page.
For more information on CHPC Data policies, visit: File Storage Policies