Using your own cluster with CyVerse
The CyVerse SDK currently guides developers through the process of creating apps that run on TACC supercomputers, which requires both a TACC account and an active allocation. For users without an active allocation, they can request to be added to the iPlant-Collabs allocation, which allows developers to prototype CyVerse applications. I stress the word prototype because the iPlant-Collabs allocation is relatively small on purpose to make sure unvetted apps aren’t burning away all the compute time allotted for CyVerse. This methodology is great to generate new apps, the apps needs to be reviewed and published by administrators before they can be run at scale on CyVerse. If a user already has access to a high-performance cluster at their own institution, they can circumvent the constrains of iPlant-Collabs by registering their own executionSystem to the Agave API and running apps on it from the CyVerse Discovery Environment.
I am still a student at Indiana University, so I will be registering Mason as an executionSystem to be used with my CyVerse applications. Mason is available to all
- IU students, staff, and faculty
- NSF-funded life sciences researchers
- XSEDE researchers
Apart from the accessibility, the fact that each of the 18 nodes also have 512GB of RAM, makes the system desirable for running memory-hungry software.
To create this and any other executionSystem, you will need
To make creation of the executionSystem JSON easier, I will be providing forms in each section below. Editing any field will automatically update the final JSON for you send to Agave. At no point is any of this information transmitted, so you can safely add your login credentials without worry. However, you have the option to fill in your actual rsa keys before submitting to Agave. The inputs to these forms are pre-populated with information about Mason, so you will need to adapt the values and the JSON description for a different system.
Whenever you launch a job on CyVerse, Agave uses the main CyVerse user credentials to access TACC systems to run jobs.
Similarly, Agave stores and uses your own personal credentials after running tacc-systems-create while following the CyVerse SDK guide.
So, the first requirement to creating a new executionSystem is having
ssh access to it.
I usually access Mason using ssh with the command
$ ssh firstname.lastname@example.org
and then enter my password when prompted. Agave does support password authentication, but handing your password over to strange systems is generally a bad habit. Allowing systems to authenticate to your account using ssh keys is much more secure, and makes it impossible for them to change your account password. Please follow the directions below to generate a set of keys for agave.
[mason]$ cd ~/.ssh/ [mason]$ ssh-keygen -q -t rsa -b 4096 -C email@example.com -f agave_rsa -N '' [mason]$ cat agave_rsa -----BEGIN RSA PRIVATE KEY----- MIIJKAIBAAKCAgEApQQ+n7AsXzAHqkd+xhkmLTIuhLIYiDGNr4DNuFiV+/LsFKRE ZyqacqOwf3ZA51C9XcsPLfaglNrBbCiRl70W8z8zybQUdW3hJPjC7qrG2lAJBfjE BcodjwtR0DVny8JGyWwSdCw6/I4TFxS/+dT6CQ7XoYpfguC8IG4JxbtKxAMaVuAj . . . Owhcksd0X5swsCLzoivhf+EpALcohwAwR6fbreavprdIJikwDIBs0ruPOZ7nk4nz OOYL2M7VZ1XdHzg70IvblLDZCOX/sghRPvBZZwdKHVbCDlLBKuOP4CdnZmpCD3Yx V/qIgePP+LanugxaRdzdZLcQJ7crk1L+3/2McMWaAtNwpvDCk31rQhIVpJE= -----END RSA PRIVATE KEY----- [mason]$ cat agave_rsa.pub ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAgEApQQ+n7AsXzAHqkd+xhkmLTIuhLIY iDGNr4DNuFiV+/LsFKREZyqacqOwf3ZA51C9XcsPLfaglNrBbCiRl70W8z8zybQU dW3hJPjC7qrG2lAJBfjEBcodjwtR0DVny8JGyWwSdCw6/I4TFxS/+dT6CQ7XoYpf guC8IG4JxbtKxAMaVuAjV1VH+Z7gl+mNu/gbWtX16GD0SCM/C7YY3fPisHDfyJGi oHMNlcPC+YrHIoRuHphSdCKUImIVzAKkKr8fnayFPXi2aP095OJAdhGhvTrIyCAg KjeAZFQK4s0HBwmUJR6BK6UwuHw0M95T8aBgSj48vbN53nvTFcnq3ECFCnKdO1F1 Q8f5zen+ExDaExdir3C3ukHf+NR2ZMrOm++YtxLjlYCK/QUj8BN+RFEeQQFBCg6j 6XjitBAKdH49cnYwvxTkbsrC50xiM0V6pVfzsX4LpHjoB+yu5etaT45bcLS5h1TR 8b7mo9ohvseLE9EqAHLhDzmiu9v0VxF3I2txkYfxWS09Syfn7914FYBgF9zQs4hW lJKIsWo03nTTtZo+z/o+5hzpfJZ61acAM7PtRgXrcGwwnMo5N45CrNj17yfzsKFN hoYscep6R3el6jxsjSxA+/JYawkZtg3dnKhG5XXQW1+SrQVwbh7u+p+5wMtQhVss 32h20jr/+hHKb8U= firstname.lastname@example.org
Your public key now needs to be added to your authorized key file for automatic authentication.
[mason]$ cat agave_rsa.pub >> ~/.ssh/authorized_keys
Copy your entire private key (agave_rsa), including the
--- header and footer and paste into the corresponding field below. Do the same with your public key (agave_rsa.pub), including the “ssh-rsa” and your email at the end.
Remember, if you ever want to revoke Agave’s access to your system, you only need to
[mason]$ rm ~/.ssh/agave_rsa*
to permanently delete the keys from the system. After granting basic access, Agave will need specific knowledge about the cluster itself.
When Agave runs a job on CyVerse, it submits a job to the SLURM schedulers at TACC. Agave needs to know what scheduler your cluster runs, which queue, and any other necessary accounting information. The SLURM scheduler on Stampede enforces many rules to ensure that user experiences are fair and consistent. Mason doesn’t have as many, but it is good practice to know the constraints of each system you use.
|Max Runtime (hours)|
|Custom queue directives|
You technically can register a single workstation to Agave by setting the executionType to
CLI, but this guide is looking to enable large-scale computing, so we and the automatic JSON are going to focus on registering a cluster.
Agave also needs information on the physical configuration of each compute node in your desired queue so resources, so resources can never be over-requested by a job.
|Max Processors per Node|
|Max Memory per Node (GB)|
Lastly, Agave needs to know what paths to use for storing job data each time an app is run.
Before submitting to a scheduler, Agave first stages all app (binaries) and input data in a folder unique to that job.
This folder is created in a location relative to a directory you choose.
Our home path should be set to our
$HOME directory so Agave can find our
.bashrc and properly load our environment.
We should also set our scratch path to a directory with a lot of storage and also capable of handling a lot of i/o activity. At TACC, we would set it to our
$SCRATCH directory, but am going to use my Data Capacitor directory for running jobs. Jobs are stored in the following layout:
scratchDir/ |- user1 |- job0001 \- job0002 |- user2 \- job0005 \- user3 |- job0006 \- job0010
To keep clutter down in the root of my personal folder, I’m going to create a CyVerse folder
[mason]$ mkdir /N/dc2/scratch/user/CyVerse
for Agave to store user and job directories in. In the fields below, please change
user to your own username, or change the locations to a more preferable directory.
Registering with Agave
Assuming you have already have an installed and working version of the CyVerse SDK, you just need to copy the JSON above into a file on your system. In my example below, I’ll be using
[mason]$ auth-tokens-refresh -S Token for iplantc.org:user successfully refreshed and cached for 14400 seconds 45098349583490859034859304859043 [mason]$ systems-addupdate -F user-mason.json Successfully added system user-IU-mason [mason]$ systems-list -Q user-IU-mason
Assuming you successfully added your new Mason system, you can quickly test it by listing the files in your home directory. If this does not work, double-check that your password is correct and consistent in both
auth sections of the JSON file.
[mason]$ files-list -S user-IU-mason . . .bash_history .bash_logout .bash_profile .bashrc .forward .history .lesshst .local .modulesbeginenv .mozilla .ssh .vim .viminfo .Xauthority 454AllContigs.fna blastScript.sh blatScript.sh combineFiles.sh gzipFiles.sh hs_err_pid1987.log runBlast.py runBlat.py scripts scripts_backup.tar.gz software splitFiles.sh tmp trinity-avo.sh [mason]$ ls -a ~ . .forward scripts .. gzipFiles.sh scripts_backup.tar.gz 454AllContigs.fna .history software .bash_history hs_err_pid1987.log splitFiles.sh .bash_logout .lesshst .ssh .bash_profile .local tmp .bashrc .modulesbeginenv trinity-avo.sh blastScript.sh .mozilla .vim blatScript.sh runBlast.py .viminfo combineFiles.sh runBlat.py .Xauthority
Your new Mason executionSystem is ready for use! In my next post, I plan on demonstrating how to clone and deploy apps to personal systems. If you can’t wait, just follow the CyVerse SDK, but specify Mason as your executionSystem in your app description.
2017-02-10 - Guide was updated to use ssh keys instead of passwords for authentication.
07 Feb 2017