maindisks.md

---

layout: default
title: Main disk areas
parent: 3. Disk areas
grand_parent: Part 1
nav_order: 1
has_children: false
has_toc: false
permalink: /hands-on/disk-areas/disk-areas-tutorial-maindisks.html

---

Where to store files in CSC's computing environment?

In this tutorial you

• Familiarize yourself with personal and project-specific disk areas and their quotas on CSC supercomputers.

• Learn how to share your files, such as software installations and data, to other project members on CSC supercomputers.

💬 Each user of CSC supercomputers (Puhti and Mahti) have access to different disk areas (or directories) for managing their data. Each disk area has its own specific purpose.

💬 Active data files needed for computational simulations and analyses should be stored and shared in directories under /scratch while any software installations and binaries should be shared under the /projappl directory.

Identify your personal and project-specific directories on Puhti and Mahti supercomputers

1. First login to Puhti using SSH (or by opening a login node shell in the Puhti web interface):

   ssh <username>@puhti.csc.fi    # replace <username> with your CSC username, e.g. [email protected]
   

2. Get an overview of your projects and directories by running the following commands on the login node:

   csc-projects
   csc-workspaces
   

3. Inspect the output information summarizing your directories and their current quotas.

4. Visit your project's /scratch directory and list its contents:

   cd /scratch/<project>/   # replace <project> with your CSC project, e.g. project_2001234
   ls
   

5. Visit your project's /projappl directory and list its contents:

   cd /projappl/<project>/   # replace <project> with your CSC project, e.g. project_2001234
   ls
   

💬 These directories can be briefly summarized as follows:

• User-specific directory (i.e. your personal home folder)

  • Your home directory (path stored in environment variable $HOME)

  • The default directory when you login to Puhti/Mahti

  • You can store configuration files and other minor data for personal use

• Project-specific directories:

  • The project's /scratch and /projappl directories

  • Each project has its own /scratch disk space where most computational tasks are performed. The /scratch area is a temporary space not intended for long-term data storage! Please move inactive data to e.g. Allas.

  • /projappl directory on the other hand is mainly for storing and sharing compiled applications and libraries etc. with other members of the project.

Sharing binaries and data files

💬 Data transfer between two supercomputers can be done e.g. with rsync.

Download the example files

☝🏻 In this example you will download data from Allas object storage.

1. Move to your home folder:

   cd
   

   💡 If you know the files are large, you should consider downloading them directly to /scratch.

2. Download an example program package (ggplot2_3.3.3_Rprogramme.tar.gz) and a data file (Merged.fasta) from the Allas object storage

   wget https://a3s.fi/CSC_training/shared_files.tar.gz
   tar -xavf shared_files.tar.gz
   cd shared_files
   

Let's assume that

• Merged.fasta is a data file intended for computational use

• ggplot2_3.3.3_Rprogramme.tar.gz is a software tool needed for the analysis.

Move the files to Puhti /scratch and /projappl

1. Create folders with your username (using environment variable $USER) in your project directories under /scratch and /projappl on Puhti.

   mkdir -p /projappl/<project>/$USER   # replace <project> with your CSC project, e.g. project_2001234
   mkdir -p /scratch/<project>/$USER    # replace <project> with your CSC project, e.g. project_2001234
   

2. Copy your ggplot2_3.3.3_Rprogramme.tar.gz file to the /projappl directory

   cp ggplot2_3.3.3_Rprogramme.tar.gz  /projappl/<project>/$USER/   # replace <project> with your CSC project, e.g. project_2001234
   

3. Copy the Merged.fasta file to the /scratch directory

   cp Merged.fasta /scratch/<project>/$USER/    # replace <project> with your CSC project, e.g. project_2001234
   

   • Note that all new files and directories are also fully accessible to other members of the project (including read and write permissions).

4. Set read-only permissions for your project members for the file Merged.fasta:

   cd /scratch/<project>/$USER/    # replace <project> with your CSC project, e.g. project_2001234
   chmod g-w Merged.fasta          # g-w means that we "subtract" write permissions for users belong to our group (g), i.e. our project
   

Copying files from Puhti to Mahti (optional)

☝🏻 For this part you must ensure you have forwarded your SSH agent to Puhti, otherwise you will not be able to connect to Mahti.

1. Check if your SSH keys are available on Puhti using command ssh-add -L.

2. If true, it will print your public key. Proceed to step 4.

3. If not:

   1. Linux/macOS: Log out and log back in using ssh -A option.

   2. Windows: Log out. Toggle option Allow agent forwarding found under "Session" -> "SSH" -> "Advanced SSH settings" -> "Expert SSH settings" (MobaXterm) or under "Connection" -> "SSH" -> "Auth" (PuTTY) before connecting again.

4. Change to the folder where you have the example files

5. Copy Merged.fasta file from Puhti to the /scratch drive of Mahti:

   rsync -P Merged.fasta <username>@mahti.csc.fi:/scratch/<project>/$USER/    # replace <username> with your CSC username and <project> with your CSC project, e.g. project_2001234
   

6. Copy the ggplot2_3.3.3_Rprogramme.tar.gz file from Puhti to the /projappl directory on Mahti:

   rsync -P ggplot2_3.3.3_Rprogramme.tar.gz <username>@mahti.csc.fi:/projappl/<project>/$USER/    # replace <username> with your CSC username and <project> with your CSC project, e.g. project_2001234
   

More information

💡 Hint: You can use your folder under /scratch for the rest of the tutorials. You can save the path using an alias (with cd or echo) or somewhere in your notes.

💡 It is sometimes required to export the paths of the /scratch or /projappl directories in environmental variables (until logout). This can be done with the following commands:

export PROJAPPL=/projappl/<project>/   # replace <project> with your CSC project, e.g. project_2001234
export SCRATCH=/scratch/<project>/   # replace <project> with your CSC project, e.g. project_2001234