====== Cyfronet ======

==== Applying for grant and access to Cluster ====

  - Sign up at https://portal.plgrid.pl/
  - Apply for grant and activate it at https://portal.plgrid.pl/services
       * **Note:** Access to the Ares cluster at the Cyfronet centre [grant required]

==== Login to Ares ====

''$ ssh <login>@ares.cyfronet.pl''

<login> = username starting with "plg"

''Add infor about ssh key''
==== Storages ====

  * $HOME
  * $SCRATCH
  * $PLG_GROUPS_STORAGE (starts with “plgg”)

''Add how to share files '' 
==== Install libraries/packages using module ====
<code>$ module avail</code> Lists all available libraries/packages
<code>$ module list</code> Lists loaded libraries/packages
<code>$ module load <package></code> Loads specified package
<code>$ module spider <keyword></code> Searches available packages with the keyword
<code>$ module purge</code> Purges the environment

==== System utilities ====
<code>$ hpc-grants</code> Shows available grants, resource allocations, consumed resourced
<code>$ hpc-fs</code> Shows available storage
<code>$ hpc-jobs</code> Shows currently pending/running jobs
<code>$ hpc-jobs-history</code> : Shows information about past jobs

==== Install Herwig7 ====

  - Load necessary modules: <code>
    module load python/2.7.18-gcccore-11.2.0
    module load libtool
    module load emacs
    module load gcc/11.2.0
    module load gcccore/11.2.0
    module load cmake/3.22.1-gcccore-11.2.0
    module load gsl/2.7-gcc-11.2.0
     </code>
   - Move to $SCRATCH or $PLG_GROUPS_STORAGE (preferred): <code> $ cd $PLG_GROUPS_STORAGE/<groupname> </code> 
   - Download bootstrap script and make it executable: <code> $ wget https://herwig.hepforge.org/downloads/herwig-bootstrap
 $ chmod +x herwig-bootstrap </code>
   - Start an interactive node session (Easier to debug if installation fails): <code> $ srun -p plgrid-now -t 5:00:00 --mem=5GB -A <grantname-cpu> --pty /bin/bash </code> **Note**: Only one interactive plgrid-now session can be run at a time. Adjust the session duration and memory allocation as required. The above specified parameters should suffice in general.
   - Run bootstrap and exit session when completed: <code>[ares][username@ac0787 ~]$ ./herwig-bootstrap -j4 $PWD/Herwig
[ares][username@ac0787 ~]$ exit </code> **Note**: The login01 node changes to ac0xxx when in a computing node.
   - More options for running bootstrap can be found with <code> $ ./herwig-bootstrap --help </code>
   - Activate Herwig environment : <code> $ source Herwig/bin/activate </code>

==== A sample run script (and common slurm commands) ====
Run script (run.sh) : 

       #!/bin/bash
       #SBATCH --job-name=Sample_Run
       #SBATCH --nodes=1
       #SBATCH --ntasks-per-node=1
       #SBATCH --cpus-per-task=1
       #SBATCH --mem-per-cpu=1GB
       # set wall time to 1 day
       #SBATCH --time=1-00:00:00
       ## Name of partition
       #SBATCH -p plgrid
       cd <Run directory>
       source <path/to/>Herwig/bin/activate
       export HW=Herwig
       ${HW} run *.run --seed=$seed -N 1000000 > /dev/null &
       HWPID="$!"
       wait "${HWPID}"


==== Run commands from CLI ====
  - Submit parallel jobs: <code>$ sbatch --array=0-5 -A <grantname-suffix> run.sh</code> 
      * Suffix = cpu, cpu-bigmem, gpu
  - ''sprio'': show status of pending jobs
  - ''sshare'': show usage of available grants
  - Run interactive bash session: <code>$ srun -p plgrid-now -t 1:00:00 --mem=1GB -A <grantname-cpu> --pty /bin/bash</code>

==== More Links ====
   - Detailed information about Ares - [[https://docs.cyfronet.pl/display/~plgpawlik/Ares#Ares-AccesstoAres]]
   - Slurm documentation - [[https://slurm.schedmd.com/quickstart.html]]