09_Containers_SRT_English_mac.srt

1
00:00:22,566 --> 00:00:27,616
Why should we care for Singularity containers in HPC environments? 

2
00:00:28,149 --> 00:00:33,166
In HPC centers applications are often installed as containers.

3
00:00:33,383 --> 00:00:36,416
CSC makes no exception to that. 

4
00:00:36,933 --> 00:00:40,000
There may be slight differences in how you use regular software 

5
00:00:40,000 --> 00:00:41,983
and software in a container.

6
00:00:42,649 --> 00:00:45,533
Check the documentation of that particular software 

7
00:00:45,533 --> 00:00:48,466
to see whether it is installed as a container. 

8
00:00:48,816 --> 00:00:52,700
Users also benefit from the use of containers.

9
00:00:53,049 --> 00:00:58,100
They can install software easily - no need for HPC admins. 

10
00:00:58,383 --> 00:01:01,516
If a container with desired software exists, 

11
00:01:01,516 --> 00:01:05,183
users need only a single command to install it.

12
00:01:12,200 --> 00:01:16,833
Let me tell you an example usecase where containers can solve the problem.

13
00:01:17,299 --> 00:01:22,750
A scientific developer writes some code and wants to hand it over to a friend for testing. 

14
00:01:23,133 --> 00:01:28,633
The friend tells the developer that the test code is not working in the test environments. 

15
00:01:29,049 --> 00:01:34,849
Turns out the first test environment has slightly different versions of the used libraries. 

16
00:01:35,083 --> 00:01:39,266
The second test environment has a different operating system. 

17
00:01:39,883 --> 00:01:42,916
How to make sure that same code works on your computer 

18
00:01:42,916 --> 00:01:46,366
and in HPC environment or cloud environment?

19
00:01:46,566 --> 00:01:50,283
That is the challenge that containers are going to solve. 

20
00:01:50,450 --> 00:01:53,299
Basically if something works on your computer, 

21
00:01:53,299 --> 00:01:57,533
it should work anywhere regardless of the underlying host environment. 

22
00:01:59,500 --> 00:02:03,250
A container packages everything needed to run the application 

23
00:02:03,250 --> 00:02:06,099
 - i.e. their code and all dependencies.

24
00:02:06,466 --> 00:02:10,266
The container image can be run like a binary file.

25
00:02:10,766 --> 00:02:13,783
There are many container platforms around.

26
00:02:13,783 --> 00:02:17,383
The most popular container engine is probably Docker.

27
00:02:17,750 --> 00:02:24,533
Singularity is very popular in HPC environments and it is used also in CSC environment. 

28
00:02:33,733 --> 00:02:38,750
Both containers and virtual machines are solving this dependency issue.

29
00:02:39,033 --> 00:02:44,066
The comparison here shows three ways of deploying your applications. 

30
00:02:44,099 --> 00:02:48,699
First is the native application - i.e. you compile and launch it. 

31
00:02:49,083 --> 00:02:53,050
Then it depends on your environment whether it works or not.

32
00:02:53,449 --> 00:02:56,933
The second one is the virtual machine approach. 

33
00:02:57,366 --> 00:03:00,083
Virtual machine applications are usually rendered 

34
00:03:00,083 --> 00:03:03,099
through a special software called hypervisor.

35
00:03:03,599 --> 00:03:08,099
Virtual machine includes OS and Kernel like a normal computer.

36
00:03:08,583 --> 00:03:13,616
It basically is a virtual computer and there you can install everything from the scratch.

37
00:03:14,199 --> 00:03:17,433
Another user can install the saved image of the system 

38
00:03:17,433 --> 00:03:20,016
and use it with good compatibility.

39
00:03:21,333 --> 00:03:24,033
Third option is the container approach, 

40
00:03:24,033 --> 00:03:27,083
which is a bit more lightweight than a virtual machine. 

41
00:03:27,650 --> 00:03:33,150
You just install the container engine, but not kernel or any operating system. 

42
00:03:33,633 --> 00:03:38,650
So the key difference between virtual machines and containers is how they use kernel.

43
00:03:39,216 --> 00:03:41,866
Containers share the kernel with the hardware, 

44
00:03:41,866 --> 00:03:44,900
whereas virtual machines have their own kernel. 

45
00:03:45,316 --> 00:03:49,833
From the user point of view: virtual machines boot the operating system 

46
00:03:49,833 --> 00:03:54,383
which takes time, and containers can be loaded in milliseconds.

47
00:04:03,150 --> 00:04:06,199
Virtual machines are useful if you need to run a different 

48
00:04:06,199 --> 00:04:08,783
Operating System than your regular system. 

49
00:04:09,166 --> 00:04:13,699
You may have MacOS or Linux and then you need some Windows software 

50
00:04:13,699 --> 00:04:18,366
 - so you can install Windows into a virtual machine and run that on your Mac.

51
00:04:18,683 --> 00:04:21,633
If you need to use Linux from another OS you can 

52
00:04:21,633 --> 00:04:25,149
open an SSH-connection to Puhti or Mahti.

53
00:04:25,149 --> 00:04:27,750
But that you probably knew already!

54
00:04:28,766 --> 00:04:33,800
Containers can not run a different OS because the share the kernel with the host.

55
00:04:34,266 --> 00:04:38,733
They can have their own libraries - and that is how they solve the dependency issue 

56
00:04:38,733 --> 00:04:43,149
which I described earlier with the example code test case, remember?

57
00:04:43,600 --> 00:04:44,433
No?

58
00:04:45,183 --> 00:04:48,483
Containers can have a different Linux distribution 

59
00:04:48,483 --> 00:04:52,216
 - like Centos or Ubuntu - than the host has.

60
00:04:59,666 --> 00:05:05,600
Containers are popular e.g. because they provide an easy way of distributing software.

61
00:05:06,050 --> 00:05:08,566
They can be used like binary files, 

62
00:05:08,566 --> 00:05:11,783
which means that once you have the container you just run it.

63
00:05:12,100 --> 00:05:17,966
Including all the dependencies inside the containers makes it portable and compatible.

64
00:05:19,216 --> 00:05:22,600
Singularity containers can be run as a normal user 

65
00:05:22,600 --> 00:05:26,000
so they work nicely in HPC environments. 

66
00:05:26,416 --> 00:05:31,016
In comparison, Docker containers are inconvenient in HPC systems

67
00:05:31,016 --> 00:05:34,016
because they require root access to run.

68
00:05:34,216 --> 00:05:41,316
Users do not have root access in multi-user IT environment, like HPC environments. 

69
00:05:42,649 --> 00:05:46,566
In general all containers need root access when built.

70
00:05:47,166 --> 00:05:52,199
For that you can use your local computer in which you probably have the root access. 

71
00:05:52,433 --> 00:05:56,850
There you can install all the packages that you want inside the container.

72
00:05:57,483 --> 00:06:00,866
Then you can copy the container into CSC enviroment

73
00:06:00,866 --> 00:06:03,350
and run it there with normal user rights.

74
00:06:03,933 --> 00:06:07,533
That provides a convenient way of transforming your code 

75
00:06:07,533 --> 00:06:11,850
 - with all your custom libraries - to the supercomputer.

76
00:06:20,683 --> 00:06:25,733
While sharing the kernel with the host, containers have their own environment.

77
00:06:26,066 --> 00:06:29,016
That means they can have different Linux distribution

78
00:06:29,016 --> 00:06:31,883
which can include different libraries.

79
00:06:32,083 --> 00:06:36,233
Your code might require Centos while the host has Ubuntu, 

80
00:06:36,233 --> 00:06:38,899
so you can have Centos in the container.

81
00:06:39,233 --> 00:06:45,816
There might be some exeptions to that e.g. when working with GPUs or MPI libraries. 

82
00:06:46,149 --> 00:06:49,983
Still in general containers solve many incompatibilities 

83
00:06:49,983 --> 00:06:53,800
and are less likely to be effected by changes in the host system.

84
00:07:03,483 --> 00:07:06,600
Containers ensure that your computational environment 

85
00:07:06,600 --> 00:07:09,366
and your collaborators environment are the same.

86
00:07:10,033 --> 00:07:12,883
You can save a whole analysis environment, 

87
00:07:12,883 --> 00:07:16,750
which means e.g. all the libraries and their versions. 

88
00:07:18,733 --> 00:07:24,466
That means you can easily share the environment with collaborators as a single file. 

89
00:07:30,016 --> 00:07:33,800
Singularity is a container platform like Docker

90
00:07:34,266 --> 00:07:38,949
Singularity containers can be run with a basic user level rights. 

91
00:07:39,350 --> 00:07:43,566
Note that you cannot elevate your user privileges inside the container 

92
00:07:43,566 --> 00:07:46,683
 - which is also good from administration point-of-view. 

93
00:07:48,399 --> 00:07:51,666
Containers are faster to deploy than virtual machines 

94
00:07:51,666 --> 00:07:56,083
because they only load the libraries, not a full operating system.

95
00:07:56,616 --> 00:07:59,750
Singularity containers support MPI, 

96
00:07:59,750 --> 00:08:04,250
but make sure to use compatible MPI libraries inside the container.

97
00:08:04,750 --> 00:08:07,883
Here compatible means the hosts should have at least the version

98
00:08:07,883 --> 00:08:10,083
which is available in the container. 

99
00:08:10,533 --> 00:08:16,266
Singularity can use also a driver stack e.g. when using GPUs.

100
00:08:16,550 --> 00:08:21,000
That means you don't need to install drivers inside the container, 

101
00:08:21,000 --> 00:08:25,866
but the container can use the driver that is already available in the whole system.

102
00:08:26,016 --> 00:08:30,649
Option --nv tells the container to use driver stack. 

103
00:08:31,083 --> 00:08:36,100
There are a lot of Docker images around, because Docker is very popular.

104
00:08:36,549 --> 00:08:40,316
You can convert Docker containers into Singularity containers 

105
00:08:40,316 --> 00:08:43,516
and use that also in an HPC environment.

106
00:08:54,000 --> 00:08:58,700
In general containers should be run as a batch job or an interactive job.

107
00:08:59,183 --> 00:09:04,716
Singularity commands are available also in the login node - no need to load a module. 

108
00:09:05,166 --> 00:09:10,200
Users can always bring their own containers and start using them right away. 

109
00:09:10,916 --> 00:09:15,500
Some software in Puhti or Mahti are installed as containers.

110
00:09:15,866 --> 00:09:21,333
Always check software information in DocsCSC for details on usage.

111
00:09:31,833 --> 00:09:36,483
There is only three commands needed for running Singularity containers. 

112
00:09:36,950 --> 00:09:40,516
The first one is singularity exec with container name 

113
00:09:40,516 --> 00:09:43,149
and a command that you want to run in the container. 

114
00:09:43,533 --> 00:09:46,616
It let's you to use same commands that you use 

115
00:09:46,616 --> 00:09:49,299
for your application also with the container.

116
00:09:49,566 --> 00:09:54,183
Command singularity run runs the default action of the container.

117
00:09:54,666 --> 00:09:58,216
The default script is defined when building the container.

118
00:09:58,600 --> 00:10:01,966
If you want to interactively work inside the container 

119
00:10:01,966 --> 00:10:05,600
 - instead of the host command line - you can open a command line

120
00:10:05,600 --> 00:10:10,433
inside the container with command singularity shell and the container name.

121
00:10:10,850 --> 00:10:16,700
Then you will be in a different file system environment that is built inside the container.

122
00:10:22,766 --> 00:10:28,066
The container file system includes the files that are put there when the container is built.

123
00:10:28,783 --> 00:10:33,450
You can always go inside the container and check out the file system there. 

124
00:10:34,200 --> 00:10:36,733
Usually containers are read only systems

125
00:10:36,733 --> 00:10:40,333
which means you cannot change the files inside the container.

126
00:10:41,750 --> 00:10:46,399
Often the container needs to access some files in the host computer.

127
00:10:46,850 --> 00:10:51,883
To do that we can mount or bind host directories inside the container. 

128
00:10:52,233 --> 00:10:56,399
Note that they are writable so file changes there are permanent. 

129
00:10:56,799 --> 00:11:01,250
To mount a directory into a container use the flag --bind. 

130
00:11:01,700 --> 00:11:08,516
If the target directory - in this example /data - does not exist, it is created automatically.

131
00:11:08,633 --> 00:11:14,450
If you don't define the target directory it will use the host directory name by default. 

132
00:11:14,683 --> 00:11:18,616
If you need to bind more than one host folders to a container, 

133
00:11:18,616 --> 00:11:22,083
just write more folders after the bind flag.

134
00:11:28,700 --> 00:11:32,633
Environment variables like HOME and USER get their values typically 

135
00:11:32,633 --> 00:11:36,283
through some functionality built into the operating system.

136
00:11:36,683 --> 00:11:41,000
Most environment variables are inherited by the containers.

137
00:11:41,433 --> 00:11:45,333
HPC systems generally have many environmental variables 

138
00:11:45,333 --> 00:11:48,183
and some of them might interfere with your container.

139
00:11:48,450 --> 00:11:53,100
If you want to prevent the container inheriting all the environmental variables

140
00:11:53,100 --> 00:11:55,616
you can use flag clean-env.

141
00:11:56,033 --> 00:11:59,116
However you can define environment variables

142
00:11:59,116 --> 00:12:01,600
so that they are used only in the container.

143
00:12:02,083 --> 00:12:07,266
Defining an environmental variable singularity env test in the host creates 

144
00:12:07,266 --> 00:12:11,183
such a variable that it shows as test inside the container.

145
00:12:17,133 --> 00:12:20,916
Already worrying too much about mounts and variables?

146
00:12:20,916 --> 00:12:22,083
Fear not!

147
00:12:22,083 --> 00:12:27,350
We at CSC provide you with a special wrapper for this singularity application. 

148
00:12:27,883 --> 00:12:32,033
This singularity_wrapper takes care of common binds so you are able to see 

149
00:12:32,033 --> 00:12:36,316
your home, scratch and projappl directories inside the container.

150
00:12:37,033 --> 00:12:40,350
Here this image.sif is the container image file.

151
00:12:40,733 --> 00:12:46,366
Most modules also set this singularity image as special environment variable.

152
00:12:46,750 --> 00:12:50,850
Then you do not need to write the image name the wrapper command.

153
00:12:58,149 --> 00:13:01,750
A lot of applications are written as Docker containers. 

154
00:13:02,083 --> 00:13:07,016
I already mentioned that you can use the Docker containers in Singularity. 

155
00:13:07,016 --> 00:13:10,366
The command singularity build and the docker address converts

156
00:13:10,416 --> 00:13:13,766
a Docker container in to a Singularity container.

157
00:13:13,766 --> 00:13:18,316
You can do that with basic user privileges - which is dope.

158
00:13:19,166 --> 00:13:22,350
Let's take this PyTorch image as an example.

159
00:13:23,016 --> 00:13:28,049
You can define the name for the new converted Singularity container by yourself.

160
00:13:28,683 --> 00:13:32,250
But the name of the source Docker container has to match to the one

161
00:13:32,250 --> 00:13:34,733
that exists in the original repository.

162
00:13:34,899 --> 00:13:37,000
So have the address right.

163
00:13:38,149 --> 00:13:40,883
Check the documentation about running containers and 

164
00:13:40,883 --> 00:13:44,916
creating Singularity containers from DocsCSC.

165
00:13:52,733 --> 00:13:56,299
It is quite clear already that Singularity containers provide 

166
00:13:56,299 --> 00:14:01,133
a convenient installation method, especially in HPC environments.

167
00:14:01,500 --> 00:14:06,016
If the software exists as a container, use that and you do not need to worry

168
00:14:06,016 --> 00:14:10,716
about the dependencies, because they can get very complex sometimes.

169
00:14:11,100 --> 00:14:14,733
Also some old versions of the dependency softwares may be

170
00:14:14,733 --> 00:14:17,266
incompatible with the host environment.

171
00:14:17,649 --> 00:14:21,350
For example with scientific publications it is important 

172
00:14:21,350 --> 00:14:25,116
that you can reproduce the results even after a few years. 

173
00:14:25,416 --> 00:14:29,416
Going back to same data analysis with same libraries and versions

174
00:14:29,416 --> 00:14:31,883
after many years could get tricky. 

175
00:14:32,250 --> 00:14:35,533
With containers you can ensure that you have the same image 

176
00:14:35,533 --> 00:14:38,100
and running it leads to similar results. 

177
00:14:38,283 --> 00:14:43,333
Consider using containers even though other methods were available.

178
00:14:50,816 --> 00:14:54,899
Here is an example of a software that has three methods of installation 

179
00:14:54,899 --> 00:14:57,583
in CSC computing environment.

180
00:14:58,183 --> 00:15:02,750
The first one is the native application compiled from the source code.

181
00:15:03,250 --> 00:15:09,716
When installed it consists of 47 files with total size of 1.9 MB. 

182
00:15:10,049 --> 00:15:14,533
The second option in comparison is Conda which is quite easy to use.

183
00:15:15,183 --> 00:15:22,683
Installing the same software with Conda results in 27000 files with total size of 1.1 GB. 

184
00:15:23,166 --> 00:15:28,666
With so many files it hammers the Lustre file system and the whole Puhti slows down.

185
00:15:29,066 --> 00:15:34,083
We have written a DocsCSC page telling why you should not use Conda.

186
00:15:34,416 --> 00:15:38,866
Then the point of this comparison: Singularity container.

187
00:15:39,516 --> 00:15:46,416
There you have only one file - the container image - with total size of 339 MB.

188
00:15:47,066 --> 00:15:52,083
So containers may get quite big which may limit the usability sometimes.

189
00:15:52,416 --> 00:15:57,100
But remember that the container has also the dependencies installed.

190
00:15:57,583 --> 00:16:01,733
With only one file it is Lustre-friendly and user-friendly.

191
00:16:11,433 --> 00:16:14,833
In case you don't find a container that you want to use, 

192
00:16:14,833 --> 00:16:16,966
you might need to build your own.

193
00:16:17,299 --> 00:16:20,799
It is a bit more involved process and requires root access 

194
00:16:20,799 --> 00:16:23,350
so you need to use your own computer.

195
00:16:23,549 --> 00:16:27,433
You can start the building with the sandbox mode - if you want.

196
00:16:28,016 --> 00:16:32,533
Open a shell inside the container to do the software installations.

197
00:16:32,833 --> 00:16:36,016
If the base system is Ubuntu, you can use apt

198
00:16:36,016 --> 00:16:40,583
or then in CentOS use yum to install packages and dependencies.

199
00:16:40,899 --> 00:16:45,950
Consult software developer documentation for spesific instructions.

200
00:16:54,333 --> 00:16:58,466
You can build the production image using the sandbox version.

201
00:16:58,816 --> 00:17:04,416
Another option is that you write a definition file which is kind of a recipe of the container.

202
00:17:04,849 --> 00:17:06,816
Once you have the production image, 

203
00:17:06,816 --> 00:17:10,983
you can transfer it to CSC environment and start working. 

204
00:17:13,750 --> 00:17:17,533
The tutorials about containers continue from here. 

205
00:17:17,900 --> 00:17:22,116
They cover the basic use cases with easy-to-follow examples.

206
00:17:22,566 --> 00:17:26,349
Container documentation covers the introduction to containers 

207
00:17:26,349 --> 00:17:28,633
including technical details.