Difference between revisions of "Containers and virtual machines"

From Wiki at Neela Nurseries
Jump to: navigation, search
m (adding couple references related to nginx config and port in use trouble shooting.)
m (→‎^ Dockerfiles: How to save a Docker image as a compressed file)
 
(66 intermediate revisions by the same user not shown)
Line 1: Line 1:
  
 
<center>
 
<center>
−
[[Unix_and_Linux_config|Unix and Linux config]] :: [[Containers_and_virtual_machines|Containers]] :: [[Compute_clusters|Clusters]] :: &lt;link&gt;
+
[[Containers_and_virtual_machines|Containers]] :: [[RTOS|RTOS Notes]] :: [[Compute_clusters|Clusters]] :: [[Unix_and_Linux_config|Unix and Linux config]]
 
</center>
 
</center>
−
 
  
 
== [[#top|^]] Container Software ==
 
== [[#top|^]] Container Software ==
Line 11: Line 10:
 
   *  https://medium.com/sysf/docker/home
 
   *  https://medium.com/sysf/docker/home
  
−
A general note following about three days' Docker container experimentation, and to question "should I be using a Docker container interactively, running multiple apps, some simultaneously?".  <i>(Note the following forum post chain is a quagmire of issues and opions!)</i>:
+
A general note following about three days' Docker container experimentation, and to question "should I be using a Docker container interactively, running multiple apps, some simultaneously?".  <i>(Note the following forum post chain is a quagmire of issues and opinions!)</i>:
  
 
   *  https://news.ycombinator.com/item?id=7950326
 
   *  https://news.ycombinator.com/item?id=7950326
  
−
=== [[#top|^]] Docker starting point ===
+
== [[#top|^]] Docker ==
  
 
Notes on Docker containerizing software.  Note, to get a practical start it is a good and or needed step to create a user account with Docker dot com.  Single user and free accounts with some advanced Docker features disabled are available.  A good starting tutorial for beginners is written by one Brian Hogan of Digital Ocean, this article published 2018 July 5.  First reference in list here:
 
Notes on Docker containerizing software.  Note, to get a practical start it is a good and or needed step to create a user account with Docker dot com.  Single user and free accounts with some advanced Docker features disabled are available.  A good starting tutorial for beginners is written by one Brian Hogan of Digital Ocean, this article published 2018 July 5.  First reference in list here:
 +
 
<ul>
 
<ul>
 
*  [https://www.digitalocean.com/community/tutorials/how-to-install-and-use-docker-on-ubuntu-18-04 Docker on Ubuntu 18.04 LTS]
 
*  [https://www.digitalocean.com/community/tutorials/how-to-install-and-use-docker-on-ubuntu-18-04 Docker on Ubuntu 18.04 LTS]
 
*  [https://www.digitalocean.com/community/tutorials/how-to-install-and-use-docker-on-ubuntu-20-04 Docker on Ubuntu 20.04 LTS]
 
*  [https://www.digitalocean.com/community/tutorials/how-to-install-and-use-docker-on-ubuntu-20-04 Docker on Ubuntu 20.04 LTS]
 
*  https://www.digitalocean.com/community/tutorials/how-to-remove-docker-images-containers-and-volumes  Remove Docker image
 
*  https://www.digitalocean.com/community/tutorials/how-to-remove-docker-images-containers-and-volumes  Remove Docker image
 +
2023-06-07:
 +
*  [https://collabnix.com/how-to-fix-cannot-connect-to-the-docker-daemon-at-unix-var-run-docker-sock-error-message/ Docker cannot connect, blog by Ajeet Raina]
 +
</ul>
 +
 +
Ted noting too there may be a daily limit to the number of docker images which a user with a free account can push to Docker's image repository.  Docker's image repository provides URLs of the form <code>https://hub.docker.com/u/&lt;user_name&gt;</code>, where username is the given person's Docker account user name.4
 +
 +
Returning to Docker a year to two further (2023-06-26) here is an article linked directly from [https://www.digitalocean.com/community/tutorials/how-to-install-and-use-docker-on-ubuntu-20-04 Brian Hogan's How To Install...] article at Digital Ocean.  This article talks about creating multiple Docker containers, many of which contain each other in a layered fashion by design:
 +
 +
<ul>
 +
*  https://www.digitalocean.com/community/tutorials/the-docker-ecosystem-an-introduction-to-common-components
 +
</ul>
 +
 +
This article introduces some basic concepts regarding the Docker ecosystem.  And a good article here to lay out the distinctions between a Docker image and a Docker container:
 +
 +
<ul>
 +
*  https://circleci.com/blog/docker-image-vs-container/
 +
</ul>
 +
 +
<b>&gt; Docker registries and repositories:</b>
 +
 +
Nick Janetakis writes a pretty concise brief article to describe what are Docker repositories and Docker registries.  An image repository holds versions (committed and tagged versions) of a given Docker image.  A registry holds zero or more Docker repositories, and is generally a full-fledged always live service that's accessible on the internet or on an intranet.
 +
 +
<ul>
 +
*  https://nickjanetakis.com/blog/docker-tip-53-difference-between-a-registry-repository-and-image
 +
</ul>
 +
 +
<!-- comment -->
 +
 +
== [[#top|^]] Dockerfiles ==
 +
 +
An article and tutorial on crafting a basic Dockerfile:
 +
 +
<ul>
 +
*  https://thenewstack.io/docker-basics-how-to-use-dockerfiles/
 +
</ul>
 +
 +
Ubuntu Docker images are the most downloaded on hub.docker.com.  Here is an article on how to customize this image:
 +
<ul>
 +
*  https://octopus.com/blog/using-ubuntu-docker-image
 +
</ul>
 +
 +
How to start Docker image with interactive shell:
 +
 +
<ul>
 +
*  https://www.letscloud.io/community/how-to-launch-a-docker-container-with-an-interactive-shell
 +
</ul>
 +
 +
<span id="nn_anchor__save_docker_image_to_file"></span>
 +
How to save a Docker image as a compressed file:
 +
<ul>
 +
*  https://stackoverflow.com/questions/23935141/how-to-copy-docker-images-from-one-host-to-another-without-using-a-repository
 
</ul>
 
</ul>
−
Ted noting too there may be a daily limit to the number of docker images which a user with a free account can push to Docker's image repository.  Docker's image repository provides URLs of the form <code>https://hub.docker.com/u/&lt;user_name&gt;</code>, where username is the given person's Docker account user name.
 
  
 +
<!-- odne komentar -->
  
−
=== [[#top|^]] specific commands ===
+
== [[#top|^]] Docker commands ==
  
−
Start a docker image with access to a particular host directory:
+
<i><b>NOTE:</b>  as of 2021-07-19 Monday this section needs organizing and annotation work.  - TMH</i><br />
 +
This first reference URL was added when Ted did not know how to start a Docker image running and grant it access to a host OS directory hierarchy:
  
 
*  https://stackoverflow.com/questions/55104551/how-to-access-files-in-host-from-a-docker-container
 
*  https://stackoverflow.com/questions/55104551/how-to-access-files-in-host-from-a-docker-container
 
<ul>
 
<ul>
−
More on Docker volumes:
+
More on Docker volumes at this next URL.  Note also that there is an older Docker volume mechanism and a newer more flexible one, but that may be detailed in another tutorial or guide:
 
*  https://docs.docker.com/storage/volumes/
 
*  https://docs.docker.com/storage/volumes/
 
</ul>
 
</ul>
  
−
==== [[#top|^]] docker commit ... ====
+
<!-- odne komentar -->
 +
 
 +
=== [[#top|^]] Docker pull ===
  
 +
Example command to pull a Docker image from a private registry:
 +
docker pull registry.some_remote_host.com/ubuntu-20-04:version-1p0
 +
 +
When pulling Docker images from Docker's primary community site, hub.docker.com, only the Docker username is needed prior to the image name:
 +
docker pull tedhavelkaad0602/ubuntu-git-plus-arm-none-eabi-gcc:version-0p8
 +
 +
<!-- odne komentar -->
 +
 +
=== [[#top|^]] Docker run ===
 +
 +
Very limited but useful example of how to run a docker image, with two host OS directory hierarchies accessible in the Docker container:
 
<pre>
 
<pre>
 +
# docker run -i -t -v /home/cpguest/projects/chibios:/home/huesped/project/chibios -v /mnt/host_os_downloads:/mnt/host_os_downloads --privileged bf980d48d9ab
 +
</pre>
 +
 +
<!-- odne komentar -->
 +
 +
=== [[#top|^]] Docker commit ===
  
 +
To commit changes within a container a user can invoke `docker commit ...`.  While it is good to know about Docker commit command, commits of container changes are not trackable and maintainable nearly as practically as Dockerfiles.  Here is the official documentation for this commit command, followed by a tutorial on Docker commit command with preliminary steps of writing a Dockerfile:
 +
 +
<ul>
 +
*  https://docs.docker.com/engine/reference/commandline/commit/
 +
 +
*  https://linuxhint.com/use-docker-commit-to-change-container-images/
 +
</ul>
 +
 +
<i>Figure:  example Docker commit:</i>
 +
<pre>
 +
docker commit -m "Created 'use marker' file in root homedir of ubuntu-20-04-lts:version-1p0 image." -a "Sr Fulano" ad494bcfe010 private_registry.some_host.com/ubuntu-20-04:version-1p0
 
</pre>
 
</pre>
  
  
−
==== [[#top|^]] docker push &lt;image_identifier_string_and_optional_tag&gt; ====
+
 
 +
<!-- odne komentar -->
 +
 
 +
=== [[#top|^]] Docker push ===
 +
 
 +
Incomplete synopsis of `docker push` command:
 +
 
 +
  $ docker push [&lt;optional_remote_registry&gt;/]&lt;image_name&gt;[:&lt;optional_tag&gt;]
  
 
Series of <code>docker push</code> invocations for a work-in-progress image.  Note the tag names are optional until one needs push an image that's being amended, such that it is a newer or different version of an existing image on the remote Docker repository:
 
Series of <code>docker push</code> invocations for a work-in-progress image.  Note the tag names are optional until one needs push an image that's being amended, such that it is a newer or different version of an existing image on the remote Docker repository:
Line 53: Line 142:
 
   962  docker push tedhavelkaad0602/ubuntu-git-plus-arm-none-eabi-gcc:version-0p4
 
   962  docker push tedhavelkaad0602/ubuntu-git-plus-arm-none-eabi-gcc:version-0p4
 
   965  docker push tedhavelkaad0602/ubuntu-git-plus-arm-none-eabi-gcc:version-0p5
 
   965  docker push tedhavelkaad0602/ubuntu-git-plus-arm-none-eabi-gcc:version-0p5
 +
</pre>
 +
 +
And here is a link to an article describing Docker push command syntax:
 +
 +
<ul>
 +
*  https://www.educba.com/docker-push/
 +
</ul>
 +
 +
We'll note this link also in the [[#references|References]] section of this page.  Takeaway here is that the third token in the above push command examples, the latter four examples, that token has the form "<docker_user_name>/<image_name>:<docker_tag>".
 +
 +
This article helped get us closer to understanding how Docker CLI parses the argument to a given <code>docker push</code> argument.  Even with this understanding some further trial and error helped clarify what is needed in the form of this argument, which identifies a Docker image to be pushed to a private, non-docker.io registry.  Some excerpts from the command line:
 +
 +
<pre>
 +
vmguest@vm-ubuntu-0p2:~$ docker images
 +
REPOSITORY                                          TAG          IMAGE ID      CREATED        SIZE
 +
ubuntu-20-04-lts                                    version-0p0  065cf14a189c  8 days ago    135MB
 +
localhost:5000/my-ubuntu                            latest        065cf14a189c  8 days ago    135MB
 +
tedhavelkaad0602/ubuntu-git-plus-arm-none-eabi-gcc  version-0p8  bf980d48d9ab  8 days ago    2.98GB
 +
tedhavelkaad0602/ubuntu-git-plus-arm-none-eabi-gcc  version-0p2  4110ca98063a  2 weeks ago    309MB
 +
tedhavelkaad0602/ubuntu-git-plus-arm-none-eabi-gcc  latest        f24e27e3d998  2 weeks ago    245MB
 +
registry                                            2            1fd8e1b0bb7e  2 months ago  26.2MB
 +
docker-registry.neelanurseries.com/hello-world      version-0p0  d1165f221234  3 months ago  13.3kB
 +
vmguest@vm-ubuntu-0p2:~$ docker push ubuntu-20-04-lts:version-0p0 docker-registry.neelanurseries.com/ubuntu-20-04-lts:version-0p0
 +
"docker push" requires exactly 1 argument.
 +
See 'docker push --help'.
 +
 +
Usage:  docker push [OPTIONS] NAME[:TAG]
 +
 +
Push an image or a repository to a registry
 +
vmguest@vm-ubuntu-0p2:~$ docker push ubuntu-20-04-lts:version-0p0
 +
The push refers to repository [docker.io/library/ubuntu-20-04-lts]
 +
d07b1c131bb8: Preparing
 +
2788d2f5dd8f: Preparing
 +
04d694391e96: Preparing
 +
48ed8163532b: Preparing
 +
denied: requested access to the resource is denied
 +
vmguest@vm-ubuntu-0p2:~$ docker push docker-registry.neelanurseries.com/ubuntu-20-04-lts:version-0p0
 +
The push refers to repository [docker-registry.neelanurseries.com/ubuntu-20-04-lts]
 +
An image does not exist locally with the tag: docker-registry.neelanurseries.com/ubuntu-20-04-lts
 +
vmguest@vm-ubuntu-0p2:~$ docker tag ubuntu-20-04-lts:version-0p0 docker-registry.neelanurseries.com/ubuntu-20-04-lts:version-0p0
 +
vmguest@vm-ubuntu-0p2:~$ docker push docker-registry.neelanurseries.com/ubuntu-20-04-lts:version-0p0
 +
The push refers to repository [docker-registry.neelanurseries.com/ubuntu-20-04-lts]
 +
d07b1c131bb8: Pushed
 +
2788d2f5dd8f: Pushed
 +
04d694391e96: Pushed
 +
48ed8163532b: Pushed
 +
version-0p0: digest: sha256:114bbce1997fa476da56c3958cb3ca13269a54b0a97dfd3667543c7778287bf2 size: 1150
 
</pre>
 
</pre>
  
Line 59: Line 195:
 
Delete a Docker repository on Docker hub:
 
Delete a Docker repository on Docker hub:
  
 +
<ul>
 
*  https://hub.docker.com/support/doc/how-do-i-delete-a-repository
 
*  https://hub.docker.com/support/doc/how-do-i-delete-a-repository
 +
</ul>
 +
 +
<!-- odne komentar -->
  
−
==== [[#top|^]] docker container rm [...], docker image rm [...] ====
+
=== [[#top|^]] docker container rm ===
−
Delete (remove) a local Docker image . . . first must remove all local containers which use given image:
+
 
−
*  https://docs.docker.com/engine/reference/commandline/container_rm/
+
An excellent tutorial on removing Docker images, containers and volumes, with many examples:
−
*  https://docs.docker.com/engine/reference/commandline/image_rm/
+
 
−
Commands which worked locally:
+
<ul>
−
<pre>
+
*  https://www.digitalocean.com/community/tutorials/how-to-remove-docker-images-containers-and-volumes
−
1036  docker ps -a
+
</ul>
−
1037  docker container rm bdc45e3021ad
+
 
−
  1038 docker ps -a
+
Note also the Docker images are removed using `docker rmi`, while Docker containers are removed via `docker rm`. The example command to remove all Docker containers with `exited` status is not quite correct. Needed to filter for the first column of Docker's `ps -a` command in order to provide only Docker containter IDs:
−
1039  docker container rm 3899ce3b4c5f 85206c6c2565
+
 
−
1040  docker ps -a
+
  $ docker rm $(docker ps -a -f status=exited | cut -d " " -f 1)
−
1041  docker container rm e530246d9393 775ef655da0f 51a5967aecf8
+
 
−
1042  docker ps -a
+
<!-- odne komentar -->
−
  .
+
 
−
  .
+
=== [[#top|^]] docker load ===
−
  .
+
 
−
  1047  docker image rm tedhavelkaad0602/ubuntu-04-vim--openssh-server--man-db
+
* https://stackoverflow.com/questions/40582300/how-to-load-a-docker-image-from-a-tar-file
−
1048  docker images
+
 
−
  1049  docker image rm tedhavelkaad0602/ubuntu-nodejs-net-tools
+
Example invocation of `docker load` command:
−
1050  docker images
+
 
−
</pre>
+
  $ docker load < image_name.tar
 +
 
 +
In the above command except 'image_name' is really file basename. The given Docker image may not be named, but rather have only a hash-based identifier.
 +
 
 +
<!-- odne komentar -->
  
 +
== [[#top|^]] SSH connection to Docker container ==
  
 
Secure shell into, or otherwise connect with a running Docker container in and of a Linux environment:
 
Secure shell into, or otherwise connect with a running Docker container in and of a Linux environment:
Line 95: Line 240:
  
 
==== [[#top|^]] docker exec [options] [command_in_container] ====
 
==== [[#top|^]] docker exec [options] [command_in_container] ====
 +
 +
Docker image ID can be obtained by running `docker images` or `docker images -a`.  This can aid in the following Docker command invocation:
 +
 
<pre>
 
<pre>
−
   937  docker exec -t -i a42e2e801d60 bash
+
   $ docker exec -it [image_id] /bin/bash
−
  938  docker exec -i -t 51a5967aecf8 bash
 
 
</pre>
 
</pre>
  
 
Note this can have unintended consequences given that there's no ssh server running to handle multiple, or additional logins!
 
Note this can have unintended consequences given that there's no ssh server running to handle multiple, or additional logins!
 +
 
<!-- comment -->
 
<!-- comment -->
  
Line 109: Line 257:
 
<!-- comment -->
 
<!-- comment -->
  
−
=== [[#top|^]] Docker volumes ===
+
== [[#top|^]] Docker volumes ==
  
 
https://www.digitalocean.com/community/tutorials/how-to-share-data-between-docker-containers
 
https://www.digitalocean.com/community/tutorials/how-to-share-data-between-docker-containers
Line 115: Line 263:
 
<!-- comment -->
 
<!-- comment -->
  
−
=== [[#top|^]] Anatomy of a Dockerfile ===
+
== [[#top|^]] Anatomy of a Dockerfile ==
−
Ok Ted needs to figure out what are Dockerfiles and how should they be authored and maintained.  One example that's annotated:
+
 
 +
<span style="color:red">This section in progress!</span>
 +
 
 +
An annotated example Dockerfile:
 +
 
 
*  https://gist.github.com/adamveld12/4815792fadf119ef41bd
 
*  https://gist.github.com/adamveld12/4815792fadf119ef41bd
 +
 
A link to a Docker based project which comes very close to what we're working on for an embedded development toolchain and environment that's fully tracked, easy to reproduce on demand:
 
A link to a Docker based project which comes very close to what we're working on for an embedded development toolchain and environment that's fully tracked, easy to reproduce on demand:
 
*  https://dev.to/dalimay28/using-docker-for-embedded-systems-development-b16
 
*  https://dev.to/dalimay28/using-docker-for-embedded-systems-development-b16
Line 123: Line 276:
 
<!-- comment -->
 
<!-- comment -->
  
−
=== [[#top|^]] Promising Docker images ===
+
== [[#top|^]] Promising Docker images ==
  
 
*  https://hub.docker.com/r/rastasheep/ubuntu-sshd
 
*  https://hub.docker.com/r/rastasheep/ubuntu-sshd
Line 129: Line 282:
 
<!-- comment -->
 
<!-- comment -->
  
−
=== [[#top|^]] to access physical devices within container ===
+
== [[#top|^]] Physical Device Access in a Docker Container ===
  
 
*  https://forums.unraid.net/topic/47595-how-do-i-pass-through-a-usb-device-to-a-docker-container/
 
*  https://forums.unraid.net/topic/47595-how-do-i-pass-through-a-usb-device-to-a-docker-container/
Line 135: Line 288:
 
<!-- comment -->
 
<!-- comment -->
  
−
=== [[#top|^]] Docker registries to share images ===
+
== [[#top|^]] Private Docker registries ==
 +
 
 +
One of the later references found, but one of the most helpful to putting all the pieces together and to understanding how to test them, and why they're there:
 +
*  https://lathonez.com/2016/docker-registry-apache-letsencrypt/
 +
This post from 2016 by writer Lathonez.
 +
 
 +
 
 
Notes on how to set up a Docker container registry, a networked server from which Docker images can be accessed and shared:
 
Notes on how to set up a Docker container registry, a networked server from which Docker images can be accessed and shared:
 
*  https://docs.docker.com/registry/deploying/
 
*  https://docs.docker.com/registry/deploying/
Line 160: Line 319:
 
<!-- comment -->
 
<!-- comment -->
  
−
==== [[#top|^]] web server configuration ====
+
=== [[#top|^]] web server configuration ===
 +
 
 
A dedicated virtual machine can easily have one web server configured to act as a reverse proxy in front of a Docker registry.  In this context we mean 'proxy' as in a proxy for the public to reach one's internal, in this case Docker registry server.  But for development purposes where our virtual machine options are not yet easily configured to be accessible on a LAN, we face a need to set up a first private registry on an existing cloud host which has a name and fixed IP.  And there is already a web server config in place there, which we want to leave intact while testing the reverse proxy and other Docker registry pieces.  Hence this section in this notes page.
 
A dedicated virtual machine can easily have one web server configured to act as a reverse proxy in front of a Docker registry.  In this context we mean 'proxy' as in a proxy for the public to reach one's internal, in this case Docker registry server.  But for development purposes where our virtual machine options are not yet easily configured to be accessible on a LAN, we face a need to set up a first private registry on an existing cloud host which has a name and fixed IP.  And there is already a web server config in place there, which we want to leave intact while testing the reverse proxy and other Docker registry pieces.  Hence this section in this notes page.
 
<!-- comment -->
 
<!-- comment -->
  
−
===== [[#top|^]] apache2 plus nginx =====
+
==== [[#top|^]] apache2 plus nginx ====
 +
 
 
This section contains references and notes regarding how to set up apache2 and nginx on one server.
 
This section contains references and notes regarding how to set up apache2 and nginx on one server.
 
*  https://stackoverflow.com/questions/23024473/how-can-i-run-both-nginx-and-apache-together-on-ubuntu
 
*  https://stackoverflow.com/questions/23024473/how-can-i-run-both-nginx-and-apache-together-on-ubuntu
 
<!-- comment -->
 
<!-- comment -->
  
−
===== [[#top|^]] to run multiple instances of apache2 =====
+
==== [[#top|^]] to run multiple instances of apache2 ====
 +
 
 +
At startpage.com entered search phrase "possible to run apache2 twice with distinct configurations".  First result helpful:
 
*  https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=115522518
 
*  https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=115522518
  
−
=== - - - - - &nbsp;- - - - -  - - - - -  - - - - -  - - - - -  - - - - -  - - - - -  - - - - -  - - - - - ===
+
=== [[#top|^]] Reverse proxy choices ===
 +
 
 +
Excerpt from https://help.sonatype.com/repomanager3/system-configuration/configuring-ssl#ConfiguringSSL-InboundSSL-ConfiguringtoServeContentviaHTTPS:
 +
<pre>
 +
Inbound SSL - Configuring to Serve Content via HTTPS
 +
Available in Nexus Repository OSS and Nexus Repository Pro
 +
 
 +
Providing access to the user interface and content via HTTPS is a best practice.
 +
 
 +
You have two options:
 +
 
 +
Use a separate reverse proxy server in front of the repository manager to manage HTTPS
 +
Configure the repository manager itself to serve HTTPS directly
 +
Using A Reverse Proxy Server
 +
A common approach is to access the repository manager through a dedicated server which answers HTTPS requests on behalf of the repository manager - these servers are called reverse proxies or SSL/TLS terminators. Subsequently requests are forwarded to the repository manager via HTTP and responses received via HTTP are then sent back to the requestor via HTTPS.
  
 +
There are a few advantages to using these which can be discussed with your networking team. For example, the repository manager can be upgraded/installed without the need to work with a custom JVM keystore. The reverse proxy could already be in place for other systems in your network. Common reverse proxy choices are Apache httpd, nginx, Eclipse Jetty or even dedicated hardware appliances. All of them can be configured to serve SSL content, and there is a large amount of reference material available online.
  
 +
Serving SSL Directly
 +
The second approach is to use the Eclipse Jetty instance that is distributed with the repository manager to accept HTTPS connections.
 +
 +
How to Enable...
 +
</pre>
 +
 +
=== [[#top|^]] local private registry with only apache2-utils ===
 +
 +
Following article good, well written!  Describes in clear detail and incrementally how to set up private Docker registry.  Gives references.  Explains need for some kind of web service provision to achieve a remotely accessible private Docker registry.  Key words<b> Gabriel Tanner private Docker registry </b>:
 +
*  https://gabrieltanner.org/blog/docker-registry
 +
 +
== [[#top|^]] Docker Features Not Done ==
 +
 +
Apparently there is no ready facility to list the images in a private Docker registry, one that's running from the official <code>registry:2</code> Docker image.  A forum post includes mentions of two possible home brew scripts to provide such a listing:
 +
*  https://forums.docker.com/t/docker-private-registry-how-to-list-all-images/21136/7
 +
The second of the two scripts is hosted at Github.  Need to check the license applied to it, looks like there are recent updates to this project, from about 2021 March:
 +
*  https://github.com/BradleyA/Search-docker-registry-v2-script.1.0
 +
 +
<!-- comment -->
 +
 +
== [[#top|^]] Outline of Docker Team How-To ==
 +
 +
Sections of document to share with work team:
 +
<ol>
 +
<li> End User Use of Docker</li>
 +
<li> How To Install Docker Client 'docker-ce'</li>
 +
<li> Basic Interactions with Docker Images</li>
 +
<ul>
 +
<li> login
 +
<li> pull
 +
<li> run
 +
<li> stop
 +
<li> modify
 +
<li> commit
 +
<li> tag
 +
<li> push
 +
<li> logoff
 +
</ul>
 +
<li> Private Docker Registry Configuration</li>
 +
</ol>
 +
 +
<!-- comment -->
 +
 +
== [[#top|^]] Docker Compose ==
 +
Docker compose files can coordinate and run multiple Docker containers which work together to provide a larger service or set of services.  A starting point for understanding how to utilize Docker compose files, which are drafted in a mark-up language named YAML, can be found here:
 +
*  https://docs.docker.com/compose/compose-file/
 +
 +
In contrast a dockerfile is a set of build instructions, which could be run at a command line to manually create a Docker image.  A dockerfile provides a way to automate the image building process, and is much lighter weight in terms of disk space taken compared with a typical image.  A one line command in a dockerfile could pull in a multi-megabyte library or binary.  Some information on dockerfiles here:
 +
*  https://docs.docker.com/engine/reference/builder/
 +
 +
== [[#top|^]] docker scan image ==
 +
 +
Starting point to understand Docker vulnerability scanning:
 +
*  https://docs.docker.com/engine/scan/
 +
An example vulnerability found in certain versions of Perl which ship with Debian:
 +
*  https://snyk.io/vuln/SNYK-DEBIAN10-LIBIDN2-474100
 +
 +
<!-- comment -->
 +
 +
== - - - - - &nbsp; - - - - - &nbsp; - - - - - &nbsp; - - - - - &nbsp; - - - - - &nbsp; - - - - - &nbsp; - - - - - &nbsp; - - - - - &nbsp; - - - - - ==
 +
 +
&nbsp;
 
<!-- comment -->
 
<!-- comment -->
  
Line 184: Line 424:
 
</ul>
 
</ul>
  
−
=== - - - - - - - - - - - - - - -   - - - - - - - - - - - - - - -   - - - - - - - - - - - - - - - ===
+
== - - - - - &nbsp; - - - - - &nbsp; - - - - - &nbsp; - - - - - &nbsp; - - - - - &nbsp; - - - - - &nbsp; - - - - - &nbsp; - - - - - &nbsp; - - - - - ==
  
 +
&nbsp;
 
<!-- comment -->
 
<!-- comment -->
  
Line 241: Line 482:
  
 
=== [[#top|^]] ssh to VM on remote Windows station ===
 
=== [[#top|^]] ssh to VM on remote Windows station ===
−
 
+
The following article gives simple instructions to set up a virtual machine to accept ssh connections from the host operating system. This normally means that when logging in we are ssh'ing <given_vm_user>@localhost, on a particular port other than port 22, to which a feature of the VM listens for ssh traffic.
 
*  https://dev.to/awwsmm/setting-up-an-ubuntu-vm-on-windows-server-2g23
 
*  https://dev.to/awwsmm/setting-up-an-ubuntu-vm-on-windows-server-2g23
−
<!-- comment -->
 
  
−
=== - - - - -  - - - - -  - - - - -  - - - - -  - - - - -  - - - - -  - - - - -  - - - - -  - - - - - ===
+
=== [[#top|^]] TinyCore Linux on Virtual Machine ===
  
 +
*  https://help.hcltechsw.com/bigfix/9.5/platform/Platform/Installation/c_install_tiny_core_linux_7_0_on.html
  
 +
= - - - - - &nbsp; - - - - - &nbsp; - - - - - &nbsp; - - - - - &nbsp; - - - - - &nbsp; - - - - - &nbsp; - - - - - &nbsp; - - - - - &nbsp; - - - - - =
  
 +
&nbsp;
  
 
<!-- comment -->
 
<!-- comment -->
Line 290: Line 533:
 
</pre>
 
</pre>
  
−
=== Git client config in VM and Docker container ===
+
=== [[#top|^]] Git client config in VM and Docker container ===
  
 
For firmware dev work whether in virtual machines and or Docker containers, we need access to remote git repositories.  SSL keys are necessary for this access.  Here are some beginning notes to explain setting up such keys within each environment:
 
For firmware dev work whether in virtual machines and or Docker containers, we need access to remote git repositories.  SSL keys are necessary for this access.  Here are some beginning notes to explain setting up such keys within each environment:
Line 319: Line 562:
 
</pre>
 
</pre>
  
 +
=== [[#top|^]] Docker set up tangential topics, references to sort ===
 
On WSL...
 
On WSL...
  
 
*  http://armofthings.com/2020/04/15/wsl-for-embedded-system-development/
 
*  http://armofthings.com/2020/04/15/wsl-for-embedded-system-development/
  
 +
*  https://help.dreamhost.com/hc/en-us/articles/215747748-How-can-I-redirect-and-rewrite-my-URLs-with-an-htaccess-file-
 +
 +
Docker <code>rmi</code> command to remove one or more but not last Docker tag associated with a given image:
 +
*  https://serverfault.com/questions/703344/how-to-remove-an-image-tag-in-docker-without-removing-the-image-itself
  
−
*  https://help.dreamhost.com/hc/en-us/articles/215747748-How-can-I-redirect-and-rewrite-my-URLs-with-an-htaccess-file-
+
=== [[#top|^]] Private docker registration configuration topics ===
  
 
References related to first steps to bring up nginx web server:
 
References related to first steps to bring up nginx web server:
Line 330: Line 578:
 
*  https://www.nginx.com/resources/wiki/start/topics/examples/full/
 
*  https://www.nginx.com/resources/wiki/start/topics/examples/full/
  
 +
To bring up nginx is actually a part of effort to configure and bring up private Docker registry.  Here is a CA related article which helps explain the TLS certificate part of this work:
 +
*  https://www.digicert.com/difference-between-dv-ov-and-ev-ssl-certificates
 +
 +
*  https://serverfault.com/questions/391457/how-does-apache-merge-multiple-matching-location-sections
 +
 +
*  https://docs.docker.com/network/links/
 +
 +
2021-06-25 Friday...
 +
Docker push syntax explained further than on Docker main site documentation pages:
 +
*  https://www.educba.com/docker-push/
 +
 +
Unable to stop a running container, one which appears to have started running automatically in Ubuntu VM host.  Searched startpage.com with 'runc did not terminate sucessfully:', found discussion of AppArmor aa-remove-unknown command:
 +
*  https://javahowtos.com/guides/124-docker/414-solved-cannot-kill-docker-container-permission-denied.html
 
<!-- comment -->
 
<!-- comment -->
 +
 +
<center>
 +
- - - [[#top|top of page]] - - -
 +
</center>

Latest revision as of 18:03, 10 July 2023

Containers :: RTOS Notes :: Clusters :: Unix and Linux config

Contents

^ Container Software

A promising starting point, collection of six articles / tutorials on Docker containers: 

 *  https://medium.com/sysf/docker/home

A general note following about three days' Docker container experimentation, and to question "should I be using a Docker container interactively, running multiple apps, some simultaneously?". (Note the following forum post chain is a quagmire of issues and opinions!): 

 *  https://news.ycombinator.com/item?id=7950326

^ Docker

Notes on Docker containerizing software. Note, to get a practical start it is a good and or needed step to create a user account with Docker dot com. Single user and free accounts with some advanced Docker features disabled are available. A good starting tutorial for beginners is written by one Brian Hogan of Digital Ocean, this article published 2018 July 5. First reference in list here:

Ted noting too there may be a daily limit to the number of docker images which a user with a free account can push to Docker's image repository. Docker's image repository provides URLs of the form https://hub.docker.com/u/<user_name>, where username is the given person's Docker account user name.4

Returning to Docker a year to two further (2023-06-26) here is an article linked directly from Brian Hogan's How To Install... article at Digital Ocean. This article talks about creating multiple Docker containers, many of which contain each other in a layered fashion by design:

This article introduces some basic concepts regarding the Docker ecosystem. And a good article here to lay out the distinctions between a Docker image and a Docker container:

> Docker registries and repositories:

Nick Janetakis writes a pretty concise brief article to describe what are Docker repositories and Docker registries. An image repository holds versions (committed and tagged versions) of a given Docker image. A registry holds zero or more Docker repositories, and is generally a full-fledged always live service that's accessible on the internet or on an intranet.


^ Dockerfiles

An article and tutorial on crafting a basic Dockerfile:

Ubuntu Docker images are the most downloaded on hub.docker.com. Here is an article on how to customize this image:

How to start Docker image with interactive shell:

How to save a Docker image as a compressed file:


^ Docker commands

NOTE: as of 2021-07-19 Monday this section needs organizing and annotation work. - TMH
This first reference URL was added when Ted did not know how to start a Docker image running and grant it access to a host OS directory hierarchy:

    More on Docker volumes at this next URL. Note also that there is an older Docker volume mechanism and a newer more flexible one, but that may be detailed in another tutorial or guide:


^ Docker pull

Example command to pull a Docker image from a private registry: 

docker pull registry.some_remote_host.com/ubuntu-20-04:version-1p0

When pulling Docker images from Docker's primary community site, hub.docker.com, only the Docker username is needed prior to the image name: 

docker pull tedhavelkaad0602/ubuntu-git-plus-arm-none-eabi-gcc:version-0p8


^ Docker run

Very limited but useful example of how to run a docker image, with two host OS directory hierarchies accessible in the Docker container: 

# docker run -i -t -v /home/cpguest/projects/chibios:/home/huesped/project/chibios -v /mnt/host_os_downloads:/mnt/host_os_downloads --privileged bf980d48d9ab


^ Docker commit

To commit changes within a container a user can invoke `docker commit ...`. While it is good to know about Docker commit command, commits of container changes are not trackable and maintainable nearly as practically as Dockerfiles. Here is the official documentation for this commit command, followed by a tutorial on Docker commit command with preliminary steps of writing a Dockerfile:

Figure: example Docker commit: 

docker commit -m "Created 'use marker' file in root homedir of ubuntu-20-04-lts:version-1p0 image." -a "Sr Fulano" ad494bcfe010 private_registry.some_host.com/ubuntu-20-04:version-1p0



^ Docker push

Incomplete synopsis of `docker push` command: 

  $ docker push [<optional_remote_registry>/]<image_name>[:<optional_tag>]

Series of docker push invocations for a work-in-progress image. Note the tag names are optional until one needs push an image that's being amended, such that it is a newer or different version of an existing image on the remote Docker repository: 

  886  docker push tedhavelkaad0602/ubuntu-git-plus-arm-none-eabi-gcc
  932  docker push tedhavelkaad0602/ubuntu-git-plus-arm-none-eabi-gcc:version-0p2
  944  docker push tedhavelkaad0602/ubuntu-git-plus-arm-none-eabi-gcc:version-0p3
  962  docker push tedhavelkaad0602/ubuntu-git-plus-arm-none-eabi-gcc:version-0p4
  965  docker push tedhavelkaad0602/ubuntu-git-plus-arm-none-eabi-gcc:version-0p5

And here is a link to an article describing Docker push command syntax:

We'll note this link also in the References section of this page. Takeaway here is that the third token in the above push command examples, the latter four examples, that token has the form "<docker_user_name>/<image_name>:<docker_tag>".

This article helped get us closer to understanding how Docker CLI parses the argument to a given docker push argument. Even with this understanding some further trial and error helped clarify what is needed in the form of this argument, which identifies a Docker image to be pushed to a private, non-docker.io registry. Some excerpts from the command line: 

vmguest@vm-ubuntu-0p2:~$ docker images
REPOSITORY                                           TAG           IMAGE ID       CREATED        SIZE
ubuntu-20-04-lts                                     version-0p0   065cf14a189c   8 days ago     135MB
localhost:5000/my-ubuntu                             latest        065cf14a189c   8 days ago     135MB
tedhavelkaad0602/ubuntu-git-plus-arm-none-eabi-gcc   version-0p8   bf980d48d9ab   8 days ago     2.98GB
tedhavelkaad0602/ubuntu-git-plus-arm-none-eabi-gcc   version-0p2   4110ca98063a   2 weeks ago    309MB
tedhavelkaad0602/ubuntu-git-plus-arm-none-eabi-gcc   latest        f24e27e3d998   2 weeks ago    245MB
registry                                             2             1fd8e1b0bb7e   2 months ago   26.2MB
docker-registry.neelanurseries.com/hello-world       version-0p0   d1165f221234   3 months ago   13.3kB
vmguest@vm-ubuntu-0p2:~$ docker push ubuntu-20-04-lts:version-0p0 docker-registry.neelanurseries.com/ubuntu-20-04-lts:version-0p0
"docker push" requires exactly 1 argument.
See 'docker push --help'.

Usage:  docker push [OPTIONS] NAME[:TAG]

Push an image or a repository to a registry
vmguest@vm-ubuntu-0p2:~$ docker push ubuntu-20-04-lts:version-0p0
The push refers to repository [docker.io/library/ubuntu-20-04-lts]
d07b1c131bb8: Preparing 
2788d2f5dd8f: Preparing 
04d694391e96: Preparing 
48ed8163532b: Preparing 
denied: requested access to the resource is denied
vmguest@vm-ubuntu-0p2:~$ docker push docker-registry.neelanurseries.com/ubuntu-20-04-lts:version-0p0
The push refers to repository [docker-registry.neelanurseries.com/ubuntu-20-04-lts]
An image does not exist locally with the tag: docker-registry.neelanurseries.com/ubuntu-20-04-lts
vmguest@vm-ubuntu-0p2:~$ docker tag ubuntu-20-04-lts:version-0p0 docker-registry.neelanurseries.com/ubuntu-20-04-lts:version-0p0
vmguest@vm-ubuntu-0p2:~$ docker push docker-registry.neelanurseries.com/ubuntu-20-04-lts:version-0p0
The push refers to repository [docker-registry.neelanurseries.com/ubuntu-20-04-lts]
d07b1c131bb8: Pushed 
2788d2f5dd8f: Pushed 
04d694391e96: Pushed 
48ed8163532b: Pushed 
version-0p0: digest: sha256:114bbce1997fa476da56c3958cb3ca13269a54b0a97dfd3667543c7778287bf2 size: 1150


Delete a Docker repository on Docker hub:


^ docker container rm

An excellent tutorial on removing Docker images, containers and volumes, with many examples:

Note also the Docker images are removed using `docker rmi`, while Docker containers are removed via `docker rm`. The example command to remove all Docker containers with `exited` status is not quite correct. Needed to filter for the first column of Docker's `ps -a` command in order to provide only Docker containter IDs: 

 $ docker rm $(docker ps -a -f status=exited | cut -d " " -f 1)


^ docker load

Example invocation of `docker load` command: 

  $ docker load < image_name.tar

In the above command except 'image_name' is really file basename. The given Docker image may not be named, but rather have only a hash-based identifier.


^ SSH connection to Docker container

Secure shell into, or otherwise connect with a running Docker container in and of a Linux environment:

A couple examples of running second bash instance of a given Docker container, not a login, but provides second interactive window or interface to a given Docker container:

^ docker exec [options] [command_in_container]

Docker image ID can be obtained by running `docker images` or `docker images -a`. This can aid in the following Docker command invocation: 

  $ docker exec -it [image_id] /bin/bash

Note this can have unintended consequences given that there's no ssh server running to handle multiple, or additional logins!


-- Enable USB access in Docker container --


^ Docker volumes

https://www.digitalocean.com/community/tutorials/how-to-share-data-between-docker-containers


^ Anatomy of a Dockerfile

This section in progress!

An annotated example Dockerfile:

A link to a Docker based project which comes very close to what we're working on for an embedded development toolchain and environment that's fully tracked, easy to reproduce on demand:


^ Promising Docker images


^ Physical Device Access in a Docker Container =


^ Private Docker registries

One of the later references found, but one of the most helpful to putting all the pieces together and to understanding how to test them, and why they're there:

This post from 2016 by writer Lathonez.


Notes on how to set up a Docker container registry, a networked server from which Docker images can be accessed and shared:

Setting up authentication using htaccess files:

Some requirements for the above link to Docker registry recipe:

 
Above tutorials and articles don't paint a clear easy path to setting up a first, basic Docker registry. Taking up a different path here at DigitalOcean:

Regarding port forwarding in apache2 config files:

There is some nginx configuration stuff that's not obvious how to port to apache2 config files, but following how-to article gives a clue with "Header set Host ..." and "RequestHeader set X-Forwarded-Proto "https"":


^ web server configuration

A dedicated virtual machine can easily have one web server configured to act as a reverse proxy in front of a Docker registry. In this context we mean 'proxy' as in a proxy for the public to reach one's internal, in this case Docker registry server. But for development purposes where our virtual machine options are not yet easily configured to be accessible on a LAN, we face a need to set up a first private registry on an existing cloud host which has a name and fixed IP. And there is already a web server config in place there, which we want to leave intact while testing the reverse proxy and other Docker registry pieces. Hence this section in this notes page.

^ apache2 plus nginx

This section contains references and notes regarding how to set up apache2 and nginx on one server.

^ to run multiple instances of apache2

At startpage.com entered search phrase "possible to run apache2 twice with distinct configurations". First result helpful:

^ Reverse proxy choices

Excerpt from https://help.sonatype.com/repomanager3/system-configuration/configuring-ssl#ConfiguringSSL-InboundSSL-ConfiguringtoServeContentviaHTTPS: 

Inbound SSL - Configuring to Serve Content via HTTPS
Available in Nexus Repository OSS and Nexus Repository Pro

Providing access to the user interface and content via HTTPS is a best practice.

You have two options:

Use a separate reverse proxy server in front of the repository manager to manage HTTPS
Configure the repository manager itself to serve HTTPS directly
Using A Reverse Proxy Server
A common approach is to access the repository manager through a dedicated server which answers HTTPS requests on behalf of the repository manager - these servers are called reverse proxies or SSL/TLS terminators. Subsequently requests are forwarded to the repository manager via HTTP and responses received via HTTP are then sent back to the requestor via HTTPS.

There are a few advantages to using these which can be discussed with your networking team. For example, the repository manager can be upgraded/installed without the need to work with a custom JVM keystore. The reverse proxy could already be in place for other systems in your network. Common reverse proxy choices are Apache httpd, nginx, Eclipse Jetty or even dedicated hardware appliances. All of them can be configured to serve SSL content, and there is a large amount of reference material available online.

Serving SSL Directly 
The second approach is to use the Eclipse Jetty instance that is distributed with the repository manager to accept HTTPS connections.

How to Enable...

^ local private registry with only apache2-utils

Following article good, well written! Describes in clear detail and incrementally how to set up private Docker registry. Gives references. Explains need for some kind of web service provision to achieve a remotely accessible private Docker registry. Key words Gabriel Tanner private Docker registry :

^ Docker Features Not Done

Apparently there is no ready facility to list the images in a private Docker registry, one that's running from the official registry:2 Docker image. A forum post includes mentions of two possible home brew scripts to provide such a listing:

The second of the two scripts is hosted at Github. Need to check the license applied to it, looks like there are recent updates to this project, from about 2021 March:


^ Outline of Docker Team How-To

Sections of document to share with work team:

  1. End User Use of Docker
  2. How To Install Docker Client 'docker-ce'
  3. Basic Interactions with Docker Images
    • login
    • pull
    • run
    • stop
    • modify
    • commit
    • tag
    • push
    • logoff
  4. Private Docker Registry Configuration


^ Docker Compose

Docker compose files can coordinate and run multiple Docker containers which work together to provide a larger service or set of services. A starting point for understanding how to utilize Docker compose files, which are drafted in a mark-up language named YAML, can be found here:

In contrast a dockerfile is a set of build instructions, which could be run at a command line to manually create a Docker image. A dockerfile provides a way to automate the image building process, and is much lighter weight in terms of disk space taken compared with a typical image. A one line command in a dockerfile could pull in a multi-megabyte library or binary. Some information on dockerfiles here:

^ docker scan image

Starting point to understand Docker vulnerability scanning:

An example vulnerability found in certain versions of Perl which ship with Debian:


- - - - -   - - - - -   - - - - -   - - - - -   - - - - -   - - - - -   - - - - -   - - - - -   - - - - -

 

^ Kubernetes Container Software

Kubernetes notes 2021-06-09 This section a stub section.

- - - - -   - - - - -   - - - - -   - - - - -   - - - - -   - - - - -   - - - - -   - - - - -   - - - - -

 

^ Oracle Corporation VirtualBox software

VirtualBox on-line manual

Retrieved from "https://wiki.neelanurseries.com/index.php?title=Containers_and_virtual_machines&oldid=7399"