Docker Workshop

By Finn Kempers

Done by Tim Fletcher, DevOps Freelancer working at Cubits in Berlin. @timjdfletcher

Slack: https://slack.flossuk.org

Schedule of day
  • “Containers, Docker, how it works”
  • “Exploring simple docker containers”
  • Lunch
  • “Deploying simple Rancher Cluster”
  • “Commissioning a service on Rancher”
Slides and files

https://drive.google.com/drive/folders/0Bz5TtpiW4_X1bWR4M0NHdHhfRzQ

Docker Introduction

It is is a isolated environment to work in. A lot of history with Unix and FreeBSD with containers in jail format and the like. However Docker is unique due to its strong customisation of restrictions. Google itself is infamous for nigh relying on containers due to its ease of use and security. Docker itself is beyond a container, it made an ecosystem that revolved around it too.

Virtual Machines are only similar in concept to containers. They are essentially a computer inside a computer but does have the problem that things can break out the hypervisor. Generally however very secure and close to full hardware “bare metal” performance. A very well established technology and simple to comprehend.

Containers are much more popular in cloud environments. They usually run inside virtual machines. The difference is that you are isolating processes, not a complete hardware setup. Containers are however not possible to “move” live, you kill it and then start it elsewhere. They should be seen as a running process within a restricted environment, hence a container. The security however is questionable, it depends simply on how well it is managed and can require time to customise to fit your needs. Docker containers only use process cache and not the kernel or such to worry about, so it is a lot more efficient. However generally Kubernetes or the like is used for putting containers out efficiently. Either way the big difference to VM’s to consider is that they share the same OS and same appropriate bins/libraries depending on your customisation. Slide 5 on “An Introduction to Docker” has a diagram of this.

Docker as a platform breaks down into several parts, see Slide 6 of the above mentioned. This will be broken down below.

The Docker Engine has been through a number of changes and has been standardised, now running with RunC. In itself it hides a lot of the more complicated stuff but can be user customised, but essentially it is what runs the guts of the Docker management.

Docker images are essentially tarballs, very akin to Vagrant files if one is familiar with them. THe images are layered and thus will “commit” the change, so that if an image wants Apache added, you install Apache and you add a layer with that in the image to the container. They are all cached on file system and are immutable file systems. They only contain changes, which add on to the base image of the container, which is usually the OS such as Debian. The layers are essentially like git in concept.

As such, also much like git, these layers are often publicly available in repositories such as Docker Hu and Quay which are similar to Github. Companies such as DIgitalOcean provide private container registries. All of this allows for branching, deduplication and add security reviewing to container images, so you can check they are up to date, since they still need to be kept up to date.

In this presentation, an example docker image is pulled made by Tim and it is noted that the image is downloaded in layers, for which it will build upon. The layers themselves appear to be identified with sha256 keys. “docker image inspect [docker image file url]” will present various information on the docker image. “docker image history” on the other hand will present the history of the layers. Due to this, secrecy of past errors is very difficult to mask. This does however also mean that anyone can assess the deployment of a container.

The Dockerfile is basically a bash script which builds the container image layer by layer and thus will also show where a layer may error. Not required for docker container building, but it’s the usual method.

The example dockerfile presented has an apt-get update and then a string of installs required, then doing some basic settings such as chmod, then explicitly defines the user to nobody to avoid being root. Entrypoints allows for changing how you run commands and presents for greater flexibility to allow making different containers for different environments. “docker run -it timjdfletcher/example-docker-container /bin/bash” let go into the docker container files itself and then looked at the example entry point with:

#!/bin/bash -e

# This if matchs the following:

# if $1 (argv[0]) is unset (ie it's null)

# or anything begining with a - (etc -hello or -test)

if [[ -z $1 ]] || [[ ${1:0:1} == '-' ]] ; then

  exec  /app/hello "$@"

else

# If we have been called with a variable that doesn’t start with a – then assume it’s a progrom and run it

# This is handy for container debugging

  exec "$@

Note that the entrypoint is useful to make sure bash is installed if running certain commands.

While building, a docker build will have layers cached of those already built and then will use the cache to build layers previously done to make builds a lot faster. This shouldn’t be done for deployment though as cache assumptions aren’t always perfect. This allows for building containers of older tools such as ssh or java and running isolated old versions of software within the container on a modern OS.

Using the dockerfile provided at https://github.com/TimJDFletcher/docker-workshop/blob/master/example-docker-container/Dockerfile, from the example container a build was made, the build of which illustrates the process the container goes through when building.

Docker allows for environment variables which compose of shell variables, basic config files and can attach network devices and tty to it. In an exercise we run busybox, a tiny tool very common in Android root management and we can run inside it with “docker run -it busybox”. With all these tools, it is possible to customise the environment of the container, defining networking, user account management and the like. It does however mean people can see these variables so not ideally secure.

Docker networking uses network namespaces that are virtualised inside the containers. All ports by default are blocked, so containers via the dockerfile will have certain ports open, then using NAT or proxy to get networking from host to containers, like the VSwitches on VMWare. Orchestration tooling solves a lot of the problem that is setting up networking on docker containers. Docker containers by default can talk to each other on the same network namespace but not to the outside world. “docker run -p 80:80 nginx” will however publish the port and make it accessible. This runs docker-proxy, which manages the traffic, however there are “better” ways of doing it and should mostly be used for quick fix service running.

Docker compose is a Docker tool that takes a prewritten YAML file and sets up whatever is defined, so in the example stack used there is a database, web server and a load balancer. While this allows us to with the example start up a web server very quickly, the data is all in a container, including the database. If the container is deleted, so is the database. There is however a way around this. Without this workaround with an in memory database, nodes are set up and duplicate each other to ensure there is always several nodes up with the database in a three node cluster.

Docker volume is essentially are a defined directory in the host filesystem, which are mounted to the container. NFS and the like also exist for solutions to allow for permanent storage for containers and file management. In the exercise we create a container with volume named html-volume, which we then run. See slide 21 for more details. It essentially demonstrates that we can edit a file on the filesystem and the container will at the same time have access to that file and using it live. Using “docker exec” one can access and be “inside” the running container. These docker volumes can be defined in the dockerfile, with a definition for example of the volume path in “dbdata” and then defining dbdata in the mysql part of the dockerfile. This allowed us to spin up a wordpress site, configure it, use “docker-compose down” to kill it, then spin it back up and find the site intact as all the mysql data is intact on the filesystem.

Docker Container Orchestration

Rancher is a UI tool for managing Rancher, which runs on a single container on the “master host”. It is simple but has a lot of power and allows for scaling and the like. The master Rancher server manags the container deployment, along with a metadata service which includes all the various details and DNS.

This afternoon is an exercise working with Rancher. The chosen master host, of which I am the victim of for my row of seats, run the master Rancher host. Various people then join as a host to the server with a given URL, which then run their hosts with the stacks. We then installed the global service Janitor which cleans up “dead” containers, running as a service across all hosts, hence “global”. There is control of the hosts and the services running from Rancher, however the access to that is available to everyone connected. However, you can set up Rancher environments, which allow for management of users and user access control and removes a lot of duplication for variables, such as running a given set of services.

Joining a Rancher master is incredibly easy as noted before by simply using a supported Docker on Linux. It also via docker-machine supports directly starting systems, allowing for very quick easy scaling to start up systems, such as needing 5 machines.

Rancher Networking is IPSec therefore allowing for all the containers in the Rancher environment to be able to talk to each other directly, which allows for large scale batch processes and various other networking processes, while offering security too. There are various security measures that can be implemented which since it is IPSec, is easy for sysadmins to generally implement.

Stacks must have a unique name, but allow for descriptions and the import of a docker-compose or rancher-compose.yml. With these stacks created, you can then create services which are scalable to a number of containers to run an instance on every host, along with the image to run and a various number of customisation options. With this, you have options such as in our exercise, pinging between hosts, or the stack as a whole, as the processes are tied together to the stack.

The Rancher Catalog itself from which the Janitor service was installed from offes a range of official Rancher services along with many made by the community (to an extent verified by Rancher), with many options for things such as GoCD stack, Minecraft server, Gogs git service and many others.

Containers that run alongside other containers are known as psychic containers. Those are however a more advanced topic that are not covered in this tutorial.

Using API keys we then set up our own wordpress stacks within Rancher, using access and secret keys created through Rancher. From there we modified the rancher-compose to be similar to the previous docker-compose, but with unique wordpress directory and db names, to ensure it didn’t clash with other stacks.

Hot Based Proxies known as Load Balancers are used in the Rancher world for getting traffic from the outside world into a container, known as Ingress with Kubernetes. The packets are delivered to the edge of the container which is then used. Allows for vhosts, http routing and the like.

Using a Load Balancer, we have made the containers accessible to the outside world with the URL of the machine (though firewalled to within the university). This meant we could using the Load Balancer “serve” the wordpress service that was running on each machine within the master host. When traffic hits the DNS of say, Docker-7, it routes that traffic to the wordpress service of Docker-7.

Using the Rancher UI, you can scale up and duplicate processes, increasing the number of containers, while theoretically still working well. Duplicating processes such as Apache does lead to memory constraint issues based on the resources available, but theoretically otherwise it will simply scale without error.

Digital forensics workshop with Simon Biles

by Gwenyth Rooijackers

A group of 8 persons gathered on Wednesday the 25th of April to start off the conference with Simon Biles’ workshop on digital forensics. As life moves more and more into the digital world, so does crime. Simon Biles gave us a short introduction to a key part in this rather new area of prosecution.

The day started with a look at the life in digital forensics. Along with a couple of insights into the emotional aspects of being an expert witness in digital forensics, we looked at the handling of evidence and guidelines by the Association of Police Officers.

In the scenario set up for the workshop, we were acting for the prosecution. The case regarded a group of people accused of stealing cars, creating new VINs, copying keys etc. After a raid on the garage only a 1 GB USB-drive was recovered. We are here to examine this USB-drive.

Before even having lunch, Biles guides us through imaging the content of the drive without making any changes to the drive itself (i.e. the evidence), mounting the image to access stored information and a browse through the partitions of the drive. We also had a look at the deleted files (including the file evil_plans.txt revealing the plans of the criminals 🙂 ).

In the afternoon we were introduced to Autopsy. This programme allows us to get a quick, automated overview of the examinations we did manually before noon.

To round up the workshop, there was a nice discussion with questions to the speaker. Thank you for today, Simon!

Want to learn more about digital forensics? Biles recommended Forensic Computing: A Practitioner’s Guide by Anthony Sammes and Brian Jenkinson for an introduction to digital forensics or File System Forensic Analysis by Brian Carrier for an introduction to file systems.

Just some stats

This year we have tried to make better use of the FLOSSUK YouTube channel. The first stage of that was to ensure that conference videos appear as quickly as possible online (we would love to stream but there are complications in doing that which we still have yet to overcome).

We also wanted to make sure we check the statistics and to try and engage  people with more notification of the videos and unique events at the conference. This was greatly helped when we unveiled the KDE Slimbook at our Spring Conference (YouTube playlist) which garnered a number of views. However all the videos this year have received a bump in their popularity. Let me show you some of the stats before I give a brief analysis.

I have placed them next to the 2016 figures. Note that we are only just under half way though 2017, though if the video viewing habits follow previous years our greatest percentage of views is in the first 4 months.

2017: Watch Time (so far)

 

2016: Watch Time

2017: Average View (so far)

2016: Average View

2017: Total Views (so far)

2016: Total Views

2017: Likes, Dislikes and Comments (so far)

2016: Likes, Dislikes and Comments

2017: Shares, Playlists, Subscribers (so far)

2016: Shares, Playlists, Subscribers

So what are the numbers telling us?

Well we have generally had a positive time. As a warning we have to note that some of the numbers are skewed by the unveiling we had of the KDE Slimbook. This gave us some disproportionate effects, not all positive, as the following screen grab shows.

So the unveiling takes the majority of our stats (about 58%) which we must take into account. However it was a conscious decision to use this form of promotion and therefore can be seen as part of the overall plan as well as serendipity.

A quick run of what we can see of 5 months versus the previous year:

  • We have 1,299 hours (77,999 minutes) more watch time, which is 520 hours more without the Slimbook;
  • The watch time is lower this is due to the Slimbook being shorter than the average videos;
  • Total views is 26,000 more, or 3,000 more sans Slimbook;
  • Likes is 188 vs 11 or 64 vs 11 (Slimbook);
  • Dislikes, well we had none of these but the Slimbook has granted us more negativity by exposure;
  • We also have positive engagement with 22 comments, 3 not on the Slimbook;
  • We have x9 as many shares and a number of these had nothing to do with the Slimbook;
  • There are 100 extra videos in playlists. Bit of a confusing stat as we didn’t add 100 videos, it is more that we also organised the channel better to have videos in playlists and this has been beneficial;
  • We have 80% more Subscribers, which is great.

So the addition of the very popular video has been hugely beneficial but the numbers also show that the positive promotion, organisation and engagement has shown a good return.

We are always interested to hear feedback, suggestions or just your general thoughts, so please drop us a comment.

Amendment 1

I rather foolishly forgot to thank some people when I first did this article that I would like to correct.

  • Tom Bloor did an excellent job of videoing, editing and rendering all of the video files all in his free time. Thanks Tom, great work and really appreciated.
  • The speakers have always put a lot of effort into presenting talks and contributing to the community, thanks to all the speakers who we are able to meet, listen to and present to others. You rock.

Pi Workshop Review

by Donald Grigor

I arrived early for this one but unfortunately the speaker withdrew for the tutorial, major blow! Les Pounder sounded a really interesting and entertaining speaker, hopefully he can be persuaded to return to a future FlossUK event. The ever resourceful FlossUK staff stepped into the breach and provided a great introduction to the platform.

Pi’s are bare metal devices and without screens keyboards and mice this workshop would have been tricky. The screens worked really well. Also good was having a bag of bits to plug in and play with the GPIO. The best thing about Pi’s is that they encourage you to give stuff a go that you wouldn’t risk more expensive or dangerous kit over (no immediate exposure to anything over 5v). After plugging in your first led display, getting the software to work, you are already visualising entire screen walls driven by Pi’s! *This* is the magic of Pi’s they are so accessible and hackable that you start out by trying tiny ideas and you are already thinking big by the time you’ve got to the end of the howto!

Along the way I played with Docker for the first time (yeah I know, but I’m busy!), checking out pi code from Github, getting introduced to node.js, Node-Red and started rethinking how our instrument teaching labs might be driven by Pi’s in the future..

Pi’s let you play without commitment because none of it takes very long. It is easier to play with hardware that is better supported so maybe not the cheap as chips full colour LED display (that had funny pin outs and would only display solid white for me) you can tangent off into funny stuff but also drill down into the genuinely useful but still disposable e.g. out of band comms, environmental monitoring, home media servers, etc, etc. There are lots of things out there and of course available to you, free, libre and open source!

Say aye, tae a PI!

Why Powershell?

by: Finn Kempers

I attended the Powershell course at FLOSSUK Spring Conference 2017. These are my day notes.

Introduction

Windows scripting cmd environment was awful. Python is needlessly complicated. Perl jokes. Shell script is simple. Data flow efficiency is required which good UNIX environments have. Microsoft realised they needed this themselves, which is why Powershell is made. Microsoft are adding bash and SSH (presumably into PowerShell). Office 365 or GUI can manage PowerShell. Registered Microsoft trademark. Scripting language, both usable and powerful.

Covered

Shell basics. Variables, Date & Time, Environment, Files, Scripts and script editors, logic, data and objects, writing scripts, and where next?
Written in mind for researchers handling data.

Commandlets

Command forms:

  • “verb-noun”. Example: “get-childitem”. This makes the commands fairly self-documentative.
  • Write-host “Hello World” (similar to echo)
  • [get-]help write-host (similar to -h or man on linux, man works too as an alias in Powershell)

Variables

Allows to reference values that can change. They will assume values however by typing for example “[float] $a=4”, it will specifically store a variable as a single point floating value. The type of a variable can be verified with “$VARIABLE.getType()”.

  • $a=4 assumed int
  • $b=7.2 assumed float
  • $name=”James”
  • $city=(“X”,”Y”)
  • $Filofax=$null

Still need to use dollar unlike shell script.

Hashes

Perl like to to key value pairs, also called a dictionary. Allows for lookups.

Environment

Get-childitem env:\ list all environment variables, or get a certain username by adding get-childitem env:\username. Typing $username shows username itself.
Environment is inherited from parent process. See Powerpoint for example commands. Environment variable in a child will only be what it is defined in the child, exiting to parent will make it use what it was defined in parent again, as it always has on linux and windows, but before on windows didn’t really access to.

  • Redirection and the pipe:
    • Symbols “>” “<” and “|”
      • It is basically straight up Unix, see the Powerpoint for the example.

Thoughts

Learning PowerShell is very useful for Unix people, because it is basically a Unix environment for Windows and offers great CV opportunity and is a possible new leader in shell environments to take on the current world.

  • “Hello world” | out-file (-append) C:\blah\hello.txt (write file, will overwrite unless appended)
  • Get-content C:\blah\hello.txt (will ping back “hello world” by reading, note also the capitalisation on commands like get-content is optional, “cat” is an alias to “get-content”)
  • $myfilecontents=get-content “filepath” and “$myfilecontent | out-file “filepath” will work on the content in the memory of a computer. Useful for big computers with lots of memory.

Scripts are only touched on here but are useful for frequent use and such. Cmlets can be scripts too, such make your own script that runs after invoking “make-dinner” if one makes such. MSDN has useful articles on learning PowerShell, simple search engine common sense to learning it. http://www.powersearchingwithgoogle.com/ can be used well with PowerShell too and useful. PowerShell is basically a ripoff for the best of Unix environments.

Rich object models, office integration and so on are features only available, when running on Windows. If running on a non-Linux OS, it is basically just a shell.

“foreach ( $file in gci . ){write-host -ForeGroundColor Cyan $file.FullName }” changes text colour. You can however click in windows the PowerShell icon on the window and click properties to change various settings too.

Also available is the Windows PowerShell Integrated Scripting Environment (ISE), which is a environment for making Powershell scripts.

Powershell gives easy scalable power that a lot of unix sysadmins will be familiar with. CSV’s can be imported into PowerShell rather nicely, see 11.3 in the Powershell.pdf.

The latter half of the tutorial was essentially chit chat, bending Powershell in a variety of ways to sate our various curiosities and discussing various technologies, though none hugely relevant to the topic itself. It is worth researching the topic of doing complex custom objects in Powershell:

Notes

Files are provided including the Powerpoint and in PDF form in the drive folder this is located in. It provides also a Powershell.pdf which is a workbook for people to try out. The workbook is also available from:

This is all property of the University of Edinburgh. Under the same site available is a Unix course available too.

About

Finn Kempers is currently studying web technologies at Lancaster and Morecambe College and as an intern developing in Perl, Javascript and Web Frameworks at Shadowcat Systems Limited.

[Note: The views and opinions represented in this review are those of the author and the author alone. FLOSSUK is not responsible for any opinions, comments or values expressed by the author. If you have any concern regarding what has been written you may contact the council at council@flossuk.org.]

Chef Workshop

by Simon Clayton

Knowing nothing at all about Chef I decided to spend the workshop day of the conference learning what I was missing out on. This turned out to be a great day of learning led by Anthony Hodson who is a solution architect at Chef.

The day started with an overview of Chef and then progressed into a series of exercises related to building environments and developing both unit and integration tests for the environments to prove that everything had deployed correctly.

Anthony had provided individual VMs for all participants so that there was no messing around with setup and configuration which was absolutely invaluable and meant that we could dive straight in.

There was certainly a lot of learning for someone who was completely new to it all but, the day progressed at a really good rate and we covered a lot of stuff. By the end of the day, we were happily expanding our configuration while using integrated debugging of the deployments and creating more unit tests while adding multiple platforms to our environment.

Overall, it was extremely useful and provided a fantastic introduction and overview of what’s possible with Chef.

About

Simon’s is a self-confessed geek who’s first computer was a ZX81 and he’s been working on PCs since MS-DOS 1.11 and *nix since 1996. He is currently a developer and devops guy in his own business (EventReference : https://eventreference.com) which provides online registration systems for the events industry.

[Note: The views and opinions represented in this review are those of the author and the author alone. FLOSSUK is not responsible for any opinions, comments or values expressed by the author. If you have any concern regarding what has been written you may contact the council at council@flossuk.org.]

Spring Conference: Hotels

Where would you like to stay…

The Tickets for the FLOSSUK Spring Conference, held in Manchester between the 14th-16th March, are now available and some of you will be planning your travel. We thought we’d take a moment to discuss the location and some of the available hotels in the area.

The Venue is at the Studio on 51 Lever Street Manchester and there are a number of nearby hotels (within 1 kilometre or 15 minutes casual walking) that can be used. The FLOSSUK Council have chosen the Britannia on Portland Street as their hotel of choice.

The following list is not comprehensive and there are others available on many popular sites such as Booking.com or Hotels.com. In particular there are apartment and hostel accommodation for those travelling in larger groups or on a budget.

There is a map (that will be updated with the hotels, conference locations and any suggestions we have for food, drink, shopping and entertainment) and it can be seen below or found here (http://bit.ly/flossuk2017gmap):

If you would like to add details to this map then please do so with the link provided (note that due to editing settings you may be required to request access to do so).

Please note that the Call for Papers and Workshops is still open and will remain so until the 24th January 2017.

Spring Conference: Raspberry Pi Workshop

Les Pounder to Present a Free Hackers Workshop

At this year’s Spring event in March we will have a traditional day of workshops on the Tuesday before the main conference (Tuesday 14th March). This event is a paid day of tutorials and introductions to essential tools and techniques. This year, however, we will also be hosting a free event sponsored by FLOSSUK on the same day and at the same location.

Les Pounder who writes a monthly column on the Raspberry Pi and Maker community, is a maker, sysadmin, Raspberry Pi tutor and You Tube blogger. Les will present a day of hacking using cheap hardware, free software and commonly available items that can be bought cheaply from the high street.

More details of the workshop and what you will need to bring will be published closer to the event itself.

The BigLesP

Les is a self-taught maker and hacker. He hosts regular Raspberry Jam events in Blackpool and helps teachers across the UK to learn more about physical computing and the maker culture.Image of the Workshop presenter Les Pounder

Les loves to tinker with broken toys and build projects using cost effective equipment to show that learning to hack doesn’t have to be expensive. Les can be found on his blog where he documents his adventures in low cost hacking using the Raspberry Pi and the BBC MicroBit Controller.

Les is a big fan of using shops such as Poundland (pronounced Pounderland in the Les Pounder verse) to build low cost fun items to both hack and learn. Aside from showing the physical skills he also documents the code and process for making hacking cheap, cheerful and lots of fun for all ages.

Please use the following links to learn more about Les:

FLOSSUK is dedicated to promoting open systems and freedoms in software, data, infrastructure, digital rights and hardware. Having Les run an event for us that is free to attend and open to all helps us to further reach out to the broader community.

Note that there will also be free places for students and those suffering financial restrictions available on our main conference days.

VM Brasseur to Keynote in 2017

VM Brasseur - keynote speaker

It is a great pleasure to announce that VM ‘Vicky’ Brasseur will open the 2017 FLOSSUK Spring conference with a keynote specially prepared for the event.*

Vicky is well known in the international open source world and has presented at a number of events so it is a real honour to have her at FLOSSUK.

In VM (aka Vicky)’s nearly 20 years in the tech. industry she has been an analyst, programmer, product manager, software engineering manager, director of software engineering, and C-level technical business and open source strategy consultant. Vicky is the winner of the Perl White Camel Award (2014) and the O’Reilly Open Source Award (2016).

Vicky occasionally blogs at {anonymous => ‘hash’}, often writes and is a moderator for opensource.com, and frequently tweets at @vmbrasseur.

You can find out more about the event by navigating here and also view the Call for Papers which is accepting submissions until January.

* More details of the Keynote will be announced in the coming weeks