Deploying Python Applications
This chapter will discuss how Python based web applications can be deployed on Virtual Server infrastructure. This is a common mode of deployment but also serves as a way of introducing virtualisation, working with Linux and configuration of web servers.
Virtual Machines
Virtualisation refers to the creation of multiple virtual computers running on one piece of hardware. One way to understand this is to think of the Virtual Machine as a software emulation of a real CPU, memory and disk running on another computer; for example, an Intel x86 based computer could simulate an old Nintendo GameBoy to run games for that platform. If the machine being emulated has the same architecture as the one it is running on then the task should be easier. The CPU instructions for the emulated machine translate directly to those for the host machine. This is now such a common and desirable thing to do that the CPU itself has support for running one or more virtual CPUs at the same time and this is then supported by the operating system with various kinds of support for running virtual machines efficiently. Importantly, the Virtual Machine is isolated from the host machine so that applications running on it cannot access host system resources and two VMs running on the same host cannot interfere with each other.
All of the major operating systems (Windows, Mac, Linux) support running virtual machines as a core part of the services that they provide. However, to run a VM, we need an application that can coordinate the services that are required. There are a number of products that do this: in the commercial space VMWare is a major player and there are a few Open Source options, the most widely used of which is VirtualBox which is now owned by Oracle. These applications do the job of managing all of the system resources and providing the bridge between the virtual machine and the host; for example, they can manage the network connection so that the VM can connect to the Internet via the host or be isolated from it. In this chapter we'll discuss examples with VirtualBox, but most should also work under other VM applications.
VirtualBox is a desktop application that coordinates the creation and running of virtual machines. It has versions for all major operating systems that run on the x86 and AMD64/Intel64 architectures. When you create a VM with VirtualBox you get a virtual computer that will start up just like a real one. When it is first created, there is no operating system installed, so the first task is to select a target operating system and install it into the VM. While it is possible to install Windows or MacOS into the VM (if you have the appropriate licence) we will concentrate on the Linux operating system. I won't go into the detail of the process of installation here - there are many guides available online. Once an operating system is installed, the virtual machine can be started and runs like a real computer, except that all parts of it are emulated on the host machine. The display of the VM is shown in a window on the desktop of the host; the mouse and keyboard from the host are routed to the VM when the window is active.
A configured VM in VirtualBox is stored as two separate files. The first
is the .vbox file that contains the configuration of the machine, the
second has the extension .vdi and is a disk image that contains the
data that is stored on the virtual hard-drive for the VM. Since these
are just files on the host system, they can be copied and shared. This
makes it possible for me to create a VM image with a particular
configuration and then publish it so that others can use the same
configuration. This turns out to be a very useful way of distributing
demonstration or development versions of some software packages since
one of the big issues with complex software is getting it to run in the
wide variety of environments that are out in the real world.
Distributing a VM image means that you can install and test the software
in a known environment and publish that for your users to test or use as
a development environment.
Since distributing VM images is so easy there are many sites around the web that offer a range of pre-built images for download. Of course one of the issues with this is the possibility that the image you download may have some mal-ware installed alongside the legitimate software that it advertises. For this reason you should be careful about downloading pre-built images unless you can be sure that the source is legitimate. The safest option is always to make a new VM and install an operating system obtained from a trusted source.
Screenshot of VirtualBox running a Linux VM
When we start up a VM having installed the Linux operating system,
VirtualBox presents us with a window that simulates the screen of the
machine. Since we've installed a basic version of the operating system,
this screen is just an old fashioned character based console - no window
system or fancy user interface. Of course it is possible to install a
window system (X Windows for Linux) on the VM; VirtualBox would then
show us this running on an emulated graphics card inside the window on
the desktop. However, our interest in using VMs is related to running
web servers so window-based user interfaces are not an important part of
the system. Web servers are always remote systems - even if they are in
your organisation they will be downstairs in the machine room - so we
need to interact with them remotely. This means that we need to learn
about the command line and how to control a computer without a mouse or
visual user interface.
Working with a Linux VM
-----------------------
In this discussion I will assume that we are working with a version of
the [Ubuntu Linux](http://www.ubuntu.com/) operating system. Ubuntu is a
variant of the Debian version of Linux; there are many other versions
since Linux is really just the name of the OS kernel and this needs to
be packaged with other software to make a full Operating System.
Versions differ in the way that they are configured and the way that new
packages are installed.
As discussed above, when we run a VM in VirtualBox we see a console
where we can login (the login name and password are set during the
installation of the OS, if you downloaded an image they should be
supplied in the documentation for the image). Once logged in we see the
command prompt (in my case `webdev@ubuntu-webdev:~$`) and the system is
waiting for us to enter a command. This is a typical prompt on Ubuntu
that contains information about who you are (webdev), the machine you
are logged into (ubuntu-webdev) and your current working directory (\~
which is your home directory). The `$` character at the end is just a
traditional prompt character. This is described as the *shell* - that
is, the program that prints the prompt and accepts your command input is
called the shell. By default it will be a program called `bash` which
stands for [Bourne Again
Shell](https://en.wikipedia.org/wiki/Bash_(Unix_shell)) (just a Unix
geek joke).
The main job of the shell from our point of view is to allow us to run
programs on the VM. In general you enter the name of a program and hit
return and it will be run. Arguments can be passed to the program.
Here's a simple example with the `ls` command that lists the files under
a directory (a directory is the linux term for what is normally called a
folder in Windows systems).
webdev@ubuntu-webdev:~$ ls /usr
bin games include lib local sbin share src
webdev@ubuntu-webdev:~$ ls -l /home
drwxr-xr-x 3 webdev webdev 4096 Aug 28 2015 webdev
$ apt-get install nginx
E: Could not open lock file /var/lib/dpkg/lock - open (13: Permission denied)
E: Unable to lock the administration directory (/var/lib/dpkg/), are you root?
`apt-get` is also used to keep your system up to date and install new
versions of packages if they have been released. You can run one command
to update the list of packages the system knows about:
$ sudo apt-get update
$ sudo apt-get upgrade
$ sudo apt-get install nginx
[sudo] password for webdev:
$ curl http://localhost/
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
body {
width: 35em;
margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif;
}
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>
The `sudo` command is an all powerful tool - take care when using it!
[ Source:
XKCD](https://xkcd.com/149/)
So in my case it is working; nginx has served up the test page included
in the distribution. If you're trying this and it did not work then a
number of things could be the cause of the problem. Firstly, did you see
any error messages when you installed nginx - if so read these carefully
and try to work out what the issue is. Note the comment above about
running `apt-get update` to enable the system to find newer pacakges. If
all seemed to go well but it just doesn't work, you can check to see
that the server is running with the following command:
$ sudo service nginx status
$ sudo service nginx restart
$ sudo apt-get install virtualbox-guest-utils
$ sudo adduser webdev vboxsf
$ sudo reboot
$ ls /media/sf_share
$ sudo apt-get install python3-bottle
$ cd /media/sf_share
$ python3 jsonlikes.py
$ python3 jsonlikes.py
Bottle v0.12.8 server starting up (using WSGIRefServer(restart=True))...
Listening on http://127.0.0.1:8080/
Hit Ctrl-C to quit.
$ python3 jsonlikes.py &> web.log &
[1] 1204
$ curl http://localhost:8080/
$ curl http://localhost:8080/likes
$ kill -SIGINT %1
$ kill -SIGINT 1204
The Bottle server we are running here is not meant for a real
deployment, it is only intended as a development server for very
lightweight use. It can only cope with one request at a time and so
would fail under any kind of load from the web. In a real production
situation you would use a real application server such as
[gunicorn](http://gunicorn.org/) or
[uWSGI](https://uwsgi-docs.readthedocs.io/en/latest/). Both of these are
Python specific application servers that can run your Bottle application
in a way that is more robust and able to handle real traffic. For
example, they will run more than one worker thread to handle multiple
requests at once. You should experiment with these servers to learn more
about a complete Python application deployment environment.
Both should have the effect of stopping the process and printing a
message to that effect. Now that the process is stopped we can examine
the log file `web.log` that should contain the output of the server when
we made the requests. Use the `cat` command to show the file contents:
$ cat web.log
Bottle v0.12.7 server starting up (using WSGIRefServer(restart=True))...
Listening on http://127.0.0.1:8080/
Hit Ctrl-C to quit.
127.0.0.1 - - [25/May/2016 02:00:32] "GET / HTTP/1.1" 200 692
127.0.0.1 - - [25/May/2016 02:00:35] "GET /likes HTTP/1.1" 200 36
/usr/lib/python3/dist-packages/bottle.py:3116: ResourceWar...
server {
listen 80 default_server;
listen [::]:80 default_server;
root /var/www/html;
# Add index.php to the list if you are using PHP
index index.html index.htm index.nginx-debian.html;
server_name _;
location / {
# First attempt to serve request as file, then
# as directory, then fall back to displaying a 404.
try_files $uri $uri/ =404;
}
}
server {
listen 80 default_server;
listen [::]:80 default_server;
root /var/www/html;
server_name _;
location / {
proxy_pass http://localhost:8080;
}
}
$ sudo cp /media/sf_share/default /etc/nginx/sites-available/default
$ sudo service nginx restart
$ curl http://localhost/
$ more /var/log/nginx/access.log
$ more /var/log/nginx/error.log
location /static {
root /media/sf_share;
}
$ sudo adduser www-data vboxsf