Installation of Frome

Granting davidf rights to manage software and services:

(root)$ visudo

and adding:

davidf  ALL = SOFTWARE, SERVICES

Preparing for source install:

(root)$ cd /usr/local/src/
(root)$ mkdir hisparc
(root)$ chown davidf.hisparc hisparc/
$ chmod g+w hisparc/

In /etc/ld.so.conf.d new file usrlocal.conf, to let ldconfig find libraries of locally installed software:

/usr/local/lib

Python

Python:

$ cd /usr/local/src/hisparc
$ wget http://www.python.org/ftp/python/2.6.4/Python-2.6.4.tgz
$ tar xvzf Python-2.6.4.tgz
$ cd Python-2.6.4
$ ./configure --enable-shared
$ make
(root)$ make install

Then, run:

(root)$ ldconfig

Now, the python libraries are registered.

Python Setuptools

From egg:

$ cd /usr/local/src/hisparc
$ wget http://pypi.python.org/packages/2.6/s/setuptools/setuptools-0.6c11-py2.6.egg#md5=bfa92100bd772d5a213eedd356d64086
(root)$ sh setuptools-0.6c11-py2.6.egg

IPython, an interactive Python shell

Download and install IPython:

(root)$ easy_install ipython

Web server

Install apache development libraries:

$ sudo yum install httpd-devel

================================================================================
 Package             Arch        Version                 Repository        Size
================================================================================
Installing:
 httpd-devel         i386        2.2.3-31.sl5.2          sl-security      147 k
 httpd-devel         x86_64      2.2.3-31.sl5.2          sl-security      147 k
Installing for dependencies:
 apr                 x86_64      1.2.7-11.el5_3.1        sl-security      118 k
 apr-devel           x86_64      1.2.7-11.el5_3.1        sl-security      237 k
 apr-util            x86_64      1.2.7-7.el5_3.2         sl-security       74 k
 apr-util-devel      x86_64      1.2.7-7.el5_3.2         sl-security       53 k
 httpd               x86_64      2.2.3-31.sl5.2          sl-security      1.2 M

Change configuration in /etc/httpd/conf/httpd.conf. Patch:

--- httpd.conf.orig     2009-12-04 14:35:39.000000000 +0100
+++ httpd.conf  2009-12-04 14:35:50.000000000 +0100
@@ -228,8 +228,8 @@
 #  when the value of (unsigned)Group is above 60000;
 #  don't use Group #-1 on these systems!
 #
-User apache
-Group apache
+User www
+Group www

 ### Section 2: 'Main' server configuration
 #

Enabling httpd on startup:

$ sudo /sbin/chkconfig --add httpd
$ sudo /sbin/chkconfig --levels 35 httpd on

Starting httpd now:

$ sudo /sbin/service httpd start

For mod_wsgi:

$ cd /usr/local/src/hisparc
$ wget http://modwsgi.googlecode.com/files/mod_wsgi-3.1.tar.gz
$ tar xvzf mod_wsgi-3.1.tar.gz
$ cd mod_wsgi-3.1
$ ./configure
$ make
(root)$ make install

Change configuration in /etc/httpd/conf/httpd.conf. Patch:

--- httpd.conf.orig     2009-12-04 15:19:01.000000000 +0100
+++ httpd.conf  2009-12-04 15:34:30.000000000 +0100
@@ -197,6 +197,7 @@
 LoadModule mem_cache_module modules/mod_mem_cache.so
 LoadModule cgi_module modules/mod_cgi.so
 LoadModule version_module modules/mod_version.so
+LoadModule wsgi_module modules/mod_wsgi.so

 #
 # The following modules are not loaded by default:

Restarting apache:

$ sudo /sbin/service httpd restart

Version control

Install git from source:

$ cd /usr/local/src/hisparc
$ wget https://git-core.googlecode.com/files/git-1.8.4.3.tar.gz
$ tar xvzf git-1.8.4.3.tar.gz
$ cd git-1.8.4.3.tar.gz
    $ make prefix=/usr/local all
    (root)$ sudo make prefix=/usr/local install

Datastore web application

The datastore application is driving our central data storage solution. It is a pure Python implementation under complete version control.

Prerequisites

The datastore application uses PyTables and the underlying HDF5 library to store binary data files. PyTables depends heavily on NumPy.:

(root)$ easy_install numpy

Now install the HDF5 library:

$ cd /usr/local/src/hisparc
$ wget http://www.hdfgroup.org/ftp/HDF5/prev-releases/hdf5-1.8.3/src/hdf5-1.8.3.tar.gz
$ tar xvzf hdf5-1.8.3.tar.gz
$ cd hdf5-1.8.3
$ ./configure --prefix=/usr/local
$ make
(root)$ make install
(root)$ ldconfig

And, finally, install PyTables itself:

(root)$ easy_install tables

Setting up datastore

In summary:

  • Created a /var/www/wsgi-bin directory from which to run the wsgi applications
  • Created a subdirectory owned by davidf.hisparc inside this wsgi-bin
  • Did a checkout of the datastore sources inside the subdirectory
  • Made a local copy of the application into the parent (wsgi-bin) and edited to set the correct local full path
  • Added the wsgi application to the Apache configuration

Here we go:

(root)$ cd /var/www
(root)$ mkdir wsgi-bin
(root)$ cd wsgi-bin
(root)$ mkdir datastore
(root)$ chown davidf.hisparc datastore
(root)$ chmod g+w datastore
$ git clone https://github.com/HiSPARC/datastore.git /var/www/wsgi-bin/datastore

Copy the application.wsgi and config.ini from the examples directory:

(root)$ cd /var/www/wsgi-bin
(root)$ cp datastore/examples/application.wsgi datastore.wsgi
(root)$ cp datastore/examples/config.ini datastore/
(root)$ chown davidf.hisparc datastore.wsgi datastore/config.ini
(root)$ chmod g+w datastore.wsgi datastore/config.ini

Edited /var/www/wsgi-bin/datastore.wsgi and set the correct paths:

sys.path.append('/var/www/wsgi-bin/datastore/wsgi')
configfile = ('/var/www/wsgi-bin/datastore/config.ini')

The config.ini now reads:

[General]
log=/var/log/hisparc/hisparc.log
loglevel=debug
station_list=/databases/frome/station_list.csv
data_dir=/databases/frome

[Writer]
sleep=1

I had to create the appropriate directory in /var/log and grant rights:

(root)$ cd /var/log
(root)$ mkdir hisparc
(root)$ chown www.hisparc hisparc
(root)$ chmod g+w hisparc

Then, added datastore to the Apache configuration:

(root)$ cd /etc/httpd/conf.d/
(root)$ touch hisparc.conf
(root)$ chown davidf.hisparc hisparc.conf
(root)$ chmod g+w hisparc.conf

And edited hisparc.conf to contain:

WSGIScriptAlias /hisparc/upload /var/www/wsgi-bin/datastore.wsgi

Reload Apache configuration:

$ sudo /sbin/service httpd reload

Writer

Write a wrapper for the writer:

$ vim /var/www/wsgi-bin/datastore/writer_app.py
"""Wrapper for the writer application"""

import sys

sys.path.append('/var/www/wsgi-bin/datastore/writer')

import writer

configfile = ('/var/www/wsgi-bin/datastore/config.ini')
writer.writer(configfile)

Start the writer app

(root)$ screen
(root)$ sudo -u www python /var/www/wsgi-bin/datastore/writer_app.py

This process will process incoming data and write them into the datastore

TODO

Run the script to receive configuration updates from the Public Database Based on the file: publicdb/scripts/fake-datastore-xmlrpc-server.py

$ runuser -l hisparc -c 'hisparc-datastore'

(Maybe) Not relevant

install: yum-utils easy_install paramiko easy_install dozer easy_install pil (requirement of dozer) easy_install mysql-python (for migration) install: gcc-gfortran easy_install virtualenvwrapper install: blas-devel lapack-devel (for scipy)