datastore WSGI app package

HiSPARC datastore WSGI application

This WSGI app is served by uWSGI on frome and lives at http://frome.nikef.nl/hisparc/upload.

example application.wsgi:

import sys
import functools

sys.path.append('/var/www/wsgi-bin/datastore')

from wsgi import wsgi_app

configfile = '/var/www/wsgi-bin/datastore/examples/config.ini'
application = functools.partial(wsgi_app.application, configfile=configfile)

configuration is read from a configuation file, shared between the WSGI app and the writer usually config.ini:

[General]
log=/tmp/hisparc.log
loglevel=debug
station_list=/tmp/station_list.csv
data_dir=/tmp/datastore

[Writer]
sleep=5

wsgi.rcodes module

Return codes for the WSGI app.

There are four types of return codes:

1. OK (100)

2. putEvent errors (20*)

   • 201: invalid input: checksum of data does not match

   • 203: wrong password

   • 206: station unknown (check station-list.csv)

   • 208: unpickling error

3. getEvent errors (30*)

   not used by WSGI app

4. Internal server error (40*)

   • 400: internal server error. Invalid post data.

wsgi.wsgi_app module

wsgi.wsgi_app.application(environ, start_response, configfile)

The hisparc upload application

This handler is called by uWSGI whenever someone requests our URL.

First, we generate a dictionary of POSTed variables and try to read out the station_id, password, checksum and data. When we do a readline(), we already read out the entire datastream. I don’t know if we can first check on station_id/password combinations before reading out the datastream without setting up a bidirectional communication channel.

When the checksum matches, we unpickle the event_list and pass everything on to store_event_list.

wsgi.wsgi_app.decode_object(o)

recursively decode all bytestrings in object

wsgi.wsgi_app.do_init(configfile)

Load configuration and passwords and set up a logger handler

This function will do one-time initialization. By using global variables, we eliminate the need to reread configuration and passwords on every request.

Configuration is read from the datastore configuation file (usually config.ini):

[General]
log=/tmp/hisparc.log
loglevel=debug
station_list=/tmp/station_list.csv
data_dir=/tmp/datastore

[Writer]
sleep=5

Station information is read from the station_list config variable. (station_list.csv on frome)

wsgi.wsgi_app.is_data_suspicious(event_list)

Check data for suspiciousness

Suspiciousness, a previously unknown quantum number that may signify the actual birth of the universe and the reweaving of past fates into current events has come to hunt us and our beloved data.

Note: Apr 7, 2019 0:00 is the default time after a cold start without GPS signal. The DAQ will happily send events even when no GPS signal has been acquired (yet). Events with timestamp Apr 7, 2019 are most probably caused by no or bad GPS signal. Such events must be eigenstates of suspiciousness.

wsgi.wsgi_app.store_data(station_id, cluster, event_list)

Store verified event data to temporary storage