datastore writer package¶

HiSPARC datastore writer application

This script polls /datatore/frome/incoming for incoming data written by the WSGI app. It then writes the data into the raw datastore.

example writer_app.py:

"""Wrapper for the writer application"""

import sys

sys.path.append('/var/www/wsgi-bin/datastore/')

from writer import writer_app

configfile = '/var/www/wsgi-bin/datastore/examples/config.ini'
writer_app.writer(configfile)

configuration is read from a configuation file, shared between the WSGI app and the writer usually config.ini:

[General]
log=/tmp/hisparc.log
loglevel=debug
station_list=/tmp/station_list.csv
data_dir=/tmp/datastore

[Writer]
sleep=5

Writer application¶

datastore writer application

This module empties the station data incoming queue and writes the data into HDF5 files using PyTables.

writer.writer_app.decode_object(o)[source]¶: recursively decode all bytestrings in object

writer.writer_app.process_data(file, data_dir)[source]¶: Read data from a pickled object and store store in raw datastore

writer.writer_app.writer(configfile)[source]¶

hisparc datastore writer application

This script polls /datatore/frome/incoming for incoming data written by the WSGI app. It then store the data into the raw datastore.

Configuration is read from the datastore configuation file (usually config.ini):

[General]
log=/tmp/hisparc.log
loglevel=debug
station_list=/tmp/station_list.csv
data_dir=/tmp/datastore

[Writer]
sleep=5

Storage classes - raw datastore tables¶

Storage docstrings

class writer.storage.HisparcComparator¶

columns = {'comparator': UInt8Col(shape=(), dflt=0, pos=5), 'count': UInt16Col(shape=(), dflt=0, pos=6), 'device': UInt8Col(shape=(), dflt=0, pos=4), 'event_id': UInt32Col(shape=(), dflt=0, pos=0), 'ext_timestamp': UInt64Col(shape=(), dflt=0, pos=3), 'nanoseconds': UInt32Col(shape=(), dflt=0, pos=2), 'timestamp': Time32Col(shape=(), dflt=0, pos=1)}¶

class writer.storage.HisparcConfiguration¶

columns = {'buffer': Int32Col(shape=(), dflt=-1, pos=20), 'coinctime': Float64Col(shape=(), dflt=0.0, pos=12), 'delay_check': Float64Col(shape=(), dflt=0.0, pos=23), 'delay_error': Float64Col(shape=(), dflt=0.0, pos=24), 'delay_screen': Float64Col(shape=(), dflt=0.0, pos=22), 'detnum': UInt16Col(shape=(), dflt=0, pos=14), 'event_id': UInt32Col(shape=(), dflt=0, pos=0), 'gps_altitude': Float64Col(shape=(), dflt=0.0, pos=4), 'gps_latitude': Float64Col(shape=(), dflt=0.0, pos=2), 'gps_longitude': Float64Col(shape=(), dflt=0.0, pos=3), 'mas_ch1_adc_gain': Float64Col(shape=(), dflt=0.0, pos=49), 'mas_ch1_adc_offset': Float64Col(shape=(), dflt=0.0, pos=50), 'mas_ch1_comp_gain': Float64Col(shape=(), dflt=0.0, pos=53), 'mas_ch1_comp_offset': Float64Col(shape=(), dflt=0.0, pos=54), 'mas_ch1_current': Float64Col(shape=(), dflt=0.0, pos=33), 'mas_ch1_gain_neg': UInt8Col(shape=(), dflt=0, pos=40), 'mas_ch1_gain_pos': UInt8Col(shape=(), dflt=0, pos=39), 'mas_ch1_inttime': Float64Col(shape=(), dflt=0.0, pos=29), 'mas_ch1_offset_neg': UInt8Col(shape=(), dflt=0, pos=44), 'mas_ch1_offset_pos': UInt8Col(shape=(), dflt=0, pos=43), 'mas_ch1_thres_high': Float64Col(shape=(), dflt=0.0, pos=26), 'mas_ch1_thres_low': Float64Col(shape=(), dflt=0.0, pos=25), 'mas_ch1_voltage': Float64Col(shape=(), dflt=0.0, pos=31), 'mas_ch2_adc_gain': Float64Col(shape=(), dflt=0.0, pos=51), 'mas_ch2_adc_offset': Float64Col(shape=(), dflt=0.0, pos=52), 'mas_ch2_comp_gain': Float64Col(shape=(), dflt=0.0, pos=55), 'mas_ch2_comp_offset': Float64Col(shape=(), dflt=0.0, pos=56), 'mas_ch2_current': Float64Col(shape=(), dflt=0.0, pos=34), 'mas_ch2_gain_neg': UInt8Col(shape=(), dflt=0, pos=42), 'mas_ch2_gain_pos': UInt8Col(shape=(), dflt=0, pos=41), 'mas_ch2_inttime': Float64Col(shape=(), dflt=0.0, pos=30), 'mas_ch2_offset_neg': UInt8Col(shape=(), dflt=0, pos=46), 'mas_ch2_offset_pos': UInt8Col(shape=(), dflt=0, pos=45), 'mas_ch2_thres_high': Float64Col(shape=(), dflt=0.0, pos=28), 'mas_ch2_thres_low': Float64Col(shape=(), dflt=0.0, pos=27), 'mas_ch2_voltage': Float64Col(shape=(), dflt=0.0, pos=32), 'mas_common_offset': UInt8Col(shape=(), dflt=0, pos=47), 'mas_comp_thres_high': Float64Col(shape=(), dflt=0.0, pos=36), 'mas_comp_thres_low': Float64Col(shape=(), dflt=0.0, pos=35), 'mas_internal_voltage': UInt8Col(shape=(), dflt=0, pos=48), 'mas_max_voltage': Float64Col(shape=(), dflt=0.0, pos=37), 'mas_reset': BoolCol(shape=(), dflt=False, pos=38), 'mas_version': Int32Col(shape=(), dflt=-1, pos=5), 'password': Int32Col(shape=(), dflt=-1, pos=15), 'postcoinctime': Float64Col(shape=(), dflt=0.0, pos=13), 'precoinctime': Float64Col(shape=(), dflt=0.0, pos=11), 'reduce_data': BoolCol(shape=(), dflt=False, pos=19), 'slv_ch1_adc_gain': Float64Col(shape=(), dflt=0.0, pos=81), 'slv_ch1_adc_offset': Float64Col(shape=(), dflt=0.0, pos=82), 'slv_ch1_comp_gain': Float64Col(shape=(), dflt=0.0, pos=85), 'slv_ch1_comp_offset': Float64Col(shape=(), dflt=0.0, pos=86), 'slv_ch1_current': Float64Col(shape=(), dflt=0.0, pos=65), 'slv_ch1_gain_neg': UInt8Col(shape=(), dflt=0, pos=72), 'slv_ch1_gain_pos': UInt8Col(shape=(), dflt=0, pos=71), 'slv_ch1_inttime': Float64Col(shape=(), dflt=0.0, pos=61), 'slv_ch1_offset_neg': UInt8Col(shape=(), dflt=0, pos=76), 'slv_ch1_offset_pos': UInt8Col(shape=(), dflt=0, pos=75), 'slv_ch1_thres_high': Float64Col(shape=(), dflt=0.0, pos=58), 'slv_ch1_thres_low': Float64Col(shape=(), dflt=0.0, pos=57), 'slv_ch1_voltage': Float64Col(shape=(), dflt=0.0, pos=63), 'slv_ch2_adc_gain': Float64Col(shape=(), dflt=0.0, pos=83), 'slv_ch2_adc_offset': Float64Col(shape=(), dflt=0.0, pos=84), 'slv_ch2_comp_gain': Float64Col(shape=(), dflt=0.0, pos=87), 'slv_ch2_comp_offset': Float64Col(shape=(), dflt=0.0, pos=88), 'slv_ch2_current': Float64Col(shape=(), dflt=0.0, pos=66), 'slv_ch2_gain_neg': UInt8Col(shape=(), dflt=0, pos=74), 'slv_ch2_gain_pos': UInt8Col(shape=(), dflt=0, pos=73), 'slv_ch2_inttime': Float64Col(shape=(), dflt=0.0, pos=62), 'slv_ch2_offset_neg': UInt8Col(shape=(), dflt=0, pos=78), 'slv_ch2_offset_pos': UInt8Col(shape=(), dflt=0, pos=77), 'slv_ch2_thres_high': Float64Col(shape=(), dflt=0.0, pos=60), 'slv_ch2_thres_low': Float64Col(shape=(), dflt=0.0, pos=59), 'slv_ch2_voltage': Float64Col(shape=(), dflt=0.0, pos=64), 'slv_common_offset': UInt8Col(shape=(), dflt=0, pos=79), 'slv_comp_thres_high': Float64Col(shape=(), dflt=0.0, pos=68), 'slv_comp_thres_low': Float64Col(shape=(), dflt=0.0, pos=67), 'slv_internal_voltage': UInt8Col(shape=(), dflt=0, pos=80), 'slv_max_voltage': Float64Col(shape=(), dflt=0.0, pos=69), 'slv_reset': BoolCol(shape=(), dflt=False, pos=70), 'slv_version': Int32Col(shape=(), dflt=-1, pos=6), 'spare_bytes': UInt8Col(shape=(), dflt=0, pos=16), 'startmode': BoolCol(shape=(), dflt=False, pos=21), 'timestamp': Time32Col(shape=(), dflt=0, pos=1), 'trig_and_or': BoolCol(shape=(), dflt=False, pos=10), 'trig_external': UInt32Col(shape=(), dflt=0, pos=9), 'trig_high_signals': UInt32Col(shape=(), dflt=0, pos=8), 'trig_low_signals': UInt32Col(shape=(), dflt=0, pos=7), 'use_filter': BoolCol(shape=(), dflt=False, pos=17), 'use_filter_threshold': BoolCol(shape=(), dflt=False, pos=18)}¶

class writer.storage.HisparcError¶

HiSPARC Error messages tables

columns = {'event_id': UInt32Col(shape=(), dflt=0, pos=0), 'messages': Int32Col(shape=(), dflt=-1, pos=2), 'timestamp': Time32Col(shape=(), dflt=0, pos=1)}¶

class writer.storage.HisparcEvent¶

columns = {'baseline': Int16Col(shape=(4,), dflt=-1, pos=6), 'data_reduction': BoolCol(shape=(), dflt=False, pos=4), 'event_id': UInt32Col(shape=(), dflt=0, pos=0), 'event_rate': Float32Col(shape=(), dflt=0.0, pos=12), 'ext_timestamp': UInt64Col(shape=(), dflt=0, pos=3), 'integrals': Int32Col(shape=(4,), dflt=-1, pos=10), 'n_peaks': Int16Col(shape=(4,), dflt=-1, pos=8), 'nanoseconds': UInt32Col(shape=(), dflt=0, pos=2), 'pulseheights': Int16Col(shape=(4,), dflt=-1, pos=9), 'std_dev': Int16Col(shape=(4,), dflt=-1, pos=7), 'timestamp': Time32Col(shape=(), dflt=0, pos=1), 'traces': Int32Col(shape=(4,), dflt=-1, pos=11), 'trigger_pattern': UInt32Col(shape=(), dflt=0, pos=5)}¶

class writer.storage.HisparcSatellite¶

columns = {'event_id': UInt32Col(shape=(), dflt=0, pos=0), 'max_n': UInt16Col(shape=(), dflt=0, pos=4), 'max_signal': UInt16Col(shape=(), dflt=0, pos=7), 'mean_n': Float32Col(shape=(), dflt=0.0, pos=3), 'mean_signal': Float32Col(shape=(), dflt=0.0, pos=6), 'min_n': UInt16Col(shape=(), dflt=0, pos=2), 'min_signal': UInt16Col(shape=(), dflt=0, pos=5), 'timestamp': Time32Col(shape=(), dflt=0, pos=1)}¶

class writer.storage.HisparcSingle¶

columns = {'event_id': UInt32Col(shape=(), dflt=0, pos=0), 'mas_ch1_high': Int32Col(shape=(), dflt=-1, pos=3), 'mas_ch1_low': Int32Col(shape=(), dflt=-1, pos=2), 'mas_ch2_high': Int32Col(shape=(), dflt=-1, pos=5), 'mas_ch2_low': Int32Col(shape=(), dflt=-1, pos=4), 'slv_ch1_high': Int32Col(shape=(), dflt=-1, pos=7), 'slv_ch1_low': Int32Col(shape=(), dflt=-1, pos=6), 'slv_ch2_high': Int32Col(shape=(), dflt=-1, pos=9), 'slv_ch2_low': Int32Col(shape=(), dflt=-1, pos=8), 'timestamp': Time32Col(shape=(), dflt=0, pos=1)}¶

class writer.storage.LightningConfig¶

columns = {'altitude': Float64Col(shape=(), dflt=0.0, pos=10), 'angle_correction': Float32Col(shape=(), dflt=0.0, pos=16), 'baud_rate': Int16Col(shape=(), dflt=0, pos=3), 'close_alarm_distance': Int32Col(shape=(), dflt=0, pos=12), 'com_port': UInt8Col(shape=(), dflt=0, pos=2), 'daq_mode': BoolCol(shape=(), dflt=False, pos=7), 'database_name': Int32Col(shape=(), dflt=-1, pos=5), 'event_id': UInt32Col(shape=(), dflt=0, pos=0), 'help_url': Int32Col(shape=(), dflt=-1, pos=6), 'latitude': Float64Col(shape=(), dflt=0.0, pos=8), 'longitude': Float64Col(shape=(), dflt=0.0, pos=9), 'minimum_gps_speed': Int32Col(shape=(), dflt=0, pos=15), 'noise_beep': BoolCol(shape=(), dflt=False, pos=14), 'severe_alarm_distance': Int32Col(shape=(), dflt=0, pos=13), 'squelch_seting': Int32Col(shape=(), dflt=0, pos=11), 'station_id': UInt32Col(shape=(), dflt=0, pos=4), 'timestamp': Time32Col(shape=(), dflt=0, pos=1)}¶

class writer.storage.LightningError¶

columns = {'event_id': UInt32Col(shape=(), dflt=0, pos=0), 'messages': Int32Col(shape=(), dflt=-1, pos=2), 'timestamp': Time32Col(shape=(), dflt=0, pos=1)}¶

class writer.storage.LightningEvent¶

columns = {'corr_angle': Float32Col(shape=(), dflt=0.0, pos=5), 'corr_distance': Int16Col(shape=(), dflt=0, pos=2), 'event_id': UInt32Col(shape=(), dflt=0, pos=0), 'timestamp': Time32Col(shape=(), dflt=0, pos=1), 'uncorr_angle': Float32Col(shape=(), dflt=0.0, pos=4), 'uncorr_distance': Int16Col(shape=(), dflt=0, pos=3)}¶

class writer.storage.LightningNoise¶

columns = {'event_id': UInt32Col(shape=(), dflt=0, pos=0), 'timestamp': Time32Col(shape=(), dflt=0, pos=1)}¶

class writer.storage.LightningStatus¶

columns = {'close_alarm': BoolCol(shape=(), dflt=False, pos=4), 'close_rate': Int16Col(shape=(), dflt=0, pos=2), 'current_heading': Float32Col(shape=(), dflt=0.0, pos=6), 'event_id': UInt32Col(shape=(), dflt=0, pos=0), 'sever_alarm': BoolCol(shape=(), dflt=False, pos=5), 'timestamp': Time32Col(shape=(), dflt=0, pos=1), 'total_rate': Int16Col(shape=(), dflt=0, pos=3)}¶

class writer.storage.WeatherConfig¶

columns = {'altitude': Float64Col(shape=(), dflt=0.0, pos=10), 'barometer': BoolCol(shape=(), dflt=False, pos=15), 'baud_rate': Int16Col(shape=(), dflt=0, pos=3), 'com_port': UInt8Col(shape=(), dflt=0, pos=2), 'daq_mode': BoolCol(shape=(), dflt=False, pos=7), 'database_name': Int32Col(shape=(), dflt=-1, pos=5), 'dew_point': BoolCol(shape=(), dflt=False, pos=23), 'evapotranspiration': BoolCol(shape=(), dflt=False, pos=20), 'event_id': UInt32Col(shape=(), dflt=0, pos=0), 'heat_index': BoolCol(shape=(), dflt=False, pos=22), 'help_url': Int32Col(shape=(), dflt=-1, pos=6), 'humidity_inside': BoolCol(shape=(), dflt=False, pos=13), 'humidity_outside': BoolCol(shape=(), dflt=False, pos=14), 'latitude': Float64Col(shape=(), dflt=0.0, pos=8), 'longitude': Float64Col(shape=(), dflt=0.0, pos=9), 'offset_bar_sea_level': Float32Col(shape=(), dflt=0.0, pos=31), 'offset_inside_humidity': Int16Col(shape=(), dflt=0, pos=27), 'offset_inside_temperature': Float32Col(shape=(), dflt=0.0, pos=25), 'offset_outside_humidity': Int16Col(shape=(), dflt=0, pos=28), 'offset_outside_temperature': Float32Col(shape=(), dflt=0.0, pos=26), 'offset_station_altitude': Float32Col(shape=(), dflt=0.0, pos=30), 'offset_wind_direction': Int16Col(shape=(), dflt=0, pos=29), 'rain_rate': BoolCol(shape=(), dflt=False, pos=21), 'solar_radiation': BoolCol(shape=(), dflt=False, pos=18), 'station_id': UInt32Col(shape=(), dflt=0, pos=4), 'temperature_inside': BoolCol(shape=(), dflt=False, pos=11), 'temperature_outside': BoolCol(shape=(), dflt=False, pos=12), 'timestamp': Time32Col(shape=(), dflt=0, pos=1), 'uv_index': BoolCol(shape=(), dflt=False, pos=19), 'wind_chill': BoolCol(shape=(), dflt=False, pos=24), 'wind_direction': BoolCol(shape=(), dflt=False, pos=16), 'wind_speed': BoolCol(shape=(), dflt=False, pos=17)}¶

class writer.storage.WeatherError¶

columns = {'event_id': UInt32Col(shape=(), dflt=0, pos=0), 'messages': Int32Col(shape=(), dflt=-1, pos=2), 'timestamp': Time32Col(shape=(), dflt=0, pos=1)}¶

class writer.storage.WeatherEvent¶

columns = {'barometer': Float32Col(shape=(), dflt=0.0, pos=6), 'dew_point': Float32Col(shape=(), dflt=0.0, pos=14), 'evapotranspiration': Float32Col(shape=(), dflt=0.0, pos=11), 'event_id': UInt32Col(shape=(), dflt=0, pos=0), 'heat_index': Int16Col(shape=(), dflt=0, pos=13), 'humidity_inside': Int16Col(shape=(), dflt=0, pos=4), 'humidity_outside': Int16Col(shape=(), dflt=0, pos=5), 'rain_rate': Float32Col(shape=(), dflt=0.0, pos=12), 'solar_rad': Int16Col(shape=(), dflt=0, pos=9), 'temp_inside': Float32Col(shape=(), dflt=0.0, pos=2), 'temp_outside': Float32Col(shape=(), dflt=0.0, pos=3), 'timestamp': Time32Col(shape=(), dflt=0, pos=1), 'uv': Int16Col(shape=(), dflt=0, pos=10), 'wind_chill': Float32Col(shape=(), dflt=0.0, pos=15), 'wind_dir': Int16Col(shape=(), dflt=0, pos=7), 'wind_speed': Int16Col(shape=(), dflt=0, pos=8)}¶

writer.storage.get_or_create_cluster_group(file, cluster)[source]¶

Get an existing cluster group or create a new one

Parameters:

file – the PyTables data file
cluster – the name of the cluster

writer.storage.get_or_create_node(file, cluster, node)[source]¶

Get an existing node or create a new one

Parameters:

file – the PyTables data file
cluster – the parent (cluster) node
node – the node (e.g. events, blobs)

writer.storage.get_or_create_station_group(file, cluster, station_id)[source]¶

Get an existing station group or create a new one

Parameters:

file – the PyTables data file
cluster – the name of the cluster
station_id – the station number

writer.storage.open_or_create_file(data_dir, date)[source]¶

Open an existing file or create a new one

This function opens an existing PyTables file according to the event date. If the file does not yet exist, a new one is created.

Parameters:

data_dir – the directory containing all data files
date – the event date

Store events module¶

writer.store_events.data_is_blob(uploadcode, blob_types)[source]¶: Determine if data is a variable length binary value (blob)

writer.store_events.store_event(datafile, cluster, station_id, event)[source]¶

Stores an event in the h5 filesystem

Parameters:

datafile – the h5 data file
cluster – the name of the cluster to which the station belongs
station_id – the id of the station this event belongs to
event – the event to store

writer.store_events.store_event_list(data_dir, station_id, cluster, event_list)[source]¶: Store a list of events

datastore writer package¶

Writer application¶

Storage classes - raw datastore tables¶

Store events module¶

Upload codes¶

Table of Contents

Previous topic

Next topic

This Page