Event Summary Data

Fetch events and other data from the event summary data (ESD).

This module enables you to access the event summary data.

If you are in a real hurry and know what you’re doing (and took the time to read this far), you can call the quick_download() function like this:

>>> from sapphire import quick_download
>>> data = quick_download(501)

For regular use, look up download_data().

sapphire.esd.get_base_url()
sapphire.esd.get_events_url()
sapphire.esd.get_weather_url()
sapphire.esd.get_singles_url()
sapphire.esd.get_lightning_url()
sapphire.esd.get_coincidences_url()
sapphire.esd.quick_download(station_number, date=None)

Quickly download some data

Parameters:

  • station_number – The HiSPARC station number.

  • date – the date for which to get data (datetime.datetime instance), passed unchanged to the download_data() as :param start:.

Returns:

handle to an open PyTables file.

Everything is handled by this function, including file creation. Expect no frills: you just get yesterday’s data.

Example usage:

>>> import sapphire.esd
>>> data = sapphire.esd.quick_download(501)
>>> print(data)
data1.h5 (File) u''
Last modif.: 'Mon Jun  9 22:03:50 2014'
Object Tree:
/ (RootGroup) u''
/s501 (Group) u''
/s501/events (Table(58898,)) ''
sapphire.esd.load_data(file, group, tsv_file, type='events')

Load downloaded event summary data into PyTables file.

If you’ve previously downloaded event summary data from the HiSPARC Public Database in TSV format, you can load them into a PyTables file using this method. The result is equal to directly downloading data using download_data().

Parameters:

  • file – the PyTables datafile handler.

  • group – the PyTables destination group, which need not exist.

  • tsv_file – path to the tsv file downloaded from the HiSPARC Public Database.

  • type – the datatype to load, either ‘events’, ‘weather’, ‘singles’, or ‘lightning’.

Example:

>>> import tables
>>> import sapphire.esd
>>> data = tables.open_file('data.h5', 'w')
>>> sapphire.esd.load_data(data, '/s501', 'events-s501-20130910.tsv')
sapphire.esd.download_data(file, group, station_number, start=None, end=None, type='events', progress=True)

Download event summary data

Parameters:

  • file – the PyTables datafile handler.

  • group – the PyTables destination group, which need not exist.

  • station_number – The HiSPARC station number for which to get data.

  • start – a datetime instance defining the start of the search interval.

  • end – a datetime instance defining the end of the search interval.

  • type – the datatype to download, either ‘events’, ‘weather’, or ‘singles’.

  • progress – if True show a progressbar while downloading.

If group is None, use ‘/s<station_number>’ as a default.

The start and stop parameters may both be None. In that case, yesterday’s data is downloaded. If only end is None, a single day’s worth of data is downloaded, starting at the datetime specified with start.

Example:

>>> import tables
>>> import datetime
>>> import sapphire.esd
>>> data = tables.open_file('data.h5', 'w')
>>> sapphire.esd.download_data(data, '/s501', 501,
...     datetime.datetime(2013, 9, 1), datetime.datetime(2013, 9, 2))
sapphire.esd.download_lightning(file, group, lightning_type=4, start=None, end=None, progress=True)

Download KNMI lightning data

Parameters:

  • file – the PyTables datafile handler.

  • group – the PyTables destination group, which need not exist.

  • lightning_type – type of lightning, see list below for the possible type codes.

  • start – a datetime instance defining the start of the search interval.

  • end – a datetime instance defining the end of the search interval.

  • progress – if True show a progressbar while downloading.

Lightning types:

0 - Single-point 1 - Cloud-cloud 2 - Cloud-cloud mid 3 - Cloud-cloud end 4 - Cloud-ground (default) 5 - Cloud-ground return

sapphire.esd.load_coincidences(file, tsv_file, group='')

Load downloaded event summary data into PyTables file.

If you’ve previously downloaded coincidence data from the HiSPARC Public Database in TSV format, you can load them into a PyTables file using this method. The result is equal to directly downloading data using download_coincidences().

Parameters:

  • file – the PyTables datafile handler.

  • tsv_file – path to the tsv file downloaded from the HiSPARC Public Database containing coincidences.

  • group – the PyTables destination group, which need not exist.

Example:

>>> import tables
>>> import sapphire.esd
>>> data = tables.open_file('coincidences.h5', 'w')
>>> sapphire.esd.load_coincidences(data, 'coincidences-20151130.tsv')
sapphire.esd.download_coincidences(file, group='', cluster=None, stations=None, start=None, end=None, n=2, progress=True)

Download event summary data coincidences

Parameters:

  • file – PyTables datafile handler.

  • group – path of destination group, which need not exist yet.

  • cluster – HiSPARC cluster name for which to get data.

  • stations – a list of HiSPARC station numbers for which to get data.

  • start – a datetime instance defining the start of the search interval.

  • end – a datetime instance defining the end of the search interval.

  • n – the minimum number of events in the coincidence.

The start and end parameters may both be None. In that case, yesterday’s data is downloaded. If only end is None, a single day’s worth of data is downloaded, starting at the datetime specified with start.

Optionally either a cluster or stations can be defined to limit the results to include only events from those stations.

Example:

>>> import tables
>>> import datetime
>>> import sapphire.esd
>>> data = tables.open_file('data_coincidences.h5', 'w')
>>> sapphire.esd.download_coincidences(data, cluster='Aarhus',
...     start=datetime.datetime(2013, 9, 1),
...     end=datetime.datetime(2013, 9, 2), n=3)
class sapphire.esd.ReadLineAndStoreEventClass(table)

Store lines of event data from the ESD

Use this contextmanager to store events from a TSV file into a PyTables table.

Parameters:

table – a PyTables Table object in which to store the data.

store_line(line)

Store a single line

Parameters:

line – the line to store as a tuple of strings (one element per column).

Returns:

timestamp of the stored event, or 0 if the given line was a comment line starting with a ‘#’.

class sapphire.esd.ReadLineAndStoreWeatherClass(table)

Store lines of weather data from the ESD

store_line(line)

Store a single line

Parameters:

line – the line to store as a tuple of strings (one element per column).

Returns:

timestamp of the stored event, or 0 if the given line was a comment line starting with a ‘#’.

class sapphire.esd.ReadLineAndStoreSinglesClass(table)

Store lines of singles data from the ESD

store_line(line)

Store a single line

Parameters:

line – the line to store as a tuple of strings (one element per column).

Returns:

timestamp of the stored event, or 0 if the given line was a comment line starting with a ‘#’.

class sapphire.esd.ReadLineAndStoreLightningClass(table)

Store lines of lightning data from the ESD

store_line(line)

Store a single line

Parameters:

line – the line to store as a tuple of strings (one element per column).

Returns:

timestamp of the stored event, or 0 if the given line was a comment line starting with a ‘#’.