Setting Up Sensors to Post to BMON

This document gives specific instructions on how to set up a couple different types of sensors to post their data to BMON system. In addition, general documentation is given to allow other types of sensors to post to BMON if they have Internet access ability and the ability to use the HTTP protocol to export data.

Specific instructions are provided for:

• Monnit Wireless Sensors

• The Mini-Monitor, a Raspberry Pi data acquisition system designed for the Alaska Housing Finance Corporation (AHFC) Building Monitoring project.

• Kors HTTP Poster for Niagara AX, a software module that allows building control systems utilizing the Niagara AX software framework to export data via HTTP Internet transactions.

Monnit Wireless Sensors

Monnit wireless sensors transfer their collected data to the iMonnit Internet site, where the data can viewed and the configuration of the sensor can be changed. Fortunately, the iMonnit software has a feature that allows the wireless sensor data to be sent to a separate Internet site, such as a BMON web site. To configure this feature for use with BMON, log in to the iMonnit system and perform the following steps to navigate to the page that allows you to configure a “Webhook”:

• On the home page after log in, click the cog icon in the upper right to access a drop-down menu. Select “API/Webhook” on that menu.

• A horizontal top menu bar is present on the resulting page. Click “Webhook”.

• On the next page, click the “Configure Webhook” button near the top right corner.

You are then presented with the form shown here:

On this screen, only one input needs to the entered, the Base URL (required) input. Here is a specific example of the input:

https://bms.ahfc.us/readingdb/reading/store/?format=monnit&storeKey=123xYz

and here is the general form:

[Base URL of BMON site]/readingdb/reading/store/?format=monnit&storeKey=[Store Key]

The two parameters that are specific to your BMON site are the Base URL of the BMON site and the Store Key associated with your BMON site (found in the settings.py file). Configuring this Webhook for the iMonnit site will cause all subsequent sensor readings to be sent to the BMON site and stored there. No configuration at the sensor level is required.

You need to substitute the web URL for your BMON site in the [URL of BMON site] placeholder, and you need to substitute your Store Key in the [store key] placeholder. When you installed BMON on the web server, you created and entered the Store Key as the BMSAPP_STORE_KEY setting in the BMON Settings File.

After Saving the dialog box, sensor data should be forwarded to the BMON site after it arrives at the iMonnit site. On the previous configuration dialog, there is a Last Result field that shows the result from the last sensor post to BMON. It should say X readings stored successfully, O rejected if the process is working. If not, an error message will appear in this field.

In the BMON system, each Monnit wireless sensor will appear with a BMON Sensor ID that is the same as the Monnit sensor ID; for example the Monnit Sensor with ID 139883 will appear in the BMON system with a Sensor ID of 139883.

Mini-Monitor

The Mini-Monitor data acquisition system based on the Raspberry Pi computer was developed for the AHFC Building Monitoring project. It directly posts data to the BMON system. As described in the Mini-Monitor Documentation, there are two critical entries for data posting in the Mini-Monitor settings file:

POST_URL = '[URL of BMON site]/readingdb/reading/store/'
POST_STORE_KEY = '[store key]'

[URL of BMON site] and [store key] have the same meaning as they did for the Monnit Wireless sensors. More information on setting up the Mini-Monitor is provided in its documentation.

Kors HTTP Poster for Niagara AX

If you have a building DDC control system that utilizes the Niagara AX software platform (e.g. the Tridium Jace system), you can add a $100 software module from Kors Engineering that will allow you to post point values to the BMON system. Documentation that comes with the Kors Poster module will assist in its install and configuration. You should use the PeriodicTrackerExtension for the Kors Poster, and the two key inputs in that extension are the Url, which should be:

[URL of BMON site]/readingdb/reading/[Sensor ID]/store/

[URL of BMON site] has the same meaning as before. For the [Sensor ID] placeholder, you need to create a unique Sensor ID for this DDC point. This will become the Sensor ID for this point in the BMON system. The second important input is Format, which should be entered as:

storeKey=[store key]&val=%newValue.value%

The [store key] placeholder has the same meaning as before.

The Kors Poster does not send any time information when it posts the point values. BMON will timestamp the readings when they arrive at the BMON system.

General Method for Gathering Data from Building Automation Systems

Most Building Automation/DDC Systems (BAS) can be configured to periodically create a file containing trend data for selected sensors and control points on the system. This file can then be used as a source of data to post to the BMON system. This section gives a general outline of how to set up a process for accomplishing this.

The general steps necessary are:

1. Install the file-sharing software Dropbox on the PC running the Building Automation System Management Software, or install it on a file server accessible to the Automation System. Create a folder in the Dropbox account that will hold the trend files created by the BAS.

2. Configure the Management Software for the Building Automation System to periodically create a trend file containing data from the sensors and control points of interest. Often, a useful time interval between recorded points is 10 to 15 minutes. Also, configure the software to create the trend file at least once daily, although for fresher data in BMON, creating the file every 4 hours is better. The best format for the trend files is Comma-Separated-Values (CSV). Configure the BAS to store these files in the Dropbox folder created in the prior step.

3. Install Dropbox on the BMON Server. Here are Instructions for a “headless” install of Dropbox on a Linux server. For the server, you can use the same Dropbox account as you set up in Step 1 or you can create a new Dropbox account associated with the BMON server; if you use a different account, Share the Dropbox folder you are using for the trend files with the Dropbox account used in the BMON server. After completion of this step, the BAS trend files will be readily accessible to the BMON server.

4. Finally, a script needs to be run on the BMON server to load the sensor data from the trend files into BMON. The csv-transfer utility is one such script that can continually monitor the Dropbox folder for new or changed files, and then load data present in those files into BMON. See the above linked documentation for that script. This script can actually reside on any PC or server that has access to the trend files and access to the Internet, including the BAS Management PC. The advantage of putting it on a cloud server running Dropbox is that it can be shared across multiple BAS systems and the owner of the BAS system does not need to install and configure the script; they have the simpler task of installing and configuring Dropbox.

This procedure has been implemented for a Siemens building automation system running the Insight APOGEE management software. This document explains how to create trend files on this system. The csv-transfer configuration file for posting the data from these files to BMON is shown below:

csv_files:
  - file_glob: "/home/amitchell2/Dropbox/cchrc_siemens/*.csv"
    file_type: siemens
    chunk_size: 10
    ts_tz: US/Alaska

consumers:
  - type: bmon
    poster_id:  an-bmon-01              # unique ID for this posting object
    bmon_store_url: https://bmon.analysisnorth.com/readingdb/reading/store/
    bmon_store_key: xyz1234567

Please note that an oftentimes better alternative to csv-transfer is available: file-to-bmon. This utility can load sensor readings from a set of files into BMON. There is more detail in the section below.

General Utility for Loading Sensor Data from Files

The file-to-bmon project provides a script that can be used to load sensor reading data from a set of files. The script is structured so that custom file Reader modules can be written in Python to parse any type of text file containing sensor readings. The required code in the Reader module is focused exclusively on parsing sensor reading data from file, so the complexity of the code is relatively low.

This script is often used to process data that is periodically uploaded to a server, such as 15-minute electric utility data or data from a conventional datalogger such as the Campbell Scientific datalogger.

Please look at the documentation at the above link for more information.

Particle WiFi and Cellular Microcontroller Boards

Particle makes microcontroller boards with built-in WiFi or Cellular data connectivity. Firmware on the board manages communication with a Particle Internet server that provides a simple and robust Application Programming Interface (API) for sending and receiving data from the board. These boards make it relatively simple to implement custom data collection systems that are able to post data to BMON. This section describes how a configure a Particle board to deliver data to a BMON server.

The essential steps involve:

• Creating a Webhook in your Particle account that forwards data from events to the BMON server.

• Writing code for the Particle board that publishes an event with data in the correct format for consumption by the BMON server.

Create a Webhook on the Particle Server

• Assuming you are logged into your Particle account, select the CONSOLE item from the right side of the top menu bar.

• From the resulting page, select the Integrations icon at the bottom of the left sidebar menu.

• Click the large plus sign to create a NEW INTEGRATION, and then choose Webhook for the type of Integration.

The top part of the resulting form should look like the following, except the following includes sample inputs:

For the Event Name input, create a name for the event that will be published by your Particle board. In this sample, the event name was called bmon_store.

In the URL input box, enter a URL with the format:

[Base URL of BMON site]/readingdb/reading/store/

In the sample shown, the Base URL of BMON site is https://bmon.analysisnorth.com.

Next, expand the Advanced Settings portion of the form to reveal the following inputs:

As shown in the above screenshot, select the JSON option in the Send Custom Data section. Then, enter a JSON object with two keys. As the value of the storeKey, enter the BMON store key inside the quotes (from your BMON Settings File). As the value of the format key, enter the word particle inside the quotes. Leave other inputs at their defaults and click the Create Webhook button at the bottom of the page to complete creation of the Webhook. Whenever a bmon_store event (in this example) is received by the Particle server, its data will be be forwarded to the BMON server.

Particle Event Format

The Particle server is now prepared to forward data to the BMON server. However, the Particle board must publish an event of the proper name in order for this transfer to occur. Given the example above where we set the Event Name to bmon_store, here is an example of a Publish statement coded on a Particle board that will send two sensor values to the BMON server:

Particle.publish("bmon_store", "Ttank=103.4 Tdhw=98.4", 60, PRIVATE);

The sensor names and values appear in the second parameter. The sensor name and the sensor value are separated by an equal sign, with no embedded spaces. The different sensors are separated by spaces. These two sensor readings will show up at the BMON server with the following two Sensor IDs, assuming the ID number of the Particle board is 260034000c47343432313031:

260034000c47343432313031_Ttank
260034000c47343432313031_Tdhw

The sensor readings will be timestamped with the time they were received by the Particle server. If you want the readings to be timestamped at some offset prior to the time they were published, you can cause that to occur by special formatting in the publish statement. Here is an example:

Particle.publish("bmon_store", "Ttank=103.4a Tdhw=98.4 a=300")

In this example, the timestamp for the Ttank reading will be 300 seconds prior to the publish time, because an a is appended to the Ttank reading. Since no a is appended to Tdhw, its timestamp will be the publish time with no adjustment. The a=300 part of the message is where the adjustment in seconds is set. The adjustment should be an integer value in seconds.

This feature is particularly useful when the Particle board is averaging a number of readings over a set interval, and the average value is being published. The most accurate point in time to associate the average value with is the middle of the averaging interval, not the time when the value is published. If the averaging interval is 10 minutes (600 seconds), moving the timestamp back 300 seconds from the publish time is desirable and can be accomplished using the feature described above.

General Instructions for Posting Data from Other Systems

If your sensors are not one of the above types, you may still be able to use the BMON system if your sensor system can make HTTP requests containing the sensor data. There are currently two methods for storing sensor data into the BMON system. The first method stores one sensor reading. The second method allows you to store multiple sensor readings from one or more sensors. Below are descriptions of those two methods.

Storing One Sensor Reading

To store one sensor reading, you can make either a GET or a POST HTTP request. The URL that the request should be made to is:

[URL of BMON site]/readingdb/reading/[Sensor ID]/store/

Substitute the URL of the BMON site for [URL of BMON site], and substitute the unique BMON Sensor ID for this sensor for the [Sensor ID] placeholder. Here is a valid URL:

https://bms.ahfc.us/readingdb/reading/gview_dhw_temp/store/

In this example, the BMON site URL is https://bms.ahfc.us, and the BMON Sensor ID for for this sensor is gview_dhw_temp; in order to see this data in the BMON system, you need to have a Sensor configured with the gview_dhw_temp Sensor ID. If sensor data is posted to the BMON system prior to configuring the Sensor, it will still be stored and will be visible once the Sensor is configured in BMON.

Next, you need to add a query string containing additional data about this sensor reading. The query string should be formatted like:

storeKey=[store key]&val=[sensor value]&ts=[date/time of reading]

The [store key] has been discussed before and is the BMSAPP_STORE_KEY setting you created in the BMON Settings File. [sensor value] is the sensor’s reading value that is being posted to the BMON database; it needs to be a number. [date/time of reading] is the time of the sensor reading expressed in Coordinated Universal Time (UTC). The date/time should be entered as a string value, such as Dec 3, 2014 10:03:45 PM, and it must be properly URL encoded. The prior date string, encoded properly, would be transmitted as: Dec+3%2C+2014+10%3A03%3A45+PM. The format of the date/time string is very flexible as the Python dateutil parser is used. All of the following strings are valid and parse to the same date/time.

December 3, 2014 10:03:45 PM
Dec 3, 2014 10:03:45 PM
12/3/2014 22:03:45

Finally, inclusion of the ts query parameter just discussed is optional. If it is not included, BMON will timestamp the sensor reading at the time it arrives at the BMON system.

Here is a valid GET request that posts a gview_dhw_temp temperature reading of 125.3 occurring at Dec 3, 2014 10:03:45 PM UTC:

https://bms.ahfc.us/readingdb/reading/gview_dhw_temp/store/?storeKey=123abc&val=125.3&ts=Dec+3%2C+2014+10%3A03%3A45+PM

The same request can be made with the POST method by sending a POST request to this URL:

https://bms.ahfc.us/readingdb/reading/gview_dhw_temp/store/

and including this query string in the message body of the POST request:

storeKey=123abc&val=125.3&ts=Dec+3%2C+2014+10%3A03%3A45+PM

The response from a successful HTTP request will be 1 readings stored successfully. If unsuccessful, an error message will be returned.

Storing Multiple Sensor Readings

It is possible to post multiple sensor readings to BMON from one or more sensors. To do so, an HTTP POST request must be made. The URL to send the request to is:

'[URL of BMON site]/readingdb/reading/store/'

where [URL of BMON site] is filled in with your BMON URL, e.g. https://bms.ahfc.us. The message body of the POST request should be a JSON string following the pattern of this example:

{"storeKey": "123abc",
 "readings": [
    [1432327040, "28.FFFA1C031400", 71.788],
    [1432327042, "test_cpu_temp", 45.527],
    [1432327040, "28.FF1A2D021400", 65.859]
    ]
}

The JSON object has two name/value pairs. The first name is the storeKey, and the associated value must contain the BMSAPP_STORE_KEY setting you created in the BMON Settings File. The next name is readings, the value of which is an array of sensor readings that you would like to store in BMON. Each element of that array is in turn a 3-element array representing one reading. In the example above, the first sensor reading is:

[1432327040, "28.FFFA1C031400", 71.788]

The 1432327040 element is the UTC timestamp of the sensor reading, expressed as a Unix timestamp (number of seconds since January 1, 1970). If you send a timestamp with fractional seconds, BMON will truncate to an integer value. If the timestamp is the JSON value null, the current time will be used for the timestamp. The "28.FFFA1C031400" element is the unique BMON Sensor ID surrounded in double-quotes. The 71.788 is the actual sensor reading and must be a number. There can be as many sensor readings (each a 3-element array) as you need.

If the Post was entirely successful, you receive a response like:

3 readings stored successfully

which tells you how many readings were successfully stored. If any errors occurred while storing readings, error messages separated by newline characters are appended to the above success phrase. If no readings were stored successfully, the response will be “0 readings stored successfully” plus an error message for each unsuccessful reading.