Using CSV Transfer¶
This document provides basic instructions on installing and using the CSV-Transfer Utility to import data files into BMON. Note that there is a newer utility that also can be used to transfer data from files of any format into BMON: file-to-bmon. Examine the features of the file-to-bmon utility before choosing the csv-tranfser utility.
The instructions provided are written based on having performed a local BMON installation but should work similarly if you have installed BMON on a Webfaction server based on these instructions.
The skills needed for installation are primarily:
- Linux command line skills - it is assumed you are familiar so explanations of basic linux command line operations are not included in this documentation.
Installing Dropbox¶
Note
The steps in this section are patterned after the general instructions from Digital Ocean’s How To Install Dropbox Client as a Service on Ubuntu 14.04
uname -a
x86_64 means 64 bit, x86 means 32 bit, this is important when installing dropbox, the wrong version won’t work
curl -Lo dropbox-linux-x86_64.tar.gz https://www.dropbox.com/download?plat=lnx.x86_64
curl -Lo dropbox-linux-x86.tar.gz https://www.dropbox.com/download?plat=lnx.x86
sudo mkdir -p /opt/dropbox
sudo tar xzfv dropbox-linux-x86_64.tar.gz --strip 1 -C /opt/dropbox
/opt/dropbox/dropboxd
Wait for the message This computer is now linked to Dropbox. Welcome USERNAME to appear on your bmon server
Kill the process by pressing Ctrl-C
sudo curl -o /etc/init.d/dropbox https://gist.githubusercontent.com/thisismitch/d0133d91452585ae2adc/raw/699e7909bdae922201b8069fde3011bbf2062048/dropbox
sudo chmod +x /etc/init.d/dropbox
sudo nano /etc/default/dropbox
example file
DROPBOX_USERS=”USERNAME”
where USERNAME is your linux username (not to be confused with your Dropbox username)
sudo systemctl daemon-reload
sudo systemctl start dropbox
sudo update-rc.d dropbox defaults
Installing Required Packages¶
cd $home
Install the Required Packages
Note
If you followed the instructions from How to Install BMON on a Local Web Server you may have already installed these packages. However, they were installed in the virtual environment which is encapsulated in its own entity. You will need to install these packages again to the non-virtual environment so the csv transfer tool can access them.
sudo pip install pyyaml
sudo pip install requests
sudo pip install pytz
Installing CSV Transfer¶
cd $home
sudo git clone https://github.com/alanmitchell/csv-transfer.git
cd csv-transfer
sudo cp sample_config.yaml config.yaml
The README file provided with csv-transfer includes more thorough documentation on explaining all of the parameters in the config file, see it for more information before proceeding either via sudo nano README
or by visiting the csv-transfer github repository online. Some basic tips are provided below to aid in properly configuring and running csv transfer successfully.
You will need to edit the config.yaml
file to instruct it where and how to read your files, do this by running sudo nano config.yaml
An example from a test config.yaml file:
file_glob:
indicates the path where your files are stored in your Dropbox folder, wild-cards (*.csv) are accepted if all of your files are in the same directory and will upload all files meeting that criteriaheader_rows:
the number of rows in the beginning of your file to be considered header or which do not contain data you wish to upload (see csv example below)name_row:
indication of which row (within the header count) contains the column names of your data, a 2 here means that of the 4 header rows, the second row contains column names (see csv example below)field_map:
is optional, in the example above field_map: “lambda nm: ‘_’.join(nm.split(‘_’)[:2])” strips the final two underscores of the column name ex. SOLAR_TundertankONEFOOT_F_Avg would become SOLAR_TundertankONEFOOT, remove this line if you do not wish to have your column names alteredts_tz:
enter the appropriate timezone for your area and/or the area the data is being generatedexclude_fields:
if you have arbitrary fields, like record numbers, you can enter them here to have them omitted from the importposter_id:
enter a unique idbmon_store_url:
is the full URL to the storage function of the BMON server, this will include http://SERVER IP OR URL/readingdb/reading/store, the only information to be changed is the portion immediately following http://bmon_store_key:
each BMON server has a unique and secret storage key string; providing this string is required for storing data on the BMON server, copy this from your bmon settings.py filesudo ./csv-transfer.py config.yaml
Incorporating Your Imported Data Into BMON¶
Follow the Adding Sensors instructions to add sensors to BMON if you haven’t done so already. The data structure within the SQLite database that BMON runs on is simple. The data from each sensor occupies its own table. The name of the table is the Sensor ID
in our case it’s the column name from our csv file.
An example .csv file
Troubleshooting¶
If you run the csv transfer tool and receive InsecurePlatformWarning or InsecureRequestWarning messages, do the following:
sudo nano /csv-transfer/consumers/httpPoster2.py
comment out the following lines by adding a # character at the beginning of each line
from requests.packages.urllib3.exceptions import InsecureRequestWarning, InsecurePlatformWarning
requests.packages.urllib3.disable_warnings(InsecureRequestWarning)
requests.packages.urllib3.disable_warnings(InsecurePlatformWarning)
to
#from requests.packages.urllib3.exceptions import InsecureRequestWarning, InsecurePlatformWarning
#requests.packages.urllib3.disable_warnings(InsecureRequestWarning)
#requests.packages.urllib3.disable_warnings(InsecurePlatformWarning)