microbuilder.io

Plotting JSON Data with Chart.js

Sun, 10 Jan 2016 21:23:00 +0000

Chart.js is a open-source, light-weight (~11KB) library based on the HTML5 canvas control. It supports a limited number of charts, but unlike Google Charts it can be embedded directly in your website and made available offline.

Table of Contents

Basic Usage
- Static JSON Data
- Loading JSON Data with JQuery
Line Charts
- Example
- Source Code
Global Settings

Basic Usage

To use Chart.js, you will need to include the Chart.js source, add a Canvas control to your page, and instantiate a new chart passing in a reference to your dataset:

<script src="/assets/js/Chart.min.js"></script>
<script src="/assets/js/jquery-1.12.0.min.js"></script> <!--Optional-->

<canvas id="myChart" width="740" height="200"></canvas>

<script>
  // Get the context of the canvas element we want to select
  var ctx = document.getElementById("myChart").getContext("2d");

  // Instantiate a new chart using 'data' (defined below)
  var myChart = new Chart(ctx).Line(data);
</script>

Note: For details on defining the default configuration settings for all of your charts see the Global Settings section later in this guide.

Static JSON Data

You can include a static dataset directly on your page by defining your data variable in the script section of your page:

var data = {
    labels: ["January", "February", "March", "April", "May", "June", "July"],
    datasets: [
      {
        label: "My First dataset",
        fillColor: "rgba(220,220,220,0.2)",
        strokeColor: "rgba(220,220,220,1)",
        pointColor: "rgba(220,220,220,1)",
        pointStrokeColor: "#fff",
        pointHighlightFill: "#fff",
        pointHighlightStroke: "rgba(220,220,220,1)",
        data: [65, 59, 80, 81, 56, 55, 40]
      },
      {
        label: "My Second dataset",
        fillColor: "rgba(151,187,205,0.2)",
        strokeColor: "rgba(151,187,205,1)",
        pointColor: "rgba(151,187,205,1)",
        pointStrokeColor: "#fff",
        pointHighlightFill: "#fff",
        pointHighlightStroke: "rgba(151,187,205,1)",
        data: [28, 48, 40, 19, 86, 27, 90]
      }
    ]
  };

This will give you the following chart:

Loading JSON Data with JQuery

jQuery can also be used to load JSON data from a URL, but in most cases the raw JSON data will also need to be converted to a format that chart.js understands.

The code below will use jQuery to load JSON data from a URL and separate it into two arrays (labels, data), and then insert that data into an object chart.js understands (tempData):

function drawLineChart() {

  // Add a helper to format timestamp data
  Date.prototype.formatMMDDYYYY = function() {
      return (this.getMonth() + 1) +
      "/" +  this.getDate() +
      "/" +  this.getFullYear();
  }

  var jsonData = $.ajax({
    url: 'http://d.microbuilder.io:8080/test/temp',
    dataType: 'json',
  }).done(function (results) {

    // Split timestamp and data into separate arrays
    var labels = [], data=[];
    results["packets"].forEach(function(packet) {
      labels.push(new Date(packet.timestamp).formatMMDDYYYY());
      data.push(parseFloat(packet.payloadString));
    });

    // Create the chart.js data structure using 'labels' and 'data'
    var tempData = {
      labels : labels,
      datasets : [{
          fillColor             : "rgba(151,187,205,0.2)",
          strokeColor           : "rgba(151,187,205,1)",
          pointColor            : "rgba(151,187,205,1)",
          pointStrokeColor      : "#fff",
          pointHighlightFill    : "#fff",
          pointHighlightStroke  : "rgba(151,187,205,1)",
          data                  : data
      }]
    };

    // Get the context of the canvas element we want to select
    var ctx = document.getElementById("myLineChart").getContext("2d");

    // Instantiate a new chart
    var myLineChart = new Chart(ctx).Line(tempData);
  });
}

drawLineChart();

Line Charts

Example

A sample chart can be seen below based on temperature data logged from MQTT publications to the /test/temp topic. Data was published to this topic via the following command (using node and the mqtt package installed via sudo npm install -g mqtt):

$ mqtt pub -h d.microbuilder.io -t test/temp -m '17.8'

Note: The MQTT broker at d.microbuilder.io is shown as an example and is not publicly accessible, although the JSON records are publicly available.

Source Code

The Javascript code required to read JSON data from a specific URL is shown below. Please note that this code will attempt to convert the .payloadString field in the JSON record to a float value, so any string based payloads will fail to parse.

<script src="/assets/js/jquery-1.12.0.min.js"></script>
<script src="/assets/js/Chart.min.js"></script>

<canvas id="myLineChart" width="740" height="200"></canvas>

<script>
  function drawLineChart() {

    // Add a helper to format timestamp data
    Date.prototype.formatMMDDYYYY = function() {
        return (this.getMonth() + 1) +
        "/" +  this.getDate() +
        "/" +  this.getFullYear();
    }

    var jsonData = $.ajax({
      url: 'http://d.microbuilder.io:8080/test/temp',
      dataType: 'json',
    }).done(function (results) {

      // Split timestamp and data into separate arrays
      var labels = [], data=[];
      results["packets"].forEach(function(packet) {
        labels.push(new Date(packet.timestamp).formatMMDDYYYY());
        data.push(parseFloat(packet.payloadString));
      });

      // Create the chart.js data structure using 'labels' and 'data'
      var tempData = {
        labels : labels,
        datasets : [{
            fillColor             : "rgba(151,187,205,0.2)",
            strokeColor           : "rgba(151,187,205,1)",
            pointColor            : "rgba(151,187,205,1)",
            pointStrokeColor      : "#fff",
            pointHighlightFill    : "#fff",
            pointHighlightStroke  : "rgba(151,187,205,1)",
            data                  : data
        }]
      };

      // Get the context of the canvas element we want to select
      var ctx = document.getElementById("myLineChart").getContext("2d");

      // Instantiate a new chart
      var myLineChart = new Chart(ctx).Line(tempData, {
        //bezierCurve: false
      });
    });
  }

  drawLineChart();
</script>

Global Settings

Chart.js uses a set of global settings that determine how every chart will be rendered. If you wish to change any of settings from the library defaults you should include the global settings section below in your script:

// Set the global Chart values (these will apply to all charts)
Chart.defaults.global = {
  // Boolean - Whether to animate the chart
  animation: true,

  // Number - Number of animation steps
  animationSteps: 60,

  // String - Animation easing effect
  // Possible effects are:
  // [easeInOutQuart, linear, easeOutBounce, easeInBack, easeInOutQuad,
  //  easeOutQuart, easeOutQuad, easeInOutBounce, easeOutSine, easeInOutCubic,
  //  easeInExpo, easeInOutBack, easeInCirc, easeInOutElastic, easeOutBack,
  //  easeInQuad, easeInOutExpo, easeInQuart, easeOutQuint, easeInOutCirc,
  //  easeInSine, easeOutExpo, easeOutCirc, easeOutCubic, easeInQuint,
  //  easeInElastic, easeInOutSine, easeInOutQuint, easeInBounce,
  //  easeOutElastic, easeInCubic]
  animationEasing: "easeOutQuart",

  // Boolean - If we should show the scale at all
  showScale: true,

  // Boolean - If we want to override with a hard coded scale
  scaleOverride: false,

  // ** Required if scaleOverride is true **
  // Number - The number of steps in a hard coded scale
  scaleSteps: null,
  // Number - The value jump in the hard coded scale
  scaleStepWidth: null,
  // Number - The scale starting value
  scaleStartValue: null,

  // String - Colour of the scale line
  scaleLineColor: "rgba(0,0,0,.1)",

  // Number - Pixel width of the scale line
  scaleLineWidth: 1,

  // Boolean - Whether to show labels on the scale
  scaleShowLabels: true,

  // Interpolated JS string - can access value
  scaleLabel: "<%=value%>",

  // Boolean - Whether the scale should stick to integers, not floats even if drawing space is there
  scaleIntegersOnly: true,

  // Boolean - Whether the scale should start at zero, or an order of magnitude down from the lowest value
  scaleBeginAtZero: false,

  // String - Scale label font declaration for the scale label
  scaleFontFamily: "'Helvetica Neue', 'Helvetica', 'Arial', sans-serif",

  // Number - Scale label font size in pixels
  scaleFontSize: 12,

  // String - Scale label font weight style
  scaleFontStyle: "normal",

  // String - Scale label font colour
  scaleFontColor: "#666",

  // Boolean - whether or not the chart should be responsive and resize when the browser does.
  responsive: false,

  // Boolean - whether to maintain the starting aspect ratio or not when responsive, if set to false, will take up entire container
  maintainAspectRatio: true,

  // Boolean - Determines whether to draw tooltips on the canvas or not
  showTooltips: true,

  // Function - Determines whether to execute the customTooltips function instead of drawing the built in tooltips (See [Advanced - External Tooltips](#advanced-usage-external-tooltips))
  customTooltips: false,

  // Array - Array of string names to attach tooltip events
  tooltipEvents: ["mousemove", "touchstart", "touchmove"],

  // String - Tooltip background colour
  tooltipFillColor: "rgba(0,0,0,0.8)",

  // String - Tooltip label font declaration for the scale label
  tooltipFontFamily: "'Helvetica Neue', 'Helvetica', 'Arial', sans-serif",

  // Number - Tooltip label font size in pixels
  tooltipFontSize: 14,

  // String - Tooltip font weight style
  tooltipFontStyle: "normal",

  // String - Tooltip label font colour
  tooltipFontColor: "#fff",

  // String - Tooltip title font declaration for the scale label
  tooltipTitleFontFamily: "'Helvetica Neue', 'Helvetica', 'Arial', sans-serif",

  // Number - Tooltip title font size in pixels
  tooltipTitleFontSize: 14,

  // String - Tooltip title font weight style
  tooltipTitleFontStyle: "bold",

  // String - Tooltip title font colour
  tooltipTitleFontColor: "#fff",

  // String - Tooltip title template
  tooltipTitleTemplate: "<%= label%>",

  // Number - pixel width of padding around tooltip text
  tooltipYPadding: 6,

  // Number - pixel width of padding around tooltip text
  tooltipXPadding: 6,

  // Number - Size of the caret on the tooltip
  tooltipCaretSize: 8,

  // Number - Pixel radius of the tooltip border
  tooltipCornerRadius: 6,

  // Number - Pixel offset from point x to tooltip edge
  tooltipXOffset: 10,
  
  // String - Template string for single tooltips
  tooltipTemplate: "<%if (label){%><%=label%>: <%}%><%= value %>",
  
  // String - Template string for multiple tooltips
  multiTooltipTemplate: "<%= value %>",

  // Function - Will fire on animation progression.
  onAnimationProgress: function(){},

  // Function - Will fire on animation completion.
  onAnimationComplete: function(){}
}

You can override these default values in individual charts during instantiation:

new Chart(ctx).Line(data, {
  bezierCurve: false
});

Plotting JSON Data with Google Charts

Wed, 30 Dec 2015 05:40:00 +0000

This example shows how to take raw JSON data and plot it using the Google Charts API. See Google’s chart gallery for examples of what you can plot based on your JSON data.

Line Charts

Example

$ mqtt pub -h d.microbuilder.io -t test/temp -m '17.8'

Note: The MQTT broker at d.microbuilder.io is shown as an example and is not publicly accessible, although the JSON records are publicly available.

Source Code

<div id="chart" style="width: 100%"></div>

<script src="//ajax.googleapis.com/ajax/libs/jquery/1.11.1/jquery.min.js"></script>
<script src="https://www.google.com/jsapi"></script>

<script>

// 3. This function fires when Google Charts has been fully loaded
function drawChart() {

  // 4. Retrieve the raw JSON data
  var jsonData = $.ajax({
    url: 'http://d.microbuilder.io:8080/test/temp',
    dataType: 'json',
  }).done(function (results) {

    // 5. Create a new DataTable (Charts expects data in this format)
    var data = new google.visualization.DataTable();

    // 6. Add two columns to the DataTable
    data.addColumn('datetime', 'Time');
    data.addColumn('number',   'Temp');

    // 7. Cycle through the records, adding one row per record
    results["packets"].forEach(function(packet) {
      data.addRow([
          (new Date(packet.timestamp)),
          parseFloat(packet.payloadString),
        ]);
    });

    // 8. Create a new line chart
    var chart = new google.visualization.LineChart($('#chart').get(0));

    // 9. Render the chart, passing in our DataTable and any config data
    chart.draw(data, {
      title:  'Ambient Temperature (/test/temp)',
      height: 150
    });

  });

}

// 1. Start loading Google Charts
google.load('visualization', '1', {
  packages: ['corechart']
});

// 2. Set a callback function to fire when loading is complete
google.setOnLoadCallback(drawChart);

</script>

MQTT to JSON HTTP Logger

Tue, 29 Dec 2015 05:40:00 +0000

This guide will help you setup an MQTT broker and HTTP server that logs all published MQTT data to a local database, and exposes the data in JSON format over HTTP. An AWS EC2 instance running Ubuntu Server 14.04 will be used here, but a Raspberry Pi would also work fine.

Table of Contents

What is MQTT?
Objectives
System Requirements
MQTT-Log Source
Server Setup
Starting MQTT-Log as a PM2 Process
Managing the MQTT-Log/PM2 Process
Testing the System
Credits

What is MQTT?

If you’re not familiar with MQTT it’s a light-weight publish-subscribe protocol that sits between sensor nodes on one side, and various data subscribers on the other.

It allows you to decouple the data provider(s) from the data consumer(s) in a scalable, flexible manner, and with a minimum of complexity.

MQTT uses a key-based system, where records are stored and accessed via a key called a topic.

Topics are hierarchical strings in a similar format as HTTP requests or file and folder names:

home/kitchen/fridge/temperature
home/kitchen/fridge/luminosity
office/secondfloor/kitchen/temperature

By publishing to a topic, you overwrite the previous value, and any device that has subscribed to that topic will receive an alert letting it know that the topic has changed, along with the updated value.

Each topic can have multiple publishers or subscribers, and new subscribers can join or leave at any moment.

Publishers can also be subscribers at the same time, meaning that communication can happen in both directions using any number of topics and involving any number of endpoints or sensor nodes.

Objectives

There are a variety of excellent platforms out there that are based on or use MQTT (Adafruit IO, etc.).

If you simply want to setup a quick test platform to capture MQTT publications, though, and make sure the data is arriving properly, the existing open source or proprietary services might be more complicated than you want or need.

This guide tries to provide the easiest possible solution to:

Setup an MQTT Broker that can be accessed from anywhere
Record all published data in a local database, including historical data
Expose all published data for a specific topic in JSON format via HTTP

System Requirements

This guide was tested using a clean t2.medium AWS EC2 instance running Ubuntu 14.04 Server, although a Raspberry Pi or any other similar Debian-based computer should also work fine.

The tutorial assumes that you have shell access, that the server has internet access, and that you have already opened up the following ports (if you plan to access the system remotely):

1883 - MQTT access
8080 - HTTP requests

Details aren’t provided here on configuring the Security Group for incoming and outgoing traffic, or the DNS setup if you want to redirect a domain or sub-domain to your server. Google should be able to help you there.

MQTT-Log Source

The node.js utility that runs the MQTT broker (aedes), writes the code to the database (nedb), opens the HTTP Server and supplies the JSON data is shown below is available on Github.

Latest Source: https://github.com/adafruit/mqtt-log

The code below is purely for illustration purposes and may be out of date! Always pull from the git repo for the latest updates.

Server Setup

To setup mqtt-log on a clean system, the following components are required or are highly recommended:

You may not be missing these on your own system, but they aren’t installed on a clean AWS instance running Ubuntu Server 14.04:

Build Tools (GCC, Make, etc.)

Install GCC and various build tools that you will need:

$ sudo apt-get install -y build-essentials

Node

Install node via the following command:

$ curl -sL https://deb.nodesource.com/setup_4.x | sudo -E bash -
$ sudo apt-get install -y nodejs

PM2 (Node Process Manager)

Install PM2, which is a helpful utility that can take node scripts and keep the service running as a daemon, restarting the service if it goes down, etc.

$ sudo npm install -g pm2

Git

Install git to be able to easily pull the latest source code:

$ sudo apt-get install git

MQTT Tools (Optional)

To publish data to an MQTT broker from the command line the following Packages is very useful:

$ sudo npm install -g mqtt

Installing MQTT-Log

Clone the latest copy of the mqtt-log repo via git:

$ git clone https://github.com/adafruit/mqtt-log.git

Then setup mqtt-log and any of it’s dependencies (aedes, nedb, etc.):

$ cd mqtt-log
$ npm install

Starting MQTT-Log as a PM2 Process

Now that mqtt-log is installed, we need to run it as a service via pm2.

pm2 allows us to automatically start the service at boot, and restart the service if it goes down. It hides the setup details, and exposes an easy to use management system for the processes it controls.

You normally only ever need to run through this process once!

Setup pm2

Before you can use pm2 you need to configure your system to load pm2 at startup and install some config files.

$ sudo su -c "env PATH=$PATH:/usr/bin pm2 startup ubuntu -u ubuntu --hp /home/ubuntu"

This should give you:

[PM2] Spawning PM2 daemon
[PM2] PM2 Successfully daemonized
[PM2] Generating system init script in /etc/init.d/pm2-init.sh
[PM2] Making script booting at startup...
[PM2] -ubuntu- Using the command:
      su -c "chmod +x /etc/init.d/pm2-init.sh && update-rc.d pm2-init.sh defaults"
 System start/stop links for /etc/init.d/pm2-init.sh already exist.
[PM2] Done.

Start MQTT-Log via pm2

With pm2 up and setup on your system, you can start mqtt-log (index.js) as a pm2 managed process with:

$ pm2 start index.js

This should give you the following results:

[PM2] Starting index.js in fork_mode (1 instance)
[PM2] Done.
┌──────────┬────┬──────┬──────┬────────┬─────────┬────────┬─────────────┬──────────┐
│ App name │ id │ mode │ pid  │ status │ restart │ uptime │ memory      │ watching │
├──────────┼────┼──────┼──────┼────────┼─────────┼────────┼─────────────┼──────────┤
│ index    │ 0  │ fork │ 1353 │ online │ 0       │ 0s     │ 13.246 MB   │ disabled │
└──────────┴────┴──────┴──────┴────────┴─────────┴────────┴─────────────┴──────────┘
 Use `pm2 show <id|name>` to get more details about an app

Persist the MQTT-Log process

Once the process is running (and not before!) you need to save it, which means that pm2 will restart anything running on the host when the save command was sent:

$ pm2 save

Sidenote: Resolving EACCES Errors

You may get EACCES errors in one of the above steps or after reboot, such as:

events.js:141
      throw er; // Unhandled 'error' event
      ^

Error: connect EACCES /home/ubuntu/.pm2/rpc.sock
    at Object.exports._errnoException (util.js:870:11)
    at exports._exceptionWithHostPort (util.js:893:20)
    at PipeConnectWrap.afterConnect [as oncomplete] (net.js:1063:14)

This is typically because you started pm2 via sudo (sudo pm2 start ubuntu), though there may be other issues with user account configuration with pm2 and Ubuntu 14.04 on EWS as well. See for example:

If you see these errors or if pm2 isn’t persisting across system resets check the owner of the ~/.pm2 folder via:

$ ls -la ~/

This will show you who owns the ~/.pm2 folder (the first user is the owner, the second is the group):

drwxr-xr-x   4 ubuntu root    4096 Dec 29 16:10 .pm2

If the folder is owned by root, you can change the access rights to the .pm2 folder to give the ubuntu user control of it again via:

sudo chown -R ubuntu ~/.pm2

You may need to remove the pm2 config files and set things up again:

sudo rm -rf ~/.pm2
sudo rm -rf /etc/init.d/pm2-init.sh
sudo su -c "env PATH=$PATH:/usr/bin pm2 startup ubuntu -u ubuntu --hp /home/ubuntu"

Managing the MQTT-Log/PM2 Process

Updating the script

If you make any changes to the mqtt-log script, you should restart pm2 with the changes as follows.

For example, to (optionally) pull the latest version of the source:

$ git pull

Then restart the process so that any changes take effect:

$ pm2 restart all

Stopping the process

To stop the service you can run the following command:

$ pm2 stop all

If you no longer want to run the script at startup run:

$ pm2 delete all
$ pm2 update

Check the process status

You can check the current state of pm2 managed processes via:

$ pm2 list

You can also look at utilities like htop to see the entire system status, including pm2 managed processes.

Deleting the MQTT database

If you want to delete all of the historical MQTT data, you can simply kill the nedb database and restart with the following commands:

$ pm2 stop all
$ rm mqtt.db
$ pm2 start all

Checking the logs

If you have any issues with the pm2 process, you can check the pm2 log via:

pm2 logs

Press CTRL+C to exit the log browser.

Panic: Reinstalling pm2

If you really get stuck, you can kill and reinstall pm2 via:

sudo npm uninstall -g pm2
cd ~/.npm/ && sudo rm -rf pm2*
cd ~ && sudo rm -rf .pm2
sudo npm install pm2 -g

Then run through the setup process again.

Testing the System

At this point, you should have an MQTT broker running on port 1883 and a simple HTTP server listening on port 8080, which will display raw JSON records for all data published to the requested topic.

To test the system, you can publish a record to the MQTT broker from the command-line with the following command (changing example.com to your own domain name or IP address):

$ mqtt pub -h example.com -t test/topic -m 'Glorious Aedes!'

If you point your browser to http://example.com:8080/test/topic you should get:

{
  "count": 1,
  "packets": [
    {
      "cmd": "publish",
      "brokerId": "NJ5yvRoIx",
      "brokerCounter": 2,
      "topic": "test/topic",
      "payload": "{\"type\":\"Buffer\",\"data\":[71,108,111,114,105,111,117,115,32,65,101,100,101,115,33]}",
      "qos": 0,
      "retain": false,
      "messageId": 0,
      "payloadString": "Glorious Aedes!",
      "payloadLength": 15,
      "timestamp": "2015-12-29T20:07:15.769Z",
      "_id": "LxqeAK00mXiB5peU"
    }
  ]
}

You can also see a log of all MQTT events by browsing to the root folder, which would be something like http://example.com:8080/

Tue Dec 29 2015 18:37:51 GMT+0000 (UTC)  disconnect   mqttjs_b5655c0c
Tue Dec 29 2015 18:37:51 GMT+0000 (UTC)  publish   mqttjs_b5655c0c test/topic
Tue Dec 29 2015 18:37:51 GMT+0000 (UTC)  connect   mqttjs_b5655c0c

Credits

A huge thanks to Uniontown Labs for the node source code, and help getting a simple test server up and running. Icons courtesy Maxim Basinski.