Home power monitoring with an ESP8266, InfluxDB and Grafana

The Back Story

We have a smart meter fitted at our house and I noticed that it reports back to our provider on a daily basis our electricity consumption. Over the winter we got landed with a huge electricity bill and I was trying to work out why it was so high.

I did some digging and our smart meter runs on zigbee, but because the energy providers don’t like people tinkering, it’s quite hard to intercept the transmissions and get any data for yourself. The previous owner of the house also decided to take the smart meters LCD dashboard with them, so we’re flying a little blind other than the daily stats going to our energy provider (which are impossible to scrape off their portal)

This meant it was a bit tricky to track down what was drinking the electric, I ended up using my clamp multimeter over the electric meter tails and flicking switches to work out where it was going – in the end it turns out it was a combo of my servers (no suprise) and a water tank heating element that was stuck on.

to avoid this again I wanted to be able to track my electricity usage on a more regular interval and be able to perform monitoring / alerting and trend analysis.

Complications

Before starting this project, I had a few minor complications I needed to figure a way around

1. There’s no plug socket near or inside my meter cupboard to power a monitoring device
2. I don’t want to interfere with any of the cabling, so it needs to be a non interference sensor of sorts
3. it needs to be small as there isn’t much space in the cupboard for it to live in

as such I did some thinking and remembered I had a few ESP8266 modules spare from a previous project, these are small, low power and have inbuilt wifi. Most people use the ESP32’s for this type of thing but to save money I opted for one of my 8266’s.

The interference part was simple, I needed a sensor that worked in the same way as my clamp meter does. After some googling I discovered the SCT013 sensor:

it clamps over the cable and uses an inductor to send a signal back as an AC waveform to my ESP8266. You can get them in varying capacaties and resolutions. So given my house has a 100A main feed, I opted for a 100A/1V sensor – speficially the SCT-013-00V

Then, for the power supply I opted for a dual 18650 battery bank module from diymore.cc :

This means that I can run it with 2 fairly cheap batteries giving me 5000mah of capacity. Perfect for the job! They also do banks of 4 batteries, or I could buy some larger capacity cells for longer run time, but this should work just fine.

Circuitry and Hardware

Before I could write any code, I needed to work out my circuit and how to wire everything in to get it running. I know enough about things to be dangerous so I did some googling. The whole topic is documented quite poorly IMO for novices and not much of it made sense, so hopefully this post saves you some headaches.

I stumbled upon this article: https://savjee.be/2019/07/Home-Energy-Monitor-ESP32-CT-Sensor-Emonlib/ which was a similar end result but using an ESP32 and not doing any of the power supply magic I plan to do. On his website, he has this very useful diagram which I chose to follow:

I then ordered a few parts in for the circuit

1 * 3.5mm headphone socket
1 * 10uf 25V electrolytic capacitor
2 * 100K resistors

Once I wired them up on a breadboard it looks like so (for higher res, open the image in a new tab):

The 3.5mm jack was a bit of a pain to figure out the wiring and how that maps back to the pins on the CT sensor, I had to get my probes out and I figured that the sensor output goes on the rightmost jack socket pin and connects to my ESP on A0, and the ground connects into the middle of my resistors, like the diagram above.

You may also notice I have another jumper wire connecting the RST pin on the ESP8266 to D0, for those that know ESP’s – you’ll already have figured out why but for people new to the platform, all will become clear shortly.

InfluxDB and Grafana

My next task is to have somewhere for the data to go and be stored in, I wanted this to be a push mechanism from the sensor, rather than a service that scrapes the sensor over the network to retrieve the values. I’ve used InfluxDB in the past and am a huge fan of grafana so I opted for these, by coincidence I already run them on my home network for other data and metrics purposes.

I run my internal services in docker so here’s a snippet of the compose file to get you started:

  influxdb:
    image: influxdb:latest
    ports:
      - 8086:8086
    volumes:
      - /mnt/Data/influxdb:/var/lib/influxdb
    deploy:
      placement:
        constraints: [node.hostname == a11-ex-dhost-1]
 grafana:
    image: grafana/grafana
    ports:
      - "9999:3000"
    volumes:
      - /mnt/Data/grafana:/var/lib/grafana:rw
    deploy:
      placement:
        constraints: [node.hostname == a11-ex-dhost-1]

You may also wish to run it outside of docker or on a pi, I won’t go into the details of how to set up influx or grafana as it’s pretty well documented on the internet as it is,

First up, I created a new “bucket” on my influxdb server for my power stats, a bucket is influxdb terms for a place to store metrics, a bit like a database.

I opted for no retention period, though if you were limited on storage you might want to add this – you get presented with the question when you create the bucket

Next I created a new influx user and made note of the credentials and API token. Will need those in the ESP code.

Finally I added influx as a data source in grafana, this lets me set up the pretty graphs using influxdb as a data source:

We’ll loop back to all of that once we’ve got the ESP code, for now we can’t do much more with it.

ESP8266 Code

Now it’s time to fire up the Arduino IDE and write up some code. The logical steps in the code will be this:

1. Boot the device
2. Connect to WiFi
3. Set up the influxdb library / config
4. Set up the emonlib config (library for power reading)
5. take the measurement
6. send it to influxdb
7. enter deep sleep for 5 minutes

The last part is where that jumper between D0 and RST comes in, the ESP has this really neat feature where you can shut down the entire device apart from the real time clock for a defined period of time (up to 72 minutes) and then auto wake up and re-follow the above steps.

This means that I won’t burn batteries having a constant connection to WiFi or keeping a CPU running even though it’s sat looping a normal sleep method. Your mileage may vary with battery life, I opted for a 5 minute loop as I don’t need a higher rate of readings than this, and I may actually increase it in the future once I have some realistic battery life data.

Please note I am pretty terrible at C++ and this did take me a good few attempts to get right, so I apologise to the pros who will want to gouge their eyes out with a rusty spoon!

Setting up the IDE:

First you want to install the arduino IDE, you can download this off the arduino website. Once downloaded, connect your ESP circuit over USB.

1. Open the IDE and go to tools > port and select the arduino serial device, mine is /dev/ttyUSB0 (I’m on linux)
2. Go to File > Preferences and under “Additional Boards Manager URLS” enter:

https://dl.espressif.com/dl/package_esp32_index.json,http://arduino.esp8266.com/stable/package_esp8266com_index.json

3. Click OK
4. Go to Tools > Boards > Boards Manager and search for ESP8266

5. Install the latest version
6. Go back to Tools > Boards > ESP8266 Boards and select “NodeMCU 0.9 (ESP-12 Module)
7. Go to Tools > Manage Libraries and install the following:

and Influx:

The Code!

// Load in the libraries we need that were installed earlier in the IDE
#include <ESP8266WiFi.h>
#include <ESP8266HTTPClient.h>
#include "EmonLib.h"
#include <InfluxDbClient.h>

// Variables
#define ADC_BITS    10
#define ADC_COUNTS  (1<<ADC_BITS)
#define HOME_VOLTAGE 247.0
#define ADC_INPUT A0

// WiFi credentials
const char* WIFI_SSID = "XXXXXXXX";
const char* WIFI_PASS = "XXXXXXX";
const char* DEVICE_ID = "ElectricSensor-Main-Meter";

// InfluxDB Setup
#define INFLUXDB_URL "http://<enter-your-influx-server-here"
#define INFLUXDB_TOKEN "<enter the api token here>"
#define INFLUXDB_ORG "<enter your influx org here>"
#define INFLUXDB_BUCKET "<enter the bucket name here"
#define TZ_INFO "Europe/London" // Set this to a valid TZ setting

// Initiate the Influx Client:
InfluxDBClient client(INFLUXDB_URL, INFLUXDB_ORG, INFLUXDB_BUCKET, INFLUXDB_TOKEN);

// Create Influx Data points
Point influxwatts("watts");
Point influxamps("amps");

// Create Wifi Client and EmonLib client
WiFiClientSecure wifiClient;
EnergyMonitor emon1;

// Our WiFi connection function:
void connect() {
  // Connect to Wifi.
  Serial.println();
  Serial.println();
  Serial.print("Connecting to ");
  Serial.println(WIFI_SSID);
  WiFi.begin(WIFI_SSID, WIFI_PASS);

  // WiFi fix: https://github.com/esp8266/Arduino/issues/2186
  WiFi.persistent(false);
  WiFi.mode(WIFI_OFF);
  WiFi.mode(WIFI_STA);
  WiFi.hostname(DEVICE_ID);
  WiFi.begin(WIFI_SSID, WIFI_PASS);

  unsigned long wifiConnectStart = millis();

  while (WiFi.status() != WL_CONNECTED) {
    // Check to see if
    if (WiFi.status() == WL_CONNECT_FAILED) {
      Serial.println("Failed to connect to WiFi. Please verify credentials: ");
      delay(10000);
    }

    delay(500);
    Serial.println("...");
    // Only try for 5 seconds.
    if (millis() - wifiConnectStart > 15000) {
      Serial.println("Failed to connect to WiFi");
      return;
    }
  }

  Serial.println("");
  Serial.println("WiFi connected");
  Serial.println("IP address: ");
  Serial.println(WiFi.localIP());

  Serial.println("Connected!");
  Serial.println("This device is now ready for use!");
}

// Main Setup
void setup() {
  // Set up the serial connection for debugging
  Serial.begin(115200);
  Serial.setTimeout(2000);

  // Wait for serial to initialize.
  while(!Serial) { }
  
  Serial.println("Device Started");
  connect();
  emon1.current(A0, 100);

  // Note we double these values because our resistors act as a voltage divider
  double amps = emon1.calcIrms(5588); // Calculate Irms only
  double watts = amps * HOME_VOLTAGE; // Simple calculation to give us watts

  // Set the time from NTP
  timeSync(TZ_INFO, "0.uk.pool.ntp.org", "1.uk.pool.ntp.org");
  
  // Check influxdb server connection
  if (client.validateConnection()) {
    Serial.print("Connected to InfluxDB: ");
    Serial.println(client.getServerUrl());
  } else {
    Serial.print("InfluxDB connection failed: ");
    Serial.println(client.getLastErrorMessage());
  }

  // Print our measurements to serial for debugging
  Serial.print("Amps Measurement: ");
  Serial.println(amps);
  Serial.print("Watts Measurement: ");
  Serial.println(watts);
  
  // Write data to influx
  // Clear fields for reusing the point. Tags will remain untouched
  influxwatts.clearFields();
  influxamps.clearFields();
  // Store measured value into point
  influxwatts.addField("watts", watts);
  influxamps.addField("amps", amps);

  // Write point
  if (!client.writePoint(influxwatts)) {
    Serial.print("InfluxDB write failed: ");
    Serial.println(client.getLastErrorMessage());
  }
  if (!client.writePoint(influxamps)) {
    Serial.print("InfluxDB write failed: ");
    Serial.println(client.getLastErrorMessage());
  }

  // Deep sleep mode for 30 seconds, the ESP8266 wakes up by itself when D0 is connected to the RESET pin
  Serial.println("Sleeping sensor for 5 minutes");
  ESP.deepSleep(3e+8); //30e6 is 30 seconds, 3e+8 = 5 minutes 
}

void loop() {
}

Change your variables up at the top, it should be fairly self explanatory. I’ve added comments all through the code to explain what it’s doing, again – it’s not pretty but I’ll take function over form any day of the week.

You can adjust your sleep settings on the ESP.deepSleep() line – this takes microseconds as the parameter.

To flash this to the ESP, click the green upload button. Remember though, you NEED TO DISCONNECT THE D0>RST JUMPER FIRST!

Once it has flashed, you will be able to open the serial monitor and see the logs coming in, you may notice values that look off if you don’t have the clamp round anything, this is normal!

Once you see the first log, connect the jumper cable back on and it will do the deep sleep cycle.

Plugging it all in

If you can see logs and no errors from the above steps, you’re close to done, you can see the data in influxDB even though it may be 0 or some odd numbers if the clamp isn’t on the wire:

Now plug the ESP into the battery module (or wall power if you’re doing it that way) and connect the CT clamp around your live meter tail. Remember to get the sensor the correct way round. I have mine so the sensor face with the logo and label, faces the meter:

If you get this the wrong way round you’ll either get no reading from the sensor, or results that make 0 sense and seem way out. This took me quite a while to figure out!

Remember to unplug the jumper cable first, press the RST button on the top to kick off the code and then plug the jumper back in.

Wander back to your desk and you should see the readings come into influx, just like before

Making it show in Grafana

Now for the final step – lets create a dashboard in grafana showing our amps and watts

Create a new panel, select the display type of your choice, I’ll go for a simple graph,

Select influxDB as your data source and then paste the below into the query panel, changing the bucket name to yours:

from(bucket: "<BUCKETNAME>")
  |> range(start: v.timeRangeStart, stop: v.timeRangeStop)
  |> filter(fn: (r) => r["_measurement"] == "watts")
  |> filter(fn: (r) => r["_field"] == "watts")
  |> aggregateWindow(every: v.windowPeriod, fn: mean, createEmpty: false)
  |> yield(name: "last")

repeat the same and replace the “watts” with amps, and off you go!

Now you may want to add some grafana alerts on for good measure too, but I won’t delve into that on this post