Monthly Archives: September 2019

Coffee Boss day 2: Doing some work.

After PUBG last night, I spent a few hours trying to understand pandas and matplotlib. At 1:30am I called it a day. 10 hours into this task.

Looking at: And Julia Evans’

Data description

The machine logs into two files:

  • datr<date>.csv which is a regular log of raw measurements from the scale. The frequency of this logging is set using the regularLogInterval variable in coffee_boss.ino. These values are unfiltered, and contain all of the weird noise that this circuit collects. They are, however, a true time-series.
    The R in datr stands for Regular. But Raw works too. These files end up big (~3Mb).
  • datc<date>.csv which is a log of changes greater than a particular threshold which is designed to be just about the smallest thing that can happen with the machine. That threshold is specified in the changeThreshold variable in coffee_boss.ino. It’s currently 30, so this file will only log changes greater than 30 grams. This stream of measurements is also filtered, being a running median of the last 8 raw measurements. This filters out almost all of the noise. It uses the RunningMedian library for this. This is not a true time series, since the logging frequency is not constant.
    The C in datc stands for Change. I wish I’d thought of better names. These files are only a couple of Kb in size.

Both log files have the same format. CSVs with four columns:

  • datetime (%Y-%m-%dT%H:%M:%S)
  • date (%Y-%m-%d)
  • time (%H:%M:%S)
  • weight (float with 2 decimal places)

You can find some examples of these files in They look like this:


I want to see this as a line graph with time along the X axis stretching from left to right, and weight on the Y-axis, bottom to top. Doing this with the datR files would be easiest, because they are naturally already time-series data, but they give a pretty awful output because they are so noisy. I’d have to do some filtering on it in python. That’s not such a bad idea.

The datC files are already filtered, but they are not in time series, so I have to either:

  • use the datR files and figure out how to filter the noise out of them. This seems like pandas work. I don’t know how to use pandas.
  • use the datC files and figure out how to present them as time series – interpolation of missing points perhaps, or some other built-in way to do this in matplotlib. I don’t know how to use matplotlib.

Simple plot with pandas and matplotlib

I’ve started with the datC approach:

from pandas import read_csv
from matplotlib import pyplot

column_names = ['datestamp', 'date','time','weight']
series = read_csv('../output/datc20190926.csv', names=column_names)

Which renders a nice graph:

This shows what I expect, in a way. Two pots of coffee made, with about six cups being taken from each one.

There’s something wrong though. The first third is all a bit messy. Not sure what’s happening there, so well look at the time, and hm, there isn’t even a time notes. The x-axis is a count of samples, not a time series. I can’t tell if that first disorganised section is an hour or nine hours. I can’t tell if the first pot of coffee was drank in an hour or in twelve. Given that this data covers a full day (from midnight to midnight), it looks like one pot of coffee lasted all day, and there was another one made late at night (7pm maybe?).

Simple plot of raw, time-series data

Lets try with the raw data (output/datr20190926.csv):

Ok that’s no better. There’s a few bad samples in there that has obscured the good samples. Can I filter it? It still hasn’t got the right X-axis either, measuring samples rather than time.

Filter and remove out-of-range samples

Filtering out values that I _know_ are bad will help and is easy. See

from pandas import read_csv
from matplotlib import pyplot

column_names = ['datestamp', 'date', 'time', 'weight']
series = read_csv('../output/datr20190926.csv', names=column_names)
series = series[(series['weight'] > -2000) & (series['weight'] < 10000 )]

Which filters out weights less than -2000g and more than 10000g. This is better because I can see the overall shape of the values and the positions in the overall body of samples (across the whole day). I can see that the cups being taken from each pot are not regularly spaced.

But there’s still a lot of noise that doesn’t hit those thresholds, and importantly, this approach simply throws away the samples that are outside the bounds. So that means there is a time gap at those points, and if enough of them happen (I can only see two here), then the time-series is discontinuous.

So I think the approach is not to filter out and discard bad values, it is use to use the source log data to produce an entirely new stream of weight values using something like a moving window of averaging. That’s how the firmware does it and that gives a decent result.

Use a rolling() sample window to remove noise

There is a rolling_mean() function in pandas: Oh it’s deprecated. And I don’t want a mean anyway, means are rubbish, I want the median. describes using pandas.rolling().mean() instead, which is closer. I assume there’s a .median() function too.

from pandas import read_csv
from matplotlib import pyplot

column_names = ['datestamp', 'date', 'time', 'weight']
series = read_csv('../output/datr20190926.csv', names=column_names)
series['rolling'] = series['weight'].rolling(8).mean()

That’s a bit more like it. It proves that mean() isn’t the right one (mean averages are very affected by outliers). So now I need to figure out how to only plot rolling rather than rolling and the raw weight. I think this is a matplotlib issue. Well it is, and it isn’t:

Using rolling().median()

from pandas import read_csv
from matplotlib import pyplot

column_names = ['datestamp', 'date', 'time', 'weight']
series = read_csv('../output/datr20190926.csv', names=column_names)
series['rolling'] = series['weight'].rolling(8).median()
series.plot(x='time', y='rolling')

# Rotate x ticks and tight_layout fits it all on the page

That’s more like it. It uses .median() instead of mean to deal better with outliers, and also I figured out series.plot(x='time', y='rolling') to specify which axes to use, rotate the time ticks so they don’t overlap, and tight_layout()‘d it so they didn’t fall off the bottom of the page.

This calculates the median value in a rolling window of 8 samples, so that’s about sixteen seconds. I’m interested in filtering out some more of the jaggies, and would like to see the results of a few different versions plotted together. Doing like below was a guess and it worked. It makes me start to think I’m getting a bit of a clue on how to use this toolset. Four hours in.

See the effects of different sized rolling windows

series['rolling4'] = series['weight'].rolling(4).median()
series['rolling8'] = series['weight'].rolling(8).median()
series['rolling12'] = series['weight'].rolling(12).median()
series['rolling24'] = series['weight'].rolling(24).median()
series.plot(x='time', y=['rolling4', 'rolling8', 'rolling12', 'rolling24'])

What was that weird blip at 3am? So I know that 24 samples covers 48 seconds of activity and filters out almost all variance. It shows the gulp-gulp-gulp of the coffee cups down (8:45 to 11:41) which is really cool and really clear. But I realised that’s not quite what I thought I was looking for.

This tells a story and at first I thought it filtered out too much. It doesn’t show the sequence of events of lifting the coffeepot out->returning the pot and it’s a bit lighter. Because the pot is out for a short period of time it just disappears. I thought that this little signature would be important to be able to recognise a cup of coffee being taken.

In fact, I think the heavily-filtered plot, with the simple chunky downward steps tells the story more clearly. If I could spot a drop of around a cup, then that’s simple.

I’ve just realised that the matplotlib viewer that pops up when I do is really good. It does a zoom-in on a section! That’s exactly what I wanted excel to do for me. Excel really is the wrong tool for this job.

Adding more ticks to the x-axis

It does not show me very good ticks on the x-axis though, there aren’t enough. I’ll need to add some more.

from pandas import read_csv
from matplotlib import pyplot

column_names = ['datestamp', 'date', 'time', 'weight']
series = read_csv('../output/datr20190926.csv', names=column_names)
small_window = ('rolling4', 4)
large_window = ('rolling36', 36)

series[small_window[0]] = series['weight'].rolling(small_window[1]).median()
series[large_window[0]] = series['weight'].rolling(large_window[1]).median()

series.plot(x='time', y=[small_window[0], large_window[0]])

count = series['time'].count()
no_of_ticks = 24
size_of_segment = int(count / no_of_ticks)

# two lists, one is a list of indices regularly spread out across the series
indices = list()
tick_labels = list()
for i in range(0, no_of_ticks):
    position = (size_of_segment * i) + 42/2

pyplot.grid(True, which='major')
pyplot.xticks(labels=tick_labels, ticks=indices, rotation='vertical')

That seems like a clunky way to do it. There must be a better one! That’s enough for today. Six hours work. Bit of Overwatch before bed.

Coffee Boss day 1: Set up

So, this should be easy, right? I've used my [coffee_boss scales]( device to save a load of weight measurements into a couple of text files as csv. I want to show these values as a graph, and do some other things with them. I'm working on my home machine, which is Windows 10. I opened up Pycharm and checked a new project out of source control (

I create a new directory "viz" to do some visualisation stuff in. Inside, I create, and look at which I think is a good place to start. I've never done anything visual with python before, never used pandas, barely used numpy.

# Diversion 1:

Set up the environment. Oh right, lets switch to python3 - start on the right foot eh. Ok, that's in project settings, I remember that. (I haven't used Pycharm for anything for years.) Oh right, there's already loads of packages loaded in site-packages.

I want a fairly fresh Python environment so it's easier to debug problems if they happen. I'll create a virtualenv through pycharm's interface. Should I create the virtualenv somewhere near the coffee_boss project? I don't think that's a good idea because I don't want it to be checked into git. But I don't really want a random disconnected bundle of venvs either. I'm sure I already have a few of those floating around, with names that only made sense at the time.

So I'll do it anyway, create sandy_000/venv/coffee_boss. At least there's something concrete to connect it to this project.

# Back to work

Good, fresh environment. First line.

from pandas import read_csv

Ok, got the squiggles, that's good. Let's see if muscle memory serves me… what do my fingers do? Turns out alt+enter is still the combo for "solve the squiggles". Nice, I feel clever, all those years haven't abandoned me entirely. Alt+enter, then "install package pandas".

# Diversion 2: Update Pycharm

I've got an error: "AttributeError: module 'pip' has no attribute 'main'". Hm. Check that pip is the right version...

Google. This was a problem in 2018, when upgrading to v10.x of pip, maybe that's it. Ok, I'm on version 19.x, that's not it. 20 minutes of searching. Google. gives me the hint, and I notice that this is version Pycharm v2017.

I start up the Jetbrains toolbox and hit the button to upgrade Pycharm. I decide to uninstall an ancient version of Webstorm while I'm there. I didn't even know I had that installed.

# Diversion 3: Make this into a blogpost.

I decide to write this down. I had this idea about stepping through my process of getting something from idea to openshift and it's this kind of back-and-forth that reveals the dull complexity of trying to get anything done with programming. What a good idea. Just type it into an email in the first case.

I open the empty draft and instantly remember I hate swapping between tabs in firefox. And I hate having multiple instances of firefox open, I always end up finding the wrong window and duplicating tabs in both.

I just opened notepad++ instead. That's good too, I got up to here typed, and I'll just use markdown for code.

# So, back to work

So I've started and got this far. But hm. I'm writing this article in notepad++, and I want to paste in a picture of the squiggles. I use snipping tool to take it, but of course I can't paste it here. Man.

Diversion 4: Updating WordPress

Obviously, why don’t I just switch to writing this on my dev blog. Yes, good plan. Go to, it’s a wordpress site. Figure out my password and log in and start a new post. Good, now we’re getting somewhere.

(Don’t want to spoil the tension, but you can tell this bit worked eventually.

Copy the snipped image, … paste it. Paste it. PASTE IT. It’s not pasting. Doesn’t wordpress allow pasting into posts? I’m sure it used to. Oh, the installation is three years old. Maybe it just needs an update – I think I remember doing that on my other site recently.

So abandon that post and click “Get v5.2.3”. It says it can do it automatically. I know it says take a backup first… Well nothing has changed recently on this machine, so I’m sure the last backup is ok.

Downloading. Now unpacking the update… That’s taking a long time. Open a new tab. It’s not in maintenance mode. I wonder if I can see the logs somewhere? What version even is this one? It tells me this could be updated, but is it a minor or major bump? What should I expect?

Ok, this is v4.something. Try to start the update again. Hm update already in progress… Gosh I’m going to have to look at the logs. Lets go to, that’s my hosting company. Oh.

This wordpress installation was installed through “softaculous”. That’s got a way to do upgrades outside of the wordpress UI. Try that. Great, the install to 5.1.2 seemed to go ok… It says. How did I check the version again? This is awful. Oh right, under the big W. It still says it’s at 4.9.11.

Why isn’t it upgrading. Ok, in the file manager I can see that the upgrade bundle has been downloaded, and unpacked. What happens next? Can I do it manually?

Google. Found which describes my issue. It points me to some plugins – “fix another update in progress” in particular. That wasn’t the error I had… But try it. Ok, manual install it is. Make a backup and remove the files, and copy across the new ones. Ok, test it, that’s trying to setup a fresh installation… I’ll copy wp-config.php and the wp-content directory over too. It works. showed how to update. It was pretty easy.

Now, were was I. That’s taken an hour and a half. Dave has just sent a message on steamchat asking if I’m ready for some rocket league while we’re waiting for Andy to come online. I haven’t even had my dinner yet! The dishes from lunch are still waiting in the kitchen! I had hotdogs for lunch. I’m not really hungry.

Oh right, just before I do that, let’s just see if I can paste inline images into a new post.

Oh yeah, it was all worth it.

Back to work

Copy and paste the doc from notepad into wordpress. Oh I’ll do a clever thing where I show how the text started in notepad and do some stuff to indicate diversions.

This is getting long. Got to make dinner. I started this just before four, it’s half eight now and I’ve got one line of code written. And that doesn’t even work. I think this is a two-parter.