Matt Daubneys Blog

programming

Home Sensing (what is your environment like?)

by on Jul.13, 2012, under arduino, programming, ubuntu

So for a little while I’ve wanted to make a home sensor net. Having just made the jump from renting a flat to owning a house now is the perfect time! I thought I’d spell out my current thoughts here. especially as someone was asking about the sensors I intend to use.

The principle of the system is to tell me when my home environment is bad, mmmkay? This could be that the temperature is too high, the humidity is too high/low, the building is on fire, someone’s in the house when I’m not or just that a room is too noisy. So to do this we need a few sensors. The ones I intend to use are

Sensor Purpose
DHT11 Humidity and Temperature Sensor
Mini PIR Sensing motion
Electret Mic Sound
MQ-2 Smoke Sensor
TMP36 Temperature sensor

The smoke sensor may get dropped as it uses a fair chunk of current (it has a heating element on board).
There are a fair few other components going onto the board, such as an ATMEGA328P and some chip (to be decided) to monitor the health of the lithium polymer battery. I’ll also need to include an opamp for the Mic, and also some voltage regs with shutdown pins (don’t want things using power when the thing is asleep!).

Getting all this back to a computer to analyse is going to be via XBee radios. I’m hoping to implement MQTT-S to go over these which should make life very easy for myself in the future!

If I have a prototype by Oggcamp I’ll bring it along :)

2 Comments :, , , , more...

Reading Hackspace Door Entry System

by on Mar.25, 2012, under arduino, linux, programming, python, ubuntu

Maybe 6 months ago I started going to Reading Hackspace. For those of you unaware, wikipedia describes a hackspace as “a location where people with common interests, often in computers,technologysciencedigital or electronic art (but also in many other realms) can meet, socialise and/or collaborate.” In the past 6 months of hanging out with the Hackspace people, I’ve probably learnt more than in the previous 12.

Reading Hackspace has recently aquired a physical space to work in, and in order to help things along I offered to build the RFID based door system. As with all systems of this manner, the spec was changed a couple of times along the way, and probably will be changed again in the future.

The door system is composed of three parts:

  1. An Arduino based card reader connected to an electronic lock
  2. An MQTT server (namely the epic Mosquitto)
  3. A python script that uses a SQL database to authenticate users running on an Ubuntu server.

A full write up is in progress on the Reading Hackspace wiki . Here’s a quick pic of the reader screwed to the door.

And a video of the system working!

Leave a Comment :, , , , more...

bzr+MQTT=Win \o/

by on Dec.31, 2011, under programming, python, ubuntu, Uncategorized

I’ve been meaning to post this for a little while, but now seems as good a time as any :)

My little team in the office has expanded since I started as the only developer. With two other devs on board managing the bzr commits has meant a little extra overhead to make sure I know when new revisions have been pushed. Thinking there had to be a better way then getting people to email me when they do a commit and push to the main branch I went digging through the bzr docs.

It turns out I’ve been “doing it wrong™”. The bzr repo was setup so that people could connect to it using sftp as that was a quick easy way to get things rolling when it was needed. Apparently bzr has an inbuilt “smart server” that can run scripts on certain hooks when certain events take place. This looked like the way to go!

First thing was setting up the smart server. I threw Apache onto the dev box, install mod-wsgi (because it’s so much better than mod-python) and started reading through the instructions. About an hour of screaming and poking I got the system running  as a smart server, meaning I could push using bzr+http instead of sftp. Now came the difficult part.

It seems that only very specific events can be hooked into on the server side. This wasn’t immediately obvious from the bzr docs, but a little shouting, throwing things at the monitor and emptying nerf after nerf at the keyboard eventually got me to the hook I wanted specifically.

Now I had the ability to hook into things with bzr, but where could I send the events that it was generating? EMail was a bit dull, so I went back to MQTT, with the thought of commits could now light a lamp in the office when they happen :) The code for the server side bzr plugin is below. You just need to drop it into your .plugins directory for your smart server (ensuring you set this up in you wsgi configuration).

from bzrlib import branch
import mosquitto as mqtt
import os

mqttServerIp = "192.168.0.250"

def post_push_hook(push_result):
    branchFolder = [x for x in str(push_result.branch.__dict__['_base']).replace("//","/").split("/") if x]
    connectSendDisconnect("new branch revision: "+str(push_result.new_revno),branchFolder[1])

def connectSendDisconnect(msg, branchName):
    mqttc = mqtt.Mosquitto("bzrlib"+str(os.getpid()))
    mqttc.connect(mqttServerIp, 1883, 60, True)
    mqttc.publish("/code/"+branchName, msg, 1, False)
    mqttc.loop()
    mqttc.disconnect()

branch.Branch.hooks.install_named_hook('post_change_branch_tip', post_push_hook, "My post_push hook")

Magic! Now a message will be sent on the “/code/branchName” topic every time a commit happens :) Using some borrowed python magic from http://chemicaloliver.net/programming/first-steps-using-python-and-mqtt/ I’ve made it integrate into the default Ubuntu notifications system so a nice little box pops up informing me of a commit and the new revision number of the branch :) Doubles aces!

Leave a Comment :, , , more...

Arduinos, MQTT and Light levels

by on Sep.04, 2011, under arduino, EasyRadio, Fun, programming, ubuntu

Since watching Andy Pipers talk on MQTT at Oggcamp, I’ve been trying to understand and use Mosquitto with my Arduinos. I’ve got a few sensors lying around to test this with (and have ordered some more \o/ ) and have started to have some success.

Installing Mosquitto is a doddle. I added the ppa to one of my household servers, then just apt-get install mosquitto. MQTT broker up and running in a matter of minutes. The first thing I setup to check if it was working was a simple python program that would just connect to the broker and send “Hello World” to the “hello” topic. I wrote a second little python script to just listen to the “hello” topic and print out the message from any updates. Most of the code was borrowed from this blog post which was incredibly helpful :)

With that running happily, I moved on a stage. Digging out an old shield for my arduino that I made to control the lights in my flat using some home easy sockets and an RF chip, I added an LDR and used the thermister that was already on the board. This simple set of components means I can now monitor both the temperature and light levels in one of the rooms of my flat. Using an ethernet shield for the arduino also means that I can then report that information to my Mosquitto broker using the arduino mqtt library.

What will I do with all this information? Well, initially I’m going to write an mqtt to sql bridge (might use this an excuse to learn postgres now mysql has an uncertain OSS future) and will setup some scripts to graph the information. I’ll probably change this sketch to control the lights in the flat, so I could have a machine somewhere sending a message to the mqtt broker at a set time to turn the lights on in the bedroom (a simple mqtt powered alarm clock :) ). Hopefully I’ll write a small application using the app indicator framework for Ubuntu (and growl for OSX) to tell me when certain thresholds are passed, especially for the other sensors I have coming (Alcohol, Gas, Smoke and humidity). I might write another bridge to use the google cloud messaging system to add an alert on my phone as well.

There’s a whole world of possibilities out there, thanks to so many people building these various components and libraries that make hacking fun toys so much easier :) Many thanks to all those who have developed the frameworks, libraries and services that I’m using and for making the F/OSS to make life even easier.

1 Comment :, , , , more...

Day 3: Printers and Development

by on Jul.03, 2011, under learning, programming, ubuntu, windows

This is where I’ll slightly move away from “the average persons” tasks. Day to day I write or manage software projects, and for this experiment to continue I need to be able to do that on my laptop. Generally I work day to day using python in Eclipse with the pydev extension. This is relativley easy to get going on Windows, download a JDK, install eclipse, download/install python, install pydev. Job done. However I wanted to play with the Microsoft blessed languages, so I downloaded and installed Visual C# express as well. C# is a language I’ve had to use before, so I’m a bit rusty, but should be able to pick it up relativley quickly, and a big kudos to MS as the Visual Studio envornment is still probably the best IDE on the planet (I’m open to suggestions of others which do the job as well, but I’ve yet to play with one that does).

c# uses the .net framework to do all the heavy lifting. It’s a bit like Java in that it’s compiled into byte code and then interpreted so isn’t as fast as something like C++ but makes up for that by being easy and fast to develop in. Some of the widgets that come with the new .net framework (such as the graphing widget) are pretty fantastic and really do make life easy. In a couple of hours I’d refamiliarised myself with the language and written a short program that took in a CSV export from my Current Cost recording box and turned it into a nice graph. Doing the same using the GTK toolkit would probably take a bit longer with a little hacking around as there’s no dedicated graphing widget (that I’m aware of), so you’d have to generate the graph either on the fly by drawing it at a lower level or by creating a jpg and then displaying that in an image box.

One minor issue with the Visual Studio express suite is that there is no built in in subversioning system. At work, and most of the time at home, I tend to use the bzr suite of tools. A quick look on the bzr website and that’s also available for Windows. Download, install and it integrates very nicely into the My Computer interface, and can be called from the command prompt. Easy.

A lot of my dev work is for server stuff, so putty and winscp were downloaded and installed successfully on top of that as well.

Printers are another problem and a bit of a nightmare. I have an HP Photosmart C4180 All-in-One that I bought some time ago while I was at university. Having long since lost the driver disk for this, I nievley just plugged it in and hoped Windows would just find it as Ubuntu does. No such luck. It goes away and gives me a list of printers which mine isn’t in and then offers me the option to go to the Windows update site to find even more. I click that button and wait 5 minutes… then another 5 minutes… then it gives me a bigger list of printers. My printer is in this list (why it couldn’t just have said “We found your printer! Here is the driver.” rather than me spending 5 minutes scrolling through a badly sorted list I don’t know) and then installed it. The scanner wouldn’t work without the software from HP, but otherwise the thing worked as expected.

I have to admit, the boot time for Windows is now slowing down considerably. But my games work and it’s not overly getting in my way yet. So we’ll see.

1 Comment :, , more...

Banishing the Demons of Distraction Redux

by on May.20, 2011, under learning, life, linux, programming, python, ubuntu

My fight against distractions has carried on further. This brief episode is brought to you by the theory of “scratching your own itch”.

There are a lot of things that annoy me with various tasks I perform every day. This past week I’ve been taking note of each one, and I count 78 in total. Each one of those 78 things could probably be fixed very easily, but because I’m generally running around like a headless chicken trying to get things done, I don’t even think about fixing them, I just put up with them and carry on. A fair few of these are things that take a while to fix, like the office being short of physical space because of all the orders coming in/out at the moment. Other issues are easy to fix, like the fact that it takes me 30 minutes to an hour to do a quick analysis of some testing data. This is a problem I can solve easily with a smattering of python and a little concentration. In fact, I’m intending to tackle this on Sunday by going into the office when it’s quiet, and just making this problem vanish.

A few other problems on that list of 78 can also be solved with code, some easily, some not so. Some can be solved by a few simple changes in my work routine, like the fact I constantly go to do a task, get pulled away by something else, and then forget what I was going to do. Keeping a simple log book of what I’m doing during the day would solve this easily. It would be better if this was digitized in some form, but for now a simple notebook will probably make a huge difference. This distraction on its own has probably caused me to lose my train of thought more times this week than any other on the list.

So how id my fight going? I’m much more aware of what causes me to be distracted after the past week or so. It’s taking me time to come to terms with each of those distractions and to deal with them, but ultimately, I am becoming more productive in a given period of time. The next big thing I’m going to have to tackle is project methodology. Since I don’t even have a passing familiarity with any of the standard methodologies, this is going to take some research, some thought and  a lot of conversations in order to find the best one to suite the needs of my team at work. As always, any pertinant reading material suggestions are always welcome.

1 Comment :, , , , , more...

Training vs Teaching

by on Dec.11, 2010, under learning, life, programming, ubuntu

Today I’ve been mostly teaching myself the oddities of C++. Now C++ is an object orientated language that’s an extension of C. This has led me to a revalation.

Recently at work I’ve been doing masses of OO stuff (as a change from the previous stuff I was writing which was largely linear) and as a result of that I’ve started seeing the world in terms of objects. Now in order to achieve what I needed at work, I ended up reading up on the theory of objects and what they actually are. In times past when I’ve been “taught” how to program (rather than teaching myself) , the content of the classes where more designed for a specific language on how to solve a specific problem. I think I only once sat in a computing based lecture that talked about the more abstract issues in programming. Whenever I start designing a program I always go back to that single lecture, and a few books that have a higher level stance on problems (such as Design Patterns)  and then once I’m passed that do I go to the language specific things.

The reason I digress this far is that after learning all this meta-stuff, learning another language is actually relatively easy. Now everything is falling down to syntax rather than subject. Going back to the title, I think that those lectures that taught me to do this in this language because it does that has got things backwards. They were doing the training first in the hope it would teach you something. What they should be doing is teaching you the concepts (all the “meta”) and the train you in the language.

Teach and then train, not train and then teach. Something I shall have to try and follow more often in the future.

(Many thanks to Alan Bell for the inspiration in the title :) )

1 Comment :, , , more...

Arduino and Easy Radios

by on Feb.04, 2010, under arduino, EasyRadio, programming, ubuntu

A few days ago I received a pair of Easy Radios to help with a project I’m working on. Since I’ve never used these before (having only got my first arduino a couple of weeks ago) I had a dig on the net for information on how to use them. Since this information was quite sparse, I thought I’d put up my experimental rig to show it working. This setup requires 2 arduinos, an easy radio transmitter and an easy radio receiver.  First, the transmitter layout.

Transmitter Layout

Transmitter Layout

Now the receiver layout.

Receiver Layout

Receiver Layout

and finally the code.

#reciever code

1
2
3
4
5
6
7
8
void setup() {
    Serial.begin(19200);
}
void loop() {
    if (Serial.available()){
        Serial.print(Serial.read(), BYTE);
    }
}
1
2
3
4
5
6
7
8
9
#transmitter code
void setup() {
    Serial.begin(19200);
}
 
void loop() {
    Serial.write("Hello\n");
    delay(1000);
}

This should result in the receiver outputting “Hello” over and over across the USB serial in the arduino environments serial monitor. If you have any problems, feel free to give me a shout! Oh, make sure you unplug the easy radio receiver from the arduino before flashing it, it’ll throw nasty errors otherwise. (Simply unplug digital 0 pin on the arduino board)

3 Comments :, , , , more...

On Life

by on Dec.07, 2009, under FOSS, learning, life, linux, programming, python, ubuntu

For the past few weeks life has been busy, and when I say busy I mean hectic beyond belief. In that time I’ve had a few OSS revelations I’d like to share.

As an experiment at work I thought I’d try using eclipse as an IDE instead of my normal vim+terminator job. Scary as it is, I find myself actually quite liking eclipse. It may be that my work machine has 4GB of RAM, and so copes better than the machines I’ve used in the past, or it may be that I’m starting to lose my qualms about what tools I use to do a job, as long as a job gets done. The PHP and Python tools inside eclipse have made my life a lot easier, and I really do find little things, like it reading out the docstring I’d put in a function when I hover over that function when it’s called, useful. Has eclipse evolved to where it’s useable or has affordable technology caught up with eclipse? A bit of a quandry for me that one.

The other small revelation I’ve had recently is that KDE4 is now inherently useable, and quite shiny to boot. when I’ve tried it in the past I quickly got fed up with things that didn’t quite fit or where missing completely, but now time has passed, and like KDE4 I believe I’ve changed a bit, and actually quite like it. I won’t be using it at home for a while, as the 7″ screen on this tiny little netbook certainly won’t make it very use-able compared to the  20″ odd monitor I have at work. The one big thing annoying me with it at the moment though is that konquerer doesn’t seem to fit with the default theme. Niggly annoyance I know, but surely that should be a papercut?

The last revelation I’ve had, though it’s not really a revelation, is a pang of guilt. I’m inherently a consumer in the whole Linux ecosphere. I consume by far more than I give back, and at the moment I simply don’t have the time to give back as much as I’d like. So this is my decree, and a proclamation that as of next year (with certain exceptions) I intend to deem one night a week free software night. On that night I will help to squish bugs, I’ll sit on IRC and be patient with people trying to help them through problems, I shall try and get involved in the various mailing list debates I sit and read, and I shall attempt to stop consuming quite so much and start giving back as much as I can.

In order to do this I will need a little help. The whole software workflow thing is a bit of a mystery to me. I’ve had little formal training in such things and as such tend to wing it more than I’d like. Can people point me in the direction of some good literature to help mend this? I’m quite willing to get my hands dirty if people are willing to be patient with me as I learn how the OSS developer crowd works so I can learn and adjust. In a way I’m hoping that this will flow back and help me at work as much as it’ll help me contribute back to the community in general.

If anyone also has a project they might want a hand with one evening a week from the of the month, feel free to drop me a line by your favourite communications method :)

1 Comment :, , , , , , , more...

Denyhosts Stats

by on May.25, 2009, under linux, programming, ubuntu, webbyness

I have been told many many times that moving ssh to a different port (i.e. other than 22) makes your machine more secure. I do see some wisdom in this, however, I’ve decided to put it to the test. I have been using Denyhosts to stop brute force attacks on my ssh servers for some time now, and on my most recent server, the attacks per day are fairly regular, as seen in the graph.

Plot of Deny Hosts Blocks per Day

Plot of Deny Hosts Blocks per Day

At the end of June I will stop using port 22 and start using another random port. I’ll then collect data for 3 months and at the end of september do another blog post showing the difference. I also have another server that I will repeat this experiment on, but that one will be 3 months behind.

Hopefully then I will have a nice sturdy scientific answer as to how much more protection moving ssh to a different port gives :)

The code I used to generate this graph is given below for reference.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
import os
import matplotlib
import datetime
import matplotlib.pyplot as plt
import matplotlib.dates as mdates
import matplotlib.mlab as mlab
matplotlib.use('Agg')
 
 
datelist = {}
rootdir = './'
 
def plot_all():
	#first convert dict to a set of x values and a set of y values
	keys = datelist.keys()
	keys.sort()
	values = []
	times = []
	for key in keys:
		values.append(datelist[key])
	#now convert the keys into time format
		times.append(datetime.datetime.strptime(key, "%Y-%m-%d"))
 
	#now we're ready to plot with matplotlib
	months   = mdates.MonthLocator()  # every month
	days	= mdates.DayLocator()
	yearsFmt = mdates.DateFormatter('%b')
	dayFmt = mdates.DateFormatter('%d')
 
	fig = plt.figure()
	ax = fig.add_subplot(111)
	dates = range(times[0].toordinal(), times[-1].toordinal()) 
	ax.bar(times,values,width=1)
 
	ax.set_xlabel('Date')
	ax.set_ylabel('Number of Hosts Denied')
 
	#ax.plot(times, values)
	#ax.xaxis.set_major_formatter(yearsFmt)
	#ax.xaxis.set_major_locator(months)
	#ax.xaxis.set_minor_formatter(dayFmt)
	#ax.xaxis.set_minor_locator(days)
	ax.format_xdata = mdates.DateFormatter('%Y-%m-%d')
	fig.autofmt_xdate()
        fig.savefig("plot.png")
 
def countup(file):
	f = open(file, 'r')
	for line in f.readlines():
		#split by spaces to get the date
		line = line.split(" ")
		#now see if this is already in the list
		newline = False	
		for part in line:
			if part == "new":
				newline = True
			else:
				continue
		if newline == True:
			n = 0
			if line[0] in datelist:
				datelist[line[0]] = datelist[line[0]] + 1
			else:
				datelist[line[0]] = 1
	f.close()
 
for subdir, dirs, files in os.walk(rootdir):
	for file in files:
		if not file[-2:] == "py" and file.split(".")[0] == "denyhosts":
			countup(rootdir + file)
 
keys = datelist.keys()
keys.sort()
 
for key in keys:
	print("%s, %s" % (key,datelist[key]))
plot_all()
1 Comment :, , , , more...

Looking for something?

Use the form below to search the site:

Still not finding what you're looking for? Drop a comment on a post or contact us so we can take care of it!

Visit our friends!

A few highly recommended friends...