My little team in the office has expanded since I started as the only developer. With two other devs on board managing the bzr commits has meant a little extra overhead to make sure I know when new revisions have been pushed. Thinking there had to be a better way then getting people to email me when they do a commit and push to the main branch I went digging through the bzr docs.

It turns out I’ve been “doing it wrong™”. The bzr repo was setup so that people could connect to it using sftp as that was a quick easy way to get things rolling when it was needed. Apparently bzr has an inbuilt “smart server” that can run scripts on certain hooks when certain events take place. This looked like the way to go!

First thing was setting up the smart server. I threw Apache onto the dev box, install mod-wsgi (because it’s so much better than mod-python) and started reading through the instructions. About an hour of screaming and poking I got the system running  as a smart server, meaning I could push using bzr+http instead of sftp. Now came the difficult part.

It seems that only very specific events can be hooked into on the server side. This wasn’t immediately obvious from the bzr docs, but a little shouting, throwing things at the monitor and emptying nerf after nerf at the keyboard eventually got me to the hook I wanted specifically.

Now I had the ability to hook into things with bzr, but where could I send the events that it was generating? EMail was a bit dull, so I went back to MQTT, with the thought of commits could now light a lamp in the office when they happen The code for the server side bzr plugin is below. You just need to drop it into your .plugins directory for your smart server (ensuring you set this up in you wsgi configuration).

from bzrlib import branch
import mosquitto as mqtt
import os

mqttServerIp = "192.168.0.250"

def post_push_hook(push_result):
    branchFolder = [x for x in str(push_result.branch.__dict__['_base']).replace("//","/").split("/") if x]
    connectSendDisconnect("new branch revision: "+str(push_result.new_revno),branchFolder[1])

def connectSendDisconnect(msg, branchName):
    mqttc = mqtt.Mosquitto("bzrlib"+str(os.getpid()))
    mqttc.connect(mqttServerIp, 1883, 60, True)
    mqttc.publish("/code/"+branchName, msg, 1, False)
    mqttc.loop()
    mqttc.disconnect()

branch.Branch.hooks.install_named_hook('post_change_branch_tip', post_push_hook, "My post_push hook")

Magic! Now a message will be sent on the “/code/branchName” topic every time a commit happens Using some borrowed python magic from http://chemicaloliver.net/programming/first-steps-using-python-and-mqtt/ I’ve made it integrate into the default Ubuntu notifications system so a nice little box pops up informing me of a commit and the new revision number of the branch Doubles aces!

Installing Mosquitto is a doddle. I added the ppa to one of my household servers, then just apt-get install mosquitto. MQTT broker up and running in a matter of minutes. The first thing I setup to check if it was working was a simple python program that would just connect to the broker and send “Hello World” to the “hello” topic. I wrote a second little python script to just listen to the “hello” topic and print out the message from any updates. Most of the code was borrowed from this blog post which was incredibly helpful

With that running happily, I moved on a stage. Digging out an old shield for my arduino that I made to control the lights in my flat using some home easy sockets and an RF chip, I added an LDR and used the thermister that was already on the board. This simple set of components means I can now monitor both the temperature and light levels in one of the rooms of my flat. Using an ethernet shield for the arduino also means that I can then report that information to my Mosquitto broker using the arduino mqtt library.

What will I do with all this information? Well, initially I’m going to write an mqtt to sql bridge (might use this an excuse to learn postgres now mysql has an uncertain OSS future) and will setup some scripts to graph the information. I’ll probably change this sketch to control the lights in the flat, so I could have a machine somewhere sending a message to the mqtt broker at a set time to turn the lights on in the bedroom (a simple mqtt powered alarm clock ). Hopefully I’ll write a small application using the app indicator framework for Ubuntu (and growl for OSX) to tell me when certain thresholds are passed, especially for the other sensors I have coming (Alcohol, Gas, Smoke and humidity). I might write another bridge to use the google cloud messaging system to add an alert on my phone as well.

There’s a whole world of possibilities out there, thanks to so many people building these various components and libraries that make hacking fun toys so much easier Many thanks to all those who have developed the frameworks, libraries and services that I’m using and for making the F/OSS to make life even easier.

c# uses the .net framework to do all the heavy lifting. It’s a bit like Java in that it’s compiled into byte code and then interpreted so isn’t as fast as something like C++ but makes up for that by being easy and fast to develop in. Some of the widgets that come with the new .net framework (such as the graphing widget) are pretty fantastic and really do make life easy. In a couple of hours I’d refamiliarised myself with the language and written a short program that took in a CSV export from my Current Cost recording box and turned it into a nice graph. Doing the same using the GTK toolkit would probably take a bit longer with a little hacking around as there’s no dedicated graphing widget (that I’m aware of), so you’d have to generate the graph either on the fly by drawing it at a lower level or by creating a jpg and then displaying that in an image box.

One minor issue with the Visual Studio express suite is that there is no built in in subversioning system. At work, and most of the time at home, I tend to use the bzr suite of tools. A quick look on the bzr website and that’s also available for Windows. Download, install and it integrates very nicely into the My Computer interface, and can be called from the command prompt. Easy.

A lot of my dev work is for server stuff, so putty and winscp were downloaded and installed successfully on top of that as well.

Printers are another problem and a bit of a nightmare. I have an HP Photosmart C4180 All-in-One that I bought some time ago while I was at university. Having long since lost the driver disk for this, I nievley just plugged it in and hoped Windows would just find it as Ubuntu does. No such luck. It goes away and gives me a list of printers which mine isn’t in and then offers me the option to go to the Windows update site to find even more. I click that button and wait 5 minutes… then another 5 minutes… then it gives me a bigger list of printers. My printer is in this list (why it couldn’t just have said “We found your printer! Here is the driver.” rather than me spending 5 minutes scrolling through a badly sorted list I don’t know) and then installed it. The scanner wouldn’t work without the software from HP, but otherwise the thing worked as expected.

I have to admit, the boot time for Windows is now slowing down considerably. But my games work and it’s not overly getting in my way yet. So we’ll see.

There are a lot of things that annoy me with various tasks I perform every day. This past week I’ve been taking note of each one, and I count 78 in total. Each one of those 78 things could probably be fixed very easily, but because I’m generally running around like a headless chicken trying to get things done, I don’t even think about fixing them, I just put up with them and carry on. A fair few of these are things that take a while to fix, like the office being short of physical space because of all the orders coming in/out at the moment. Other issues are easy to fix, like the fact that it takes me 30 minutes to an hour to do a quick analysis of some testing data. This is a problem I can solve easily with a smattering of python and a little concentration. In fact, I’m intending to tackle this on Sunday by going into the office when it’s quiet, and just making this problem vanish.

A few other problems on that list of 78 can also be solved with code, some easily, some not so. Some can be solved by a few simple changes in my work routine, like the fact I constantly go to do a task, get pulled away by something else, and then forget what I was going to do. Keeping a simple log book of what I’m doing during the day would solve this easily. It would be better if this was digitized in some form, but for now a simple notebook will probably make a huge difference. This distraction on its own has probably caused me to lose my train of thought more times this week than any other on the list.

So how id my fight going? I’m much more aware of what causes me to be distracted after the past week or so. It’s taking me time to come to terms with each of those distractions and to deal with them, but ultimately, I am becoming more productive in a given period of time. The next big thing I’m going to have to tackle is project methodology. Since I don’t even have a passing familiarity with any of the standard methodologies, this is going to take some research, some thought and  a lot of conversations in order to find the best one to suite the needs of my team at work. As always, any pertinant reading material suggestions are always welcome.

Recently at work I’ve been doing masses of OO stuff (as a change from the previous stuff I was writing which was largely linear) and as a result of that I’ve started seeing the world in terms of objects. Now in order to achieve what I needed at work, I ended up reading up on the theory of objects and what they actually are. In times past when I’ve been “taught” how to program (rather than teaching myself) , the content of the classes where more designed for a specific language on how to solve a specific problem. I think I only once sat in a computing based lecture that talked about the more abstract issues in programming. Whenever I start designing a program I always go back to that single lecture, and a few books that have a higher level stance on problems (such as Design Patterns)  and then once I’m passed that do I go to the language specific things.

The reason I digress this far is that after learning all this meta-stuff, learning another language is actually relatively easy. Now everything is falling down to syntax rather than subject. Going back to the title, I think that those lectures that taught me to do this in this language because it does that has got things backwards. They were doing the training first in the hope it would teach you something. What they should be doing is teaching you the concepts (all the “meta”) and the train you in the language.

Teach and then train, not train and then teach. Something I shall have to try and follow more often in the future.

(Many thanks to Alan Bell for the inspiration in the title )

Transmitter Layout

Now the receiver layout.

Receiver Layout

and finally the code.

#reciever code

1
2
3
4
5
6
7
8

void setup() {
    Serial.begin(19200);
}
void loop() {
    if (Serial.available()){
        Serial.print(Serial.read(), BYTE);
    }
}

1
2
3
4
5
6
7
8
9

#transmitter code
void setup() {
    Serial.begin(19200);
}
 
void loop() {
    Serial.write("Hello\n");
    delay(1000);
}

This should result in the receiver outputting “Hello” over and over across the USB serial in the arduino environments serial monitor. If you have any problems, feel free to give me a shout! Oh, make sure you unplug the easy radio receiver from the arduino before flashing it, it’ll throw nasty errors otherwise. (Simply unplug digital 0 pin on the arduino board)

As an experiment at work I thought I’d try using eclipse as an IDE instead of my normal vim+terminator job. Scary as it is, I find myself actually quite liking eclipse. It may be that my work machine has 4GB of RAM, and so copes better than the machines I’ve used in the past, or it may be that I’m starting to lose my qualms about what tools I use to do a job, as long as a job gets done. The PHP and Python tools inside eclipse have made my life a lot easier, and I really do find little things, like it reading out the docstring I’d put in a function when I hover over that function when it’s called, useful. Has eclipse evolved to where it’s useable or has affordable technology caught up with eclipse? A bit of a quandry for me that one.

The other small revelation I’ve had recently is that KDE4 is now inherently useable, and quite shiny to boot. when I’ve tried it in the past I quickly got fed up with things that didn’t quite fit or where missing completely, but now time has passed, and like KDE4 I believe I’ve changed a bit, and actually quite like it. I won’t be using it at home for a while, as the 7″ screen on this tiny little netbook certainly won’t make it very use-able compared to the  20″ odd monitor I have at work. The one big thing annoying me with it at the moment though is that konquerer doesn’t seem to fit with the default theme. Niggly annoyance I know, but surely that should be a papercut?

The last revelation I’ve had, though it’s not really a revelation, is a pang of guilt. I’m inherently a consumer in the whole Linux ecosphere. I consume by far more than I give back, and at the moment I simply don’t have the time to give back as much as I’d like. So this is my decree, and a proclamation that as of next year (with certain exceptions) I intend to deem one night a week free software night. On that night I will help to squish bugs, I’ll sit on IRC and be patient with people trying to help them through problems, I shall try and get involved in the various mailing list debates I sit and read, and I shall attempt to stop consuming quite so much and start giving back as much as I can.

In order to do this I will need a little help. The whole software workflow thing is a bit of a mystery to me. I’ve had little formal training in such things and as such tend to wing it more than I’d like. Can people point me in the direction of some good literature to help mend this? I’m quite willing to get my hands dirty if people are willing to be patient with me as I learn how the OSS developer crowd works so I can learn and adjust. In a way I’m hoping that this will flow back and help me at work as much as it’ll help me contribute back to the community in general.

If anyone also has a project they might want a hand with one evening a week from the of the month, feel free to drop me a line by your favourite communications method

Plot of Deny Hosts Blocks per Day

At the end of June I will stop using port 22 and start using another random port. I’ll then collect data for 3 months and at the end of september do another blog post showing the difference. I also have another server that I will repeat this experiment on, but that one will be 3 months behind.

Hopefully then I will have a nice sturdy scientific answer as to how much more protection moving ssh to a different port gives

The code I used to generate this graph is given below for reference.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77

import os
import matplotlib
import datetime
import matplotlib.pyplot as plt
import matplotlib.dates as mdates
import matplotlib.mlab as mlab
matplotlib.use('Agg')
 
 
datelist = {}
rootdir = './'
 
def plot_all():
	#first convert dict to a set of x values and a set of y values
	keys = datelist.keys()
	keys.sort()
	values = []
	times = []
	for key in keys:
		values.append(datelist[key])
	#now convert the keys into time format
		times.append(datetime.datetime.strptime(key, "%Y-%m-%d"))
 
	#now we're ready to plot with matplotlib
	months   = mdates.MonthLocator()  # every month
	days	= mdates.DayLocator()
	yearsFmt = mdates.DateFormatter('%b')
	dayFmt = mdates.DateFormatter('%d')
 
	fig = plt.figure()
	ax = fig.add_subplot(111)
	dates = range(times[0].toordinal(), times[-1].toordinal()) 
	ax.bar(times,values,width=1)
 
	ax.set_xlabel('Date')
	ax.set_ylabel('Number of Hosts Denied')
 
	#ax.plot(times, values)
	#ax.xaxis.set_major_formatter(yearsFmt)
	#ax.xaxis.set_major_locator(months)
	#ax.xaxis.set_minor_formatter(dayFmt)
	#ax.xaxis.set_minor_locator(days)
	ax.format_xdata = mdates.DateFormatter('%Y-%m-%d')
	fig.autofmt_xdate()
        fig.savefig("plot.png")
 
def countup(file):
	f = open(file, 'r')
	for line in f.readlines():
		#split by spaces to get the date
		line = line.split(" ")
		#now see if this is already in the list
		newline = False	
		for part in line:
			if part == "new":
				newline = True
			else:
				continue
		if newline == True:
			n = 0
			if line[0] in datelist:
				datelist[line[0]] = datelist[line[0]] + 1
			else:
				datelist[line[0]] = 1
	f.close()
 
for subdir, dirs, files in os.walk(rootdir):
	for file in files:
		if not file[-2:] == "py" and file.split(".")[0] == "denyhosts":
			countup(rootdir + file)
 
keys = datelist.keys()
keys.sort()
 
for key in keys:
	print("%s, %s" % (key,datelist[key]))
plot_all()

System Log Files

Linux distributions have fantastic logging capabilities built in. If there is a hardware fault it will more than likely appear in the system log.  The most useful log files are kept in /var/log/ and can be accessed directly or through the Log File Viewer (System -> Administration -> Log File Viewer.)

System Log Viewer

The one thats most useful is the dmesg log. This can be found in the log viewer or on the command line by typing dmesg.  There’s normally lots of information here and depending on what the fault is with depends on what’s not working. If you have a non-functional USB peripheral, like a USB wifi card or a printer, a look at what dmesg says before it’s plugged in and after it’s plugged in can supply you with some information to start hitting google, the mailing lists or IRC with. Sometimes it will tell you what needs to be done, for example I have stuck a dodgy memory stick in one machine which didn’t automount, dmesg quite happily told me the stick had errors and to check it. Fantastic!

The Tops

Is something slowing your machine down to a crawl? There are a few tools that can help here, largley from the “top” family of applications. Starting with the ones that are graphical, you can use the System Monitor (System -> Administration -> System Monitor) and have a look under the processes tab to see which process is eating the most cpu. Easy peasy.

System Monitor Tool

The next one is top itself. Open a terminal and type “top”. This gives us a nice list of running processes and how much cpu/memory it’s using and its pid (process id). You can kill processes by hitting k and then typing in the pid.

top

The main reason I introduce people to top is because it is installed by default on most distros. A better version is htop, which while still in the terminal, is much easier to read.

htop

As you can see, it gives you all the info top gives you, but in a slightly nicer, more intuitive layout. What if somethings really giving your hard drive a bad time? Want to find out what process is doing that, simply install iotop (sudo apt-get install iotop).

iotop

This gives a list of the processes running on the machines, who’s running it and how much io it’s using.

Other Useful Things to Know

Along with these diagnosis tools, there are also a few tools which can identify what hardware is connected to your machine. These are all in the terminal, and are “lspci”, “lsusb” and “lsmod”. These will give you a list of hardware connected to a pci like adapter, a usb like adapter and the modules loaded into your running kernel. If you’re asking for help somewhere, you will more than likely be asked for the information from one of these.

Rounding Up

The purpose of giving access to all this information is because diagnosis is half way to a fix. If you know what is causing the problem you know where to start looking, and what to start google-ing for. Hopefully this post will be helpful to someone I’ll write a few more in the future going into a bit more depth into each of the commands listed, but a useful place to find more info on them is in their man pages, in a terminal simply run “man <command>” where<command> is the command you’re running.

I’d been intending to learn Lua for a little while now, so this entire project was written in lua, with some bash scripts to call stuff. Essentially what it does, is every minute it checks the time. If the hour is greater than or equal too that set in /etc/bedtimer then it uses iptables to close all the ports I use that go out to the internet. This allows me to continue using my local network stuff (except ssh) if I had things like coursework to do. The server part continues to run every minute after that, incase I try and override it (bad me). There is also a gnome bit that is loaded when I log in. This part simply uses the ubuntu notify-send thing to warn me when I have 15 minutes before the net shuts off, and when it shuts off.

The bit that does all the work is loaded at boot using an initscript. I have yet to reboot and test this, as it’s my first initscript, but it looks ok

Hopefully this will force me to actually stop wasting time on the net in the evenings and actually go read a book or do something to wind down at the end of the day.

The files can be found here. There is a shell script to install it all, but it’s rubbish and may not work on your machine! BE WARNED!