Matt Daubneys Blog

A few days ago I received a pair of Easy Radios to help with a project I’m working on. Since I’ve never used these before (having only got my first arduino a couple of weeks ago) I had a dig on the net for information on how to use them. Since this information was quite sparse, I thought I’d put up my experimental rig to show it working. This setup requires 2 arduinos, an easy radio transmitter and an easy radio receiver.  First, the transmitter layout.

Transmitter Layout

Now the receiver layout.

Receiver Layout

and finally the code.

#reciever code

1
2
3
4
5
6
7
8

void setup() {
    Serial.begin(19200);
}
void loop() {
    if (Serial.available()){
        Serial.print(Serial.read(), BYTE);
    }
}

1
2
3
4
5
6
7
8
9

#transmitter code
void setup() {
    Serial.begin(19200);
}
 
void loop() {
    Serial.write("Hello\n");
    delay(1000);
}

This should result in the receiver outputting “Hello” over and over across the USB serial in the arduino environments serial monitor. If you have any problems, feel free to give me a shout! Oh, make sure you unplug the easy radio receiver from the arduino before flashing it, it’ll throw nasty errors otherwise. (Simply unplug digital 0 pin on the arduino board)

For the past few weeks life has been busy, and when I say busy I mean hectic beyond belief. In that time I’ve had a few OSS revelations I’d like to share.

As an experiment at work I thought I’d try using eclipse as an IDE instead of my normal vim+terminator job. Scary as it is, I find myself actually quite liking eclipse. It may be that my work machine has 4GB of RAM, and so copes better than the machines I’ve used in the past, or it may be that I’m starting to lose my qualms about what tools I use to do a job, as long as a job gets done. The PHP and Python tools inside eclipse have made my life a lot easier, and I really do find little things, like it reading out the docstring I’d put in a function when I hover over that function when it’s called, useful. Has eclipse evolved to where it’s useable or has affordable technology caught up with eclipse? A bit of a quandry for me that one.

The other small revelation I’ve had recently is that KDE4 is now inherently useable, and quite shiny to boot. when I’ve tried it in the past I quickly got fed up with things that didn’t quite fit or where missing completely, but now time has passed, and like KDE4 I believe I’ve changed a bit, and actually quite like it. I won’t be using it at home for a while, as the 7″ screen on this tiny little netbook certainly won’t make it very use-able compared to the  20″ odd monitor I have at work. The one big thing annoying me with it at the moment though is that konquerer doesn’t seem to fit with the default theme. Niggly annoyance I know, but surely that should be a papercut?

The last revelation I’ve had, though it’s not really a revelation, is a pang of guilt. I’m inherently a consumer in the whole Linux ecosphere. I consume by far more than I give back, and at the moment I simply don’t have the time to give back as much as I’d like. So this is my decree, and a proclamation that as of next year (with certain exceptions) I intend to deem one night a week free software night. On that night I will help to squish bugs, I’ll sit on IRC and be patient with people trying to help them through problems, I shall try and get involved in the various mailing list debates I sit and read, and I shall attempt to stop consuming quite so much and start giving back as much as I can.

In order to do this I will need a little help. The whole software workflow thing is a bit of a mystery to me. I’ve had little formal training in such things and as such tend to wing it more than I’d like. Can people point me in the direction of some good literature to help mend this? I’m quite willing to get my hands dirty if people are willing to be patient with me as I learn how the OSS developer crowd works so I can learn and adjust. In a way I’m hoping that this will flow back and help me at work as much as it’ll help me contribute back to the community in general.

If anyone also has a project they might want a hand with one evening a week from the of the month, feel free to drop me a line by your favourite communications method

I have been told many many times that moving ssh to a different port (i.e. other than 22) makes your machine more secure. I do see some wisdom in this, however, I’ve decided to put it to the test. I have been using Denyhosts to stop brute force attacks on my ssh servers for some time now, and on my most recent server, the attacks per day are fairly regular, as seen in the graph.

Plot of Deny Hosts Blocks per Day

At the end of June I will stop using port 22 and start using another random port. I’ll then collect data for 3 months and at the end of september do another blog post showing the difference. I also have another server that I will repeat this experiment on, but that one will be 3 months behind.

Hopefully then I will have a nice sturdy scientific answer as to how much more protection moving ssh to a different port gives

The code I used to generate this graph is given below for reference.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77

import os
import matplotlib
import datetime
import matplotlib.pyplot as plt
import matplotlib.dates as mdates
import matplotlib.mlab as mlab
matplotlib.use('Agg')
 
 
datelist = {}
rootdir = './'
 
def plot_all():
	#first convert dict to a set of x values and a set of y values
	keys = datelist.keys()
	keys.sort()
	values = []
	times = []
	for key in keys:
		values.append(datelist[key])
	#now convert the keys into time format
		times.append(datetime.datetime.strptime(key, "%Y-%m-%d"))
 
	#now we're ready to plot with matplotlib
	months   = mdates.MonthLocator()  # every month
	days	= mdates.DayLocator()
	yearsFmt = mdates.DateFormatter('%b')
	dayFmt = mdates.DateFormatter('%d')
 
	fig = plt.figure()
	ax = fig.add_subplot(111)
	dates = range(times[0].toordinal(), times[-1].toordinal()) 
	ax.bar(times,values,width=1)
 
	ax.set_xlabel('Date')
	ax.set_ylabel('Number of Hosts Denied')
 
	#ax.plot(times, values)
	#ax.xaxis.set_major_formatter(yearsFmt)
	#ax.xaxis.set_major_locator(months)
	#ax.xaxis.set_minor_formatter(dayFmt)
	#ax.xaxis.set_minor_locator(days)
	ax.format_xdata = mdates.DateFormatter('%Y-%m-%d')
	fig.autofmt_xdate()
        fig.savefig("plot.png")
 
def countup(file):
	f = open(file, 'r')
	for line in f.readlines():
		#split by spaces to get the date
		line = line.split(" ")
		#now see if this is already in the list
		newline = False	
		for part in line:
			if part == "new":
				newline = True
			else:
				continue
		if newline == True:
			n = 0
			if line[0] in datelist:
				datelist[line[0]] = datelist[line[0]] + 1
			else:
				datelist[line[0]] = 1
	f.close()
 
for subdir, dirs, files in os.walk(rootdir):
	for file in files:
		if not file[-2:] == "py" and file.split(".")[0] == "denyhosts":
			countup(rootdir + file)
 
keys = datelist.keys()
keys.sort()
 
for key in keys:
	print("%s, %s" % (key,datelist[key]))
plot_all()

Every so often people ask for help and are not sure where the problem lies. When diagnosing problems in Ubuntu, and some of these tips should apply to other linux distros as well, there are a few places you should look at.

System Log Files

Linux distributions have fantastic logging capabilities built in. If there is a hardware fault it will more than likely appear in the system log.  The most useful log files are kept in /var/log/ and can be accessed directly or through the Log File Viewer (System -> Administration -> Log File Viewer.)

System Log Viewer

The one thats most useful is the dmesg log. This can be found in the log viewer or on the command line by typing dmesg.  There’s normally lots of information here and depending on what the fault is with depends on what’s not working. If you have a non-functional USB peripheral, like a USB wifi card or a printer, a look at what dmesg says before it’s plugged in and after it’s plugged in can supply you with some information to start hitting google, the mailing lists or IRC with. Sometimes it will tell you what needs to be done, for example I have stuck a dodgy memory stick in one machine which didn’t automount, dmesg quite happily told me the stick had errors and to check it. Fantastic!

The Tops

Is something slowing your machine down to a crawl? There are a few tools that can help here, largley from the “top” family of applications. Starting with the ones that are graphical, you can use the System Monitor (System -> Administration -> System Monitor) and have a look under the processes tab to see which process is eating the most cpu. Easy peasy.

System Monitor Tool

The next one is top itself. Open a terminal and type “top”. This gives us a nice list of running processes and how much cpu/memory it’s using and its pid (process id). You can kill processes by hitting k and then typing in the pid.

top

The main reason I introduce people to top is because it is installed by default on most distros. A better version is htop, which while still in the terminal, is much easier to read.

htop

As you can see, it gives you all the info top gives you, but in a slightly nicer, more intuitive layout. What if somethings really giving your hard drive a bad time? Want to find out what process is doing that, simply install iotop (sudo apt-get install iotop).

iotop

This gives a list of the processes running on the machines, who’s running it and how much io it’s using.

Other Useful Things to Know

Along with these diagnosis tools, there are also a few tools which can identify what hardware is connected to your machine. These are all in the terminal, and are “lspci”, “lsusb” and “lsmod”. These will give you a list of hardware connected to a pci like adapter, a usb like adapter and the modules loaded into your running kernel. If you’re asking for help somewhere, you will more than likely be asked for the information from one of these.

Rounding Up

The purpose of giving access to all this information is because diagnosis is half way to a fix. If you know what is causing the problem you know where to start looking, and what to start google-ing for. Hopefully this post will be helpful to someone I’ll write a few more in the future going into a bit more depth into each of the commands listed, but a useful place to find more info on them is in their man pages, in a terminal simply run “man <command>” where<command> is the command you’re running.

I’ve discovered recently that I don’t sleep well if I sit on the internet too late at night. Here is my solution.

I’d been intending to learn Lua for a little while now, so this entire project was written in lua, with some bash scripts to call stuff. Essentially what it does, is every minute it checks the time. If the hour is greater than or equal too that set in /etc/bedtimer then it uses iptables to close all the ports I use that go out to the internet. This allows me to continue using my local network stuff (except ssh) if I had things like coursework to do. The server part continues to run every minute after that, incase I try and override it (bad me). There is also a gnome bit that is loaded when I log in. This part simply uses the ubuntu notify-send thing to warn me when I have 15 minutes before the net shuts off, and when it shuts off.

The bit that does all the work is loaded at boot using an initscript. I have yet to reboot and test this, as it’s my first initscript, but it looks ok

Hopefully this will force me to actually stop wasting time on the net in the evenings and actually go read a book or do something to wind down at the end of the day.

The files can be found here. There is a shell script to install it all, but it’s rubbish and may not work on your machine! BE WARNED!

My knowledge of linux is sadly lacking, but every day I improve on it. Since I had a few days free over the easter holidays I thought I’d try and improve this further by installing Linux From Scratch. This is essentially building up a linux system from it’s base packages and takes a LONG time. This to me seemed like a good idea, it would exponentially increase my knowledge of what makes linux tick, and what depends on what.

I started on Monday night by reading through all of the LFS documentation. As bedtime reading goes, it’s actually quite interesting, and it seemed like a good idea to get an overview of what needs doing before I started.

Luckily for me my laptop has a fairly large HD (320GB) so finding some space on a partition was quite easy. A quick boot into a livecd and resizing my /home partition created a nice little 10GB partition for LFS to go in. That was at 9am in the morning. By 10:00am I had downloaded all the required programs in the mounted partition as directed by the book and started to build them.  In the first day, I managed to build the initial toolchain and got into the chroot and got to chapter 6.15 in the LFS PDF. The longest thing to build was probably either glibc or GCC, which took just over an hour, but did give me time to cook some tortillas from scratch for lunch

The second day, it took me from about 10 in the morning till roughly 3 in the afternoon to finish installing everything.  Was a bit quicker than I expected, but also a lot more involved.

I did hit a few roadbumps, I didn’t have gawk installed on the host system, so at one point something didn’t compile properly and freaked out a bit. A simple sudo aptitude install gawk fixed that though. When I was recompiling glibc all of the tests failed, it took me a few minutes to realise that the reason this was happening was because I’d forgotten to run “make”.. oops.

The last major bump was that when I chose which options to compile with the kernel, I forgot to add the drivers for my ethernet card and my wireless card, so when I booted the machine I had no network! A quick scan through the options and a recompile sorted that out (to include the b44 driver)

What have I learnt from this? Two things mainly.

1. Where everything should live on a linux system and why its there
2. Package Managers are beyond fantastic. They save so much time and hassle!

Tomorrow I intend to try and make my LFS install useful by installing openssh initially, wget and possible x.org and gnome. That may be a bit ambitious at the moment though!

This is a simple jabber bot which lets you jot down simple memos. Use the command “add:” to add a memo, e.g. “add: This is a memo”. the command “list” will give you all the memos in the database including their ids, and “delete: 3″ will delete the memo with id 3.

It’s written in Python with sleekxmpp as the xmpp library, and uses sqlite for the database

Code can be found here

Today I decided to start looking into packaging a few of my python scripts. The first step to this to me looks to be setting up pythons own distribution system. I’ve just known this as a setup.py file, the meaning of the contents of which has confused me. A quick search through google brings me to this document. It’s a bit confusing at first, but makes much more sense once you sit down with it and start hacking away.

The first thing I realise I need to do is sort my programs into some kind of logical directory structure and inside the directory with all the modules place a __init__.py file. I’m not entirely sure what this file does… but it doesn’t work without it! Anyway, moving swiftly on! Once I’d done that I copied across the simple example from the documentation and then started playing. My final setup.py looked like this.

from distutils.core import setup

setup(name=’MattsLibrary’,
version=’0.1′,
author = ‘Matthew Daubney’,
author_email = ‘my@email’,
description = ‘A very short library for recording serial numbers and books’,
package_dir = {‘MattsLibrary’: ‘library’},
scripts=['library/mattslibrary']
)

Let’s examine this line by line. The first line imports the distutils module, this is the module that will do all the work for us.

The second line just calls the setup function, passing it all the stuff we want. Also on this line I set the name of the module/program I’m defining. The next line sets the version number of the program and the next two tell it my name and set my email address to it. The line after that gives a short description of the program.

The next two lines are the important ones here. Package_dir tells the setup function that all my modules are in the library folder. ['MattsLibrary':'library'] is of the format ['ProgramTitle':'folder'] however here (from what I understand) the ProgramTitle bit is not needed, and I’m not entirely sure of it’s purpose. However, this, works. When I have more time I’ll try and understand it a bit more.

The final line tells the setup function that there is a script that needs to be installed. These are installed in /usr/bin/ instead of /usr/lib/python2.5/site-packages/. This is set as a list such as scripts=['library/mattslibrary', 'library/script2', 'library/script3'].

These settings work for my little simple program. But the documentation seems to be fairly thorough, however I recommend you actually do this with a program as you read the docs. It makes more sense that way!

With all this set up I can now install my little program by typing:

python setup.py install

And all is well…

After we’ve had a few issues with Virgin Media, I decided it was time to take the plunge and find out if it was us or them. The simple way to do this was to measure the amount of bandwidth we use during a day, see if we exceed their fair use terms for the cap to come into effect. I also wanted to throttle people using Bittorrent in the house, so it was time for a network redesign! Out came Dia and we end up with this.

Once I had a network design, all I had to do was put together an old, low power box and install IPCop. The first box I built had a duff motherboard on it, and the second one didn’t like one of the three network cards I was feeding it. A little fiddling around and I had a working box. Turned it on and it sounded like a 747 taking off. The solution to this was simple, a quick clean with an air duster and an application of spray grease on the fan spindles. This solved the noise problem quite nicely and allowed me to continue on to install IPCop.

The install for IPCop from CD is the same as installing linux in the pre-LiveCD age. You simply follow the onscreen prompts and let it do what it needs to. As in the diagram, I set up a red, green and orange network. I decided this after a bit of reading around the subject on the IPCop site that recommended putting any webservers and the like into the so called orange zone to prevent people from breaking into the trusted green zone. The red zone is the wild wild west of the Internet, so it’s allowed the least access into the house.

Once IPCop had installed nicely I discovered a small problem. My previous network setup had completley relied on DHCP to give out the IP addresses, it turns out that IPCop does not do DHCP in the Orange Zone. A quick google about for some commands and I discovered how to set the ipaddress properly and the nameservers so it could talk to the interwebs properly.

Now I had IPCop set up correctly I clicked on to it and gave everyone a fixed IP, so I can track individuals bandwidth use easily. This was done easily through the GUI. I also set up a transparent proxy in order to play practical jokes easily

The next part was the hardest. How to actually record individuals bandwidth usage. To do this I dug about again and found an addon called Net-Traffic. This gave me oodles of information, but not to the level I was after. A further dig around led me to the idea of adding more rules to ipac-ng. Remembering that IPCop puts ssh on port 222, I shelled in and found the rules file in /etc/ipac-ng/rules.conf. I then added a couple of lines to this such as

#bandwidth by Wii
internet to Wii|ipac~fi|eth2|all||192.168.1.106||
Wii to internet|ipac~fo|eth2|all|192.168.1.106|||

Once this was accomplished, I could check on peoples bandwidth usage by running

ipacsum -t today

which would give todays usage.

I shall give it a few weeks and then look over these nmbers and see just who is eating the internet alive!

So I’ve decided to learn to write C# as I have been repeatedly told that it is a good language (even if it’s designed by MS). So i’ve dug out and installed a copy of MonoDevelop and got started with the Wrox book “Beggining Visual C# 2005″.

I start by writing a simple Hello World application as shown on page 19 of the book, which in Mono looks like the following:

// project created on 9/1/2007 at 8:08 AM

using System;namespace HelloWorld

{

 class MainClass

 {

 	public static void Main(string[] args)

 	{

 		Console.WriteLine("Hello World! My first C# proggy!!");

 		Console.ReadKey ();

 	}

 }

}

A quick build attempt and the first problem comes along with an error of System.Console does not contain a definition for ReadKey (CS0117)
A quick scout around on Google and I find that MonoDevelop assumes you’re writing for .net version 1.1 and you have to change this under Project -> Options -> Runtime Options. Then set Runtime Version to 2.0.

After trying to compile it again nothing happens. A read through the build logs shows that I haven’t installed gcms (some kind of mono compiler I assume??) one apt-get install mono-gcms later and everything compiles and runs as it should