Andrew Williams

Python developer, serial tinkerer, and committed geek.

Open Sourcing Past Projects

Over a year ago now I stepped back from being a system administrator and developer for the EVE Online alliance Test Alliance Please Ignore, part of my role there I spent hundreds of hours developing their internal authentication system and other small applications that hang off it. At the time it was quite unique and only a handful of other alliances had that level of technical setup. So as you would expect like a small company with something to lose the code was buried away on private servers and rarely looked over by new people.

Today is a very different place, Auth was created at the start of a open source revolution for EVE Online applications, and over time more and more have started becoming open with now specific projects being spun up (such as ECM) to create tools, or large alliances (Brave Newbies) opening their backend for everyone to use.

The repository copies of the code I have a quite out of date, and i'm purely the copyright holder of them, which gives me the power to license and open them as I see fit. Now that its been a good amount of time since I left I feel I can safely release these into the public domain without doing any disservice to TEST and the current sysadmins.

So over the next few days i'll be looking to move the following repositories from my private Bitbucket over onto GitHub:

All in various states, but hopefully useful for someone.

[Update - 2014/03/13]

Also i'll update with links once they're over.

Python Packaging, the right way

Last night I spent a hour or so packaging up some Python I made to scratch an itch into a distributable module. Packaging has never been my strong point and I always ended up making a fiddly setup.py that had some minor problems or didn't work as expected. This time was especially noteworthy as I had a product that was both Python 2.7 and Python 3.3 compatible.

Thankfully Jeff Knupp posted about open sourcing a python project the right way, which covers getting your project setup right, making it easily testable, and getting it working on TravisCI.

So, my project is live on GitHub, is it useful? Probably not, but at least i've put it out there incase people want to use it or improve on it. Next on the todo list is some better documentation and more tests.

Easy bag organisation with the "Grid-All"


I’ve been eyeing up the Cocoon Grid-It for some time, it looked like the perfect tool to keep all that loose crap in my bag organised in a sensible and easy accessible way. The only thing that pushed me away was the price which just seemed a tiny too high for the actual product itself, I understand its quite a unique idea and they put a tax on that, but still for a stiff piece of card with some woven elastic over it?

Thankfully the other day I came across a Chinese “competitor" version, a little bit cheaper and into the price range I was looking for, so I snapped one up on Amazon and gave it a go. When I received the package yesterday I was a little shocked that it came in the exact same packaging as the Cocoon versions you see in the Apple store. It seems that Cocoon has suffer the same fate as other companies, their unique product designs are re-badged by the factory that produces them for their own direct cut of the product. The most common victims of this are the electronic cigarette community where frequently you’ll have clones or the actual mass produced products rebadged and sold at a large discount over the original creators. Its a shame to support these type of tactics but I originally thought it would be a simple competitor instead of a direct factory rebadge and rip-off.

The product works exactly as its designed, holding tight even small USB OTG cables, and the rubber threading through the elastic keeps good firm grip on any items you put in it. I picked up the 30cm x 20cm version which is large enough to hold the bits I need on the go, and it easily fits into my Crumpler laptop bag with lots of room to spare.

So, i’d highly recommend it, but get one from Cocoon, and support the original innovators and not the knock-offs.

Always catch errors

On the 31st of Jan my NAS stopped responding, no idea what was going on and with zero response to the power button I did a hard reset, I spent the next few hours double checking all my config to find out what the hell had happened. I couldn’t find a solid reason, but at least none of the hardware was failing which give me some good news, I marked it down as an odd issue and carried on.

The same happened tonight, exact same result but this time I was prepared to some point. After attempting to login in the console and seeing memory allocation errors, then SSH dying on its arse, I checked my Munin install and notice the memory was heavy swapping. This machine has about 8GB of RAM but at any time its using about 600mb, at first I thought it was a memory leak in something but usually OOM Killer does a good job smiting any unruly processes. Then I checked my process list and noticed it was well over 4,000 sleeping processes, something had obviously gone wrong.

On my Deluge setup, due to the instability of a few of the trackers I use, I have a small Python script that checks the current state of the torrent and if they’re “red" it restarts them. Deluge’s API uses the Twisted framework to make everything async and accordingly a lot easier to work with, this was my first venture into the land of Twisted and it seems I made an error; I didn’t catch the “unable to connect" error. So after it was unable to connect the Twisted reactor was sitting there and running constantly, and as this job was running every 5 minutes it stacked up over 24 hours and killed the machine.

So, its always worth checking for errors, and not assuming that it’ll sort itself out. Lesson learned.

New Project Woes

Finding a new project is hard. Very hard.

Since i've stopped playing EVE Online i've found it very difficult to come with new development ideas. In recent months i've padded my time with some smaller projects and ideas but usually lost my drive behind them within a few weeks. So what to do?

This weekend i've been thinking of a few things that slow down my workflow, either time tracking, task planning or something to that nature. For a long time i've used Dave Seah's Emergent Task Timer to track my time usage in work, for a while I tried the Flash app but it didn't really worked how I wanted it. After a while I ended up going offline with Dave's excellent templates and i've been a happy user for well over three months.

So the idea popped into my head to make a persistent, online ETT that allows for some quick exporting into a format I can use for my timesheets in work, but after a while I decided how much time would I have to spend to improve the paper system that Dave has honed over a long time? Especially now that he has expanded onto Amazon and selling pre-printed notepads of his ETP sheet.

The next idea was to create a CLI for Trello, and after about five minutes I realised its already a heavily done solution, fine no Go or Python client, but still done quite a bit.

So here I am, twiddling my thumbs, waiting for the next idea to pop into my head.

How not to sell an item

More of just a follow on from my previous post, I remembered quite a little amusing nugget that happened last night.

I was picking up the Macbook Pro and I was eyeing through the accessories for a spare power adapter for the home office, saving me time moving it around with me, an employee approached me and asked the usual questions, after I explained I was picking up a Macbook Pro and a extra power cable he started telling me about the Superdrive (paraphrasing what he said);

"If you're looking to get a Macbook its an essential item, i've got one with mine, i've probably only used it once in about 6 months, but you can't be without it."

Not really the stellar use-case I was looking for, and not really something you should say if you're trying to upsell. I do get what he was aiming for, i've got a small Samsung portable DVD drive that is USB powered and it really came into its own when I realized not a single PC in the house had a optical drive, but still, he could of worded it a little better.

Anyway, if I wanted a £65 super drive to reinstall OSX or something I can grab a Thunderbolt to Firewire adapter and use my 12" Powerbook in target mode, Ooooh yes.

The Big Purchase

For what feels like forever i’ve been agonising over which laptop to buy to provide me with a decent, portable development environment. I dislike being stuck at a desk, for long periods i’ve always had a decent machine to sit on my lap and pound away at when the ideas come to me, in a previous life it was my tiny Eee PC, and before that a 12" Powerbook. For the last four or so years most of my code has been wrote on a i5 home build PC running Windows, and now that the machine is a little long in the tooth and starting to drag I decided it was time to change.

So today I went to pickup a 13" Macbook Pro, middle of the line one with 8GB RAM and a 256GB SSD, the old PC will be retired to gaming and the Macbook will be my go-to machine for general browsing and development work.

I’m not a “cult of Mac" member, I’ve had the odd Apple devices in my past but theres no way in hell i’m going to drop my Android phones for iOS devices. For me, Apple provide some excellent hardware that can’t really be equaled at the moment, and the “Apple Tax" has slowly disappeared since the switch to Intel hardware they present themselves as a well price device in the upper mid range laptop market. I had a look through the various competitors over the last few days and while they produce hardware around about the same specs and price as a Macbook it seems theres always concessions, better CPU, lower spec GPU, lower price, lower quality screen.

I’ve also come to realise that the majority of my workflow (Evernote, RTM, YNAB) have all OSX versions with equal or better quality, the odd game I enjoy playing on the go have equal versions on OSX as well (Prison Architect, FTL). So for me the Macbook sneaked in as a choice. I know some people will berate me, but hell its just a laptop, and it works for me.

Doxie One Scanner

As some may know, I have a utter deep hatred for all things Epson now. Around about 3 months ago I went to print something and the printer delightfully informed me that my Pink was low and that I should replace it, never mind that I was printing a letter purely in black and white, it still wanted the pink. We didn't have a cartridge to spare so I had to go get one from the shop, thinking that all the other levels were ok. When I got back I was frustrated when I switched it out and the printer ended up prompting for cyan as well.

Needless to say, its now sat in the junk pile and a nice cheap laser printer has replaced it.

...but this isn't what this post is about. The real frustrating part of using a multifunction Epson printer is that you can't use any other functions of it unless the ink cartridges are ok, which really kicked me in the arse as I used the printer to scan in all my documents into Evernote. So now I had to find a replacement.

I had a look through the market, and the amazing expensive but excellent "Evernote Scanner" which is a rebranded Fujitsu iX500, and some dirty cheap flat beds from eBay but one did catch my eye;

Doxie One

The Doxie One. Available for £99 (at the time of writing) on Amazon.co.uk its a small and compact scanner that can operate without a PC, dumping the resulting files on a SD card in a standard DCIM file layout, so anything that can read a camera's SD card can take these images. Excellent.

The Doxie series of scanners are made to be small and useful, its bigger brother the Doxie Go also adds a higher scanning resolution (600 DPI) and a internal lithium-ion battery. This isn't to say the One is inferior, it has a 300 DPI resolution and takes AAA batteries to just as portable as the Go.

So the scanner itself works wonderfully, it chewed through my 40 page tenancy agreement with ease and the software allowed me to clean up the pages and "staple" them all together as a nice single PDF to store in Evernote, while it doesn't win any awards for image quality (after all its 300 DPI) the speed and portability of the device really does make up for.

If you need a small, portable scanner to plow through your odd documents and receipts ready to be stored in Evernote or its kin, its an excellent buy and i'd suggest it.

Deluge Web Interface and Ngnix

After a short, frustrating time, i've finally got the Deluge WebUI to proxy through Nginx without any errors. The revelation came when digging through the Deluge forums I found a little nugget of information which solved it all, a small header call X-Deluge-Base that when passed will prefix any media calls made in the page with that text. So instead of setting up weird aliases and fiddling around with Nginx's options to get it to work I could just specify that and use a very basic server config.

upstream deluge  {
  server localhost:8112;
server {
  server_name  deluge.home;
  location / {
    proxy_pass  http://deluge;
    proxy_next_upstream error timeout invalid_header http_500 http_502 http_503 http_504;
    proxy_redirect off;
    proxy_buffering off;
    proxy_set_header        Host            $host;
    proxy_set_header        X-Real-IP       $remote_addr;
    proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header        X-Deluge-Base   "/";

Back to YNAB and avoiding the traps

YNAB has always been an excellent tool for me. I started using it around this time last year when I discovered the snappy Youtube videos, and the free demo. Within a few weeks I was hooked on using it, and I saw the massive advance quickly when I was able to manage my Christmas pay and put a hefty lump of money into a savings account, and all I needed was something to track my money.

After 7-8 months the sheen worn off as I was fighting with 60+ categories, a newly added credit card, and quirky statement importing, and around October I essentially gave up. From there my financial situation didn't get worse, but I started to slip on the credit card and with them upping my limit another £1,000 really didn't help matters.

So, another year, another fresh start with YNAB. Looking back I identified a few problems which really started to kick me after time and made my budget a hell of a lot more complicated;

Multiple categories for the same type of things

I found that I had multiple categories for essentially the same thing. Best example is my personal site hosting which was split down into Linode, Digital Ocean, Amazon, and Domains. I suppose splitting out is a good way to identify and budget for specific recurring bills, but for me Amazon store purchases got mixed in with my AWS charges and it started getting messy quickly.

So this time round i'm having a generic "Hosting" category to budget into, £15 a month should cover all my costs and also save up some money for those 2-3 year domain renewals. Sticking to one category should hopefully make my money a lot easier to manage in this area.

Make use of split transactions

For a long time I would do our weekly shop at Tesco and end up picking something up for the house, either new bulbs, paint brushes, or a duvet. The problem is I never sat down and split out the transactions into the food shopping and the house budget, which usually ended up with the food category taking a beating all the time while the various smaller categories never get touch, and end up being raided to make up the shortfall in other areas.

I suppose this was compounded by split transactions being unavailable in the mobile client up until the middle of last year, by which point I was already in a bad habit and wouldn't change so easily. The lesson here is that splitting your transactions will pay off, even if it does feel like you're wasting time doing them.

Simplify your categories

At the end I had 72 categories for my spending, as you can imagine trying to fill out the budget on payday took some time. I spent so much time flicking through my categories and assigning £10/£20 here and there that by the time I got down to my savings accounts I was putting in around 10-20% of what I originally was doing in January. Because I had a budget for damn near everything it seemed like I could validate spending it, on new techy toys and vaping gear.

At the restart i've settled at about 20 categories that seem to represent the core spending in my life. I have the bonus that my girlfriend takes care the household bills from the joint account, which simplifies my view as a single payment to the account on a monthly basis. If you get too far down into the categories then you'll be in danger of validating any type of spending, just as I did.

Reconcile your accounts

At first I never bothered using the reconcile function within YNAB as my totals always "worked out". After a few months of using it I discovered strange situations where the total amount of budget money didn't match up to the total available, its a common problem with YNAB and its heavily covered in various forum posts.

One of the biggest tools to assist you with any oddities is reconciliation. Grab your statement and sit down with YNAB and comb through your statements, its a great way to pick out that incorrectly categorized purchase or the missing 53p you can't find anywhere.

Also, if your bank offers the feature, import and match your bank records. It takes a few minutes to do and can quickly pick out oddities. Also it has the bonus of marking items as "Cleared" so showing you within YNAB your cleared balance, which is handy if you're out and about with just your phone.

Use the community

YNAB has a large and popular forum community, make use of it to answer your questions or generally get support with your financial situation. While its mostly keyed towards the American audience, they'll help you with YNAB. In the UK i'd suggest using Money Saving Expert for asking any financial question without paying for legal or accounting support.

So, I hope this has been helpful to you, i'll keep you posted with how I do in 2014.