tensixtyone

The Effects of Propaganda

Propaganda.

Everyone who hears that word always thinks of the negative conitations, North Korea with the ideal that the rest of the world doesn't exist. Propaganda can be used for numerous things and I'd like to show you a recent use of it by the EVE Online corporation I'm a member of.

This video is to celebrate our recent victory in the Fountain region of New Eden and also Dreddit's (the corporation) first birthday. The video itself had a stunning side effect, applications to join the corporation spiked, fellow redditors were being pulled into the game by our recent success.

Be obviously don't want us

I’ve had a fair share of telecoms related dramas, some self inflicted, some not. This recent one however was a minor fuckup with BT that they “corrected” by reordering the line for us, making us lose our number and stuff.

Be informed us that our line has been “ceased”, and a fee of £10 could be waived if I called them in the next 7 days. I called them today and they informed me of a £25 reconnection fee on top of the £10 reactivation fee. I begrugingly agreed but I was told I had to wait until the “new” phoneline was reactivated anyway and that i’ll need to give them a ring back then.

Less than twenty minutes after getting off the phone with them I got this:

We’re really sorry you don’t have a landline with us anymore, it’s been sad seeing you go. Your last day of service with us was . You’ll get a final bill within the next 60 days or so. If you change your mind, (and let’s face it we’d love you to) you’re welcome back with open arms. Please call us on .

All the best Everyone @ BE

Guess they don’t want me as a customer anymore.

BT Problems

You know something is wrong when you receive a email from your broadband provider saying that my phone line has been disconnected before the phone line provider has even told us.

Django & MultiDB to the rescue

One of the key components of Auth is the ability to communicate with numerous other systems and manage their user authentication systems in situations were we can’t modify the application to use our authentication API. To achieve this we have the Services API, which is a generic interface designed for basic operations such as creating a user, disabling an account as so on. If we wanted to support a new system we write up a simple Python module with these functions and the API does all the required importing and abstraction out for Auth.

In previous versions of the API we handled databases in a very weird way, either by writing our own SQLAlchemy queries or boding the Django ORM to give us a basic cursor to work with, while it was far from perfect it allowed us to edit the databases of other applications without much hassle.

Recently, Django has updated into version 1.2 and with it came the MultiDB functionality which allows you to access multiple databases natively in the ORM. For our database layer this presents some new options not available to us in the old versions.

Using the Django database introspect you are able to generate a Model from a existing database schema, for this example we’re working with Mediawiki’s native database in MySQL. So first of all we need to define the database in our settings.

DATABASES = {
    'dreddit_wiki': {
        'NAME': 'wiki',
        'ENGINE': 'django.db.backends.mysql',
        'USER': 'wiki',
        'PASSWORD': 'passwordgoeshere',
    }
}

Next we fire off the inspect command to produce our database layout

./manage.py inspectdb --database=wiki > wikimodels.py

After a short time you’ll have a fresh Python module with all your database models nearly ready to go, first thing to do would be to edit this file and change any foreign keys to the required Django ForeignKey() field. While inspect does as much as it can it can’t detect foreign keys.

Once your read to rock its a simple case of getting your shell out and giving it a test run.

./manage.py shell
>>> from wiki.wikimodels import User
>>> User.objects.using('wiki').get(user_id=1).user_name
'Matalok'

Simple! No more hassling with db cursors, just simple ORM access without the hassle. The next big leap is defining the database connections at runtime, injecting into the DATABASES variable, by doing this I can remove the problem of having to manage the services’s database connection in the settings.py and instead have them defined on a per service basis.

Django and Celery - Death to Cron

For those who have been following me, you will know that I’ve been working on the techinical side of managing a EVE Online corporation named Dreddit. As part of this I was tasked (what seems a lifetime ago) to create a access management system for the corporation’s services and tools.

EVE Online provides a basic API, allowing for external users to probe the game for information, it is now a common thing for corporations to use these API keys as permission codes, the webapp can connect to the EVE API and deem if the user is allowed to access the service.

The tool I created is simply called Auth, and is designed to be automated as possible allowing users to gain access to their accounts without admin intervention. One of the key processes in Auth is the update job which checks all the API keys and updates the permissions as needed. In a corporation which has in the region of 1700 players and is the biggest corporation in EVE you can imagine that this job is a problem.

On the old design it was a simple cronjob that interated through the keys and updated them, after time this job was taking in the region of a hour to complete so it was split up to complete batches of 50 keys at a time, even after diving the load we’re now still hampered by this job.

Over the last 24 hours we’ve decided to move this job to a task management system, and we’ve picked Celery for its excellent Django integration and its distributed nature, this has allowed us to do something we couldn’t of done before.

The idea is that you break down your common processes into tasks, nuggests that can be shipped to any machine and run in a time-insensitive manner. That wouldn’t work well for Django views and UI, but for the backend processing a app does (sending mails, calling webservices, updating permissions) it takes the need away from running cronjobs or slowing down the user’s response times.

So a few key features are:

Distributed

With RabbitMQ as its backend, its allowed us to spin up specific queue workhorses for Auth, as our infrastructure is based on XenServer this will allow us to add more workers to the processing pool without significant overhead and work, just a simple deployment of the Django app and configuration to point to our RabbitMQ and database instance.

Task Queueing

We can throw those hundreds of keys on the queue, to be processed when available, enabling us to save time and load by not checking the database every five minutes for keys that require a update.

Rate Limiting

This is a key item, Recently CCP decided to start enforcing limits on the EVE API server, if your service makes too many invalid requests then it could end up with a nasty letter in your mailbox.

With Celery we can limit the number of queries for specific tasks, limiting the authenticated requests (like reading your character list) while allowing the anonymous cached requests to be sent through more often.

No more sleep(), no more worrying that CCP will hassle me for hammering the API, bliss!

I’m sure i’ll have a lot more to write about it over time, but for the moment its giving that massive productivity boost we’ve been looking for. Hopefully now this will scale to handle one of the largest alliances in the game as well.

Switch to Posterous

It has been a long time in the making, but finally I got round to switching from Wordpress to Posterous for my blog. Wordpress is a large lumbering beast that I'd rather avoid running now, especially since i'm trying to slim down my current hosting situation.

Posterous' tools made it really simple and all 350 posts from my old blog have been moved over, including the media and other bits and pieces. I'm actually wondering why I didn't do this a long time ago.

High Quality?

Bit rate : 260 Kbps Width : 384 pixels Height : 288 pixels Display aspect ratio : 4:3

Yes, I bet that'll look fantastic on a 32" HDTV...

BBC News - Speaker Bercow's wife in spat with fellow Twitter user

via news.bbc.co.uk

I'm loving this, wife of the Speaker uses Twitter a lot, then when EyeSpyMP starts tracking her she throws a fit over it. Its the unfortunate side-effect of the "web revolution", if you post stuff on a public website expect it to be read by people.

Anyway, her argument is moot, since shes starting to do political commentary via Twitter, which i'm sure people will be interested in:

"Have reflected. 2 questions arise: 1) Will Ashcroft ‘fess up re. tax status. Today. Public have right to know. 2) When is Lady G @ 02? "

"Letter re. BBC bias against Tories (why complain 2 me?). Was actually VERY cross w BBC myself (4 letting BNP on QT). But am fan again now :) "

BT Infinity – Fibre optic broadband from BT

via productsandservices.bt.com

BT Infinity is BT's new FTTC broadband service offering up to 40mbps downstream and 10mbps upstream on their higher package. This all looks amazing until you realise they do network shaping the same as Virgin Media. So you can fully expect that during the weekend you'll have only 20mbps availalbe or even less.

I'll personally wait until this is available to the LLUs, I love Be's products so i'll have a hard time giving them up now after the abmomination that was Virgin Media / NTL.

Tom Watson MP sets the record straight on bad Metro headline