This post relates to programming in general, but specifically to Python (because that’s where I’m working at the moment).
I’ve been working on groundstation lately, which leverages git as it’s storage backend, but also assumes that all operations are threadsafe and concurrent. As such, I want to write testcases for it’s internal implementation that actually excercise these race conditions to assert their sanity.
Obviously, I can litter my code with optional callbacks, but that’s pretty much awful. I never thought I’d genuinely wish python had macros or a preprocessor. Do you know of a sane way to approach this?
I’d like to discard time for a week.
This year I’ve decide to embark more on a quest to explore sleep, and the way I experience it personally. It’s something I’ve struggled with as long as I remember, and I figure that by gathering more data points I can at least start to unwind this puzzle.
A few days ago I was considering an experiment, and this post serves as a braindump to flesh out the idea, as well as hopefully a place to gather feedback and ideally hear from someone who’s tried something like this before.
The basic premise is that for a week I’d like to lose the circadian rhythm, sleep when I feel like it and until I wake up naturally, and only afterwards inspect the data.
This has some immediately obvious flaws, the first being that if I’m to lose all perception of time I’d have to black out my house and not go outside. I suspect that by not deliberately structuring my life anything specific this can be overcome.
In more detail, I plan to pull the clock off the devices I own and the interface for the software I’m likely to use in that time (my system taskbar, irssi and tmux all show my timestamps), and use my cli twitter client twat to tweet like a madman, giving my timestamped data points to look at afterwards.
There’s some planning to be done and some preliminary research to be done, but I’m hoping that at the very least it’ll be interesting, even if it serves only as a cautionary tale to others.
In my last post I unveiled groundstation, a supremely pre-beta cut of a tool I’m building to automagically sync objects in several git repos with any and all nearby peers. Up until tonight, I had been testing with two laptops, connected to the same wireless network (more or less the usecase I envisage).
This evening, I had only my laptop with me: but “Not to worry,” I thought, “I’ll just light up my dev VM!”. At work we use vagrant to light up ondemand VM’s, bootstrap them with babushka and get on with it. We use some trickery in the vagrant-dns gem to make the VM addressible from the host, with vagrant taking care of NAT for us.
Which is where things got interesting. groundstation uses UDP broadcast to find it’s peers, which WILL penetrate most NAT configurations, but with the caveat that the source address will be rewritten- in this instance rewriting it to my external IP address, causing my daemon to attempt to connect to it’s “peer” and sync it’s objects- with itself.
After vanishing off the radar for a few days, I can now unveil groundstation.
Born of a discussion at work about how all existing solutions to issue tracking and project management pretty much suck, and my natural flair for finding the most awkward way imaginable to do something, I started building a framework for decentralised issue management, based on gossip protocols (full points to @wolfeidau for planting that seed in my head) and utilising git’s object store as a backend.
Right now all it can do is propogate git objects over the local network via broadcast discovery, but the TODO list is pretty significant. I’m planning to implement:
- Verification of changesets based on RSA/ECDSA cryptography (choosing those algorithms because of their ubiquity with developers)
- Implementation of an arbiter node making the broadcast discovery an effective local means, but not the only way to “sync” your events
- Better control over “channel” subscription to avoid the obvious DoS attacks that are trivial to exploit with the current implementation
- Better support for event based propagation, instead of polling as currently stands
… and if there’s time maybe a frontend or something so you can actually interact with it. Who can say.
You can try it out right now if you’re ok with not syncing any git objects other than blobs (the only primitive type that I’m planning to use in my tracker).
Clone the sources down on a few local machines on the same subnet,
pip install -r requirements.txt; python stationd
and watch whatever’s in
get synced up.
As I eluded to in yesterday’s post I used doko to do some GPS lookups for mapgit.
After playing with it briefly after lars sent it through, it looked like a much more flexible solution (and one that’d work on more platforms than just OSX, meaning I wasn’t entirely reliant on Mac like I had been).
I haven’t got much to report, beyond having implemented filebacked caching, resolution strategies, some privacy stuff (limit precision) and support for timeouts throughout.
Get doko 0.2.0 from it’s bitbucket repo, and again thanks to lars for spending the time on writing it!
Leaving work on friday to take a few days off, rather than relaxing and drinking like most at this time of year, I had a few ideas for projects I’d like to realise.
At some point on the first night, it occurred to me that a ridiculous but potentially achievable feat might be to build a thing for every day I’m off work. I’ve already missed that goal, but keeping in spirit, I’ll be posting things as they come off the ranks. I’m starting late, and I’ve already gotten a few to a point where they’re worth talking about, so hopefully this will be enough filler to allow me to post daily until I go back to work. Which brings us to..
A while ago I started geotagging my commits with a post-commit hook and a tool called whereami. I silently collected some (massively skewed- due to an oversight on my part) data for a while, and then remembered about it recently when lars whom I work with kindly offered to do some plotting magics in R.
He came up with this:
Which ultimately inspired me to start looking into this again and produce a more general solution.
I built http://mapgit.com in reasonably short order the next day. Right now it’s basically just a thin layer around redis which allows you to upload pairs of “commit location” pairs and have them transparently stored in redis.
Before the break is over I intend to built on the side of the github API to allow for selecting all the locations (both as a set and as a distribution) for a given rev-list or branch, fetched directly from github’s API or passed straight into mapgit, and export them in a format that R likes for easy plotting.
So far it’s quite barebones, but I’m happy with how it turned out for only a few hours invested.
Until tomorrow, Happy Haxmas all.