chrishowie.com

I’ve already sent out a message to those I contact frequently, but for the rest of you, my email address has changed from cdhowie@gmail.com to me@chrishowie.com.

I have decided to migrate away from Gmail for a variety of reasons. I figured I’d use my domain so that I have flexibility in my provider choice. For example, if I decide to change my mail provider I don’t have to get a new address. So this change should be permanent.

Now, being a tinkerer, I figured I’d set up a mail system for myself that retains the features I like from Gmail while doing away with the downsides of using Gmail. My final system is complicated, but effective. And it was a fun four days setting it up! (No, that wasn’t sarcasm. This is the kind of thing I enjoy.)

My MTA is Postfix, running on mail.chrishowie.com. It accepts mail for me and delivers mail from me. The standard security features are in place: SPF/blacklist checking and no unauthenticated relaying. I also established SPF records for my domain. I do not have a spam filter, but I might set up SpamAssassin later if I actually start seeing spam in my inbox; no spam has made it past the sender blacklist check yet.

For downloading mail, I set up Courier as a POP3 server. The observant will note that this would nullify one of the most useful aspects of Gmail: access to your email from anywhere. Once you download from a POP3 server and delete, your mail is gone from the server and lives in your mail client.

That’s where the rest of the rig comes into play. On my home LAN server I have a multi-piece system that provides me with this anywhere-access. I have a getmail4 cron job that fetches mail from my POP3 server, as well as from my Gmail account (so people can still reach me using my Gmail address), and some of my other mail accounts, and delivers the mail to my maildir using Dovecot’s delivery agent. From there, the agent processes my sieve rules, sorting my mail into various folders. (It’s like your favorite mail program’s “filters” only it runs on my server instead of my mail clients, so the configuration is centralized.)

For reading all this mail, I run Dovecot, an IMAP server. All my mail clients fetch mail from this server, and since it is IMAP, changes to messages (like moving a message between folders, or adding tags, or whatever) are actually pushed back to the server. So I can use several mail clients at once and they all have a consistent view of my mailbox.

The only major piece I have yet to set up is an LDAP server for centralization of my address book. The rest has been working quite well. I can use Thunderbird at home, or my mobile phone’s email client when I’m out of the house. No limiting or inconvenient web interfaces required.

Or at least as they are currently implemented. (Rambling rant alert!)

Almost every IM client on the planet has the concept of a group. You create a group, give it a name, and populate it with your contacts. At first glance, this is useful.

Having used IM clients since the latter years of AIM’s domination, I have consistently struggled with the best way to group my contacts. To this day I have no solution. Why?

Because the group model is inherently broken in that every major IM client out there allows one group per contact. This assumes that each person I know has exactly one type of relationship to me. A contact cannot be a friend, coworker, family member, and developer. You must pick one.

This problem is exacerbated by multi-protocol clients. Not through any specific fault of their own, but when they support connecting to many different services, you suddenly wind up with an overwhelming number of groups with no coherent purpose. If you didn’t name your groups exactly the same, or you used a different grouping paradigm, you are SOL and must merge the groups somehow. This becomes an impossible mess when the service on the other end (Facebook, for example) allows contacts to be in multiple groups. Which group they wind up in on your IM client may as well be random, and may not even be the same each time you connect. (Edit: Since writing this I have discovered that the XMPP server operated by Facebook does actually indicate to clients when contacts are in more than one group. Pidgin honors this.)

What we need is a cross-service “tagging” mechanism that lets me say “this person is a coworker at X company, and worked with me on Y open-source project.” Then, whether I am looking for coworkers at X or codevelopers on Y, this person shows up. Rules could be established so that “coworker at X company” membership implies “coworker” membership. This may at first glance seem like nested groups, but it is very different. It is completely free-form, and allows me to model my contact structure after how I interact with people in real life. Not some bizarre mutually-exclusive set of too-shallow or too-specific relationships.

Multi-protocol clients should then support tag equivalency, so that service-mandated groups can be rolled into a differently-named tag. Or ignore one account’s groups entirely, with everyone being rolled into one tag, with the possibility of local memorization of extra tags.

But the way things stand now, having groups at all is actually a hindrance to my communication. “Did I put that person in friends or coworkers? Or are they still in the service-specific default friends/buddies/oxygen-converters group?”

In lieu of a real solution to this problem, I want to disable groups entirely. Except… whoops… Pidgin doesn’t even let me do that.

I welcome everyone’s thoughts on this issue.

I am migrating to a new OpenPGP key. If this concerns you, or if you are interested, please read my key transition statement and take appropriate action. Please verify that the transition is signed by both my old and new keys.

At work, we use Subversion for source control. This is quite the popular VCS, but I’ve grown accustomed to (and much prefer) Git. Don’t get me wrong, SVN has its advantages, but since using Git my workflow has changed quite radically, and it’s difficult to revert to the rather inflexible and tedious SVN workflow. Anyway, I’ve been using git-svn for the past month or so, and thought I’d share some of my practices.

In my clone, master is my “local” storage branch. I use it to version things like my .gitignore, and my commit message template. I would also use it for my dcommit/rebase scripts too, but since this is on Windows, Git becomes angry when it attempts to remove scripts that are executing.

Master is then the common root for my topic branches. I’ll do some work and commit, then do more work, as the usual Git workflow goes. The ability to create local branches and commits is great for several reasons:

First, I can commit much more often, without fearing that I will break somebody else’s working copy — I do frequently commit broken code now, because the commits don’t get sent to SVN automatically.

Second, and a side effect of the above, I am much more agile. Sometimes I’ll be working on two projects at once, and keeping separate branches for them means that the broken state of one branch does not affect my ability to build/debug another. This means I can even drop everything (after a commit or stash) and help out with an urgent QA or support issue, without either having lots of uncommitted work interfering or committing broken code to a production repository first.

Third, I can version changes I make to code that might make my life easier, but that would require approval to commit. I have not yet done this, but thanks to my local repository, it’s an option.

Fourth, I can develop against a stable codebase. If I need a specific fix from SVN to work, I can cherry-pick it into my topic branch — fetching commits from SVN does not mandate that my code must be merged with them. Along the same lines, I also have the ability to rebase all my topic branches against a specific SVN commit, which is great when somebody commits broken code that causes build errors or incorrect runtime behavior. If I were using SVN, I would either have to fix it myself, wait for the owner of the code to fix it, or revert to an earlier commit. And since commits are linear, that might mean I would lose some of my code. Git allows me to retain my commits while backing out changes made by others. And if I did already commit some of my changes after the broken code, I can still rebase master and cherry-pick my subsequent commits onto a new branch and continue my work.

And finally, I can review all my commits before pushing them out. I’ve had several occasions to fix up commit messages already.

Of course, I do eventually need to commit to SVN. I have a few scripts to help here. Obviously I don’t want to commit my local-only stuff in master. So I have one script that simply does git svn fetch && git rebase --onto git-svn master. For the Git-impaired, this fetches any new commits from SVN and adds corresponding Git commits on the git-svn branch. The rebase command then takes all the commits on the current branch since master’s latest commit, and applies them as patches to the tip of the git-svn branch, creating a new commit for each. This effectively removes the local changes I’ve made in master from the commits, as well as merges the changes in SVN into my commits. (If one of my commits fails to apply as a patch, then I have to resolve the conflicts manually, just like an SVN update. But in this case, I still retain all my individual commits.)

After running this script, I then git svn dcommit, which commits my local commits to SVN, one by one.

At this point, I usually run my post-commit script, which rebases master on to git-svn, then rebases all my other branches on top of master. I might not do this if I don’t want to rebase some of my other topic branches yet.

In closing, I leave you with my trick for changing commit messages. I used to use git filter-branch for this, which is like using a sledgehammer for surgery. (My similes are awesome, I know.) Now I use this process:

git checkout -b temp $COMMIT_TO_CHANGE (Create a new branch called “temp” at the commit I’m changing, and switch to it.)

git commit --amend (Open an editor to amend my commit. I change to message here, then save and quit. The temp branch now has the new commit, whose parent commit is the same as it was before being amended.)

git checkout $ORIGINAL_BRANCH (Switch back to the branch we are amending.)

git rebase temp (Rebase the branch on top of the amended commit. Since the original commit will be applied on top of the amended commit, it is dropped during the rebase. The other commits will apply with no conflicts. The history is now corrected.)

git branch -d temp (Remove our temporary branch.)

And there you have it. In a history with many thousands of commits, this is much faster than git filter-branch, and for those relatively fluent in Git, is also easier to remember how to do.

Update 2010-02-08: Jonathan Pryor has merged many of my extension methods into Cadenza. I’d strongly suggest checking it out.

It’s no secret to my friends that I love to program… even more so as I’ve been developing a FarmVille client in C# and having them test it. (As much as you might hate FarmVille, you must agree that there’s a certain awesome factor in LINQ-to-FarmVille: service.Plow(from i in service.World.Objects.OfType<Plot>() where i.State == "fallow" || i.State == "withered" select i)

Well, among this project and others I consistently find myself writing the same code over and over. I know of many programmers who have developed personal toolkits for the languages they use frequently, but for some reason I’ve been programming for about 13-15 years now and haven’t ever built my own library. This application gave me an excuse to do so, and so I’ve started on the Cdh.Toolkit suite of libraries.

Here is a summary of the classes available:

• Cdh.Toolkit.Collections: Some useful collection types, all designed to be derived.
  • ReadOnlyCollection<T>, ReadOnlyDictionary<TKey, TValue>, and ReadOnlyList<T>: Wrappers around the corresponding interface types ICollection<T>, IDictionary<TKey, TValue>, and IList<T>, throwing exceptions on all write attempts. While there is a ReadOnlyCollection<T> as part of the .NET framework, it is not designed to be derived, and the other two classes do not have a counterpart at all.
  • SynchronizedCollection<T>, SynchronizedDictionary<TKey, TValue>, and SynchronizedList<T>: Wrappers around the corresponding interfaces. All accesses are synchronized against a ReaderWriterLockSlim, allowing for multiple concurrent read operations. The enumeration behavior can be specified as either lock, which holds a read lock for the duration of the enumeration, or copy, which creates a copy of the collection and enumerates it instead.
  • ObservableCollection<T>: A collection that fires events when modified. ObservableDictionary and ObservableList are currently not provided, due to some implementation complexities. However, the interfaces IObservableCollection<T>, IObservableDictionary<TKey, TValue>, and IObservableList<T> and some EventArgs classes are provided to allow developers to implement their own observable collections easily.
  • ObservableHashSet<T>: An observable and API-compatible wrapper around HashSet<T>.
  • ReadOnlyObservableCollection<T>, ReadOnlyObservableDictionary<TKey, TValue>, and ReadOnlyObservableList<T>: Wrappers around the IObservable* interfaces mentioned above. Events from the wrapped collections are forwarded. This allows one to have a read only observable collection without sacrificing the IObservable* interface, which would happen if such a collection were wrapped in one of the normal read only classes listed above.
• Cdh.Toolkit.Extensions: Extension libraries designed to ease the use of many classes in the .NET framework.
  • Collections: Extensions specific to collection classes.
    • TValue IDictionary<TKey, TValue>.GetOrDefault(TKey key): Returns default(TValue) if the key is not present in the dictionary.
    • TValue IDictionary<TKey, TValue>.GetOrValue(TKey key, TValue fallback): Returns fallback if the key is not present in the dictionary.
  • ComponentModel: Extensions specific to the System.ComponentModel namespace.
    • void ISynchronizeInvoke.AutoInvoke(Action action): Executes the action delegate on the ISynchronizeInvoke object if required, otherwise does so on the current thread.
    • object ISynchronizeInvoke.AutoInvoke(Delegate method, params object[] args): Executes the delegate on the ISynchronizeInvoke object if required, otherwise does so on the current thread, and returns the return value of that method in either case.
    • AsyncCallback AsyncCallback.Invoked(ISynchronizeInvoke obj): Returns a wrapper around the AsyncCallback delegate that will invoke it using the AutoInvoke extension above. Useful for async callbacks that need to operate on a Winforms GUI.
  • Enumerable: Extensions to enumerable objects.
    • IEnumerable<T> IEnumerable<T?>.NotNull() where T : struct: Returns all values from the non-null nullable objects in the sequence.
    • void IEnumerable<T>.Walk(): Enumerates the sequence, discarding all values obtained. Useful as an alternative to .ToList() when you need to make sure that a query executes, but do not need to use the result.
    • void IEnumerable<T>.CopyInto(IList<T> list): Copies a sequence into a list.
  • Events: Extensions that make writing event logic easier. All of these extensions return if the event handler in question is null, making event-firing code simpler and easier to read.
    • void EventHandler.Fire(object sender): Uses EventArgs.Empty as the event arguments.
    • void EventHandler.Fire(object sender, EventArgs args)
    • void EventHandler.Fire(object sender, Func<EventArgs> argsFactory): Calls the factory function only if the event handler is not null. Useful when construction of the event arguments can take a long time.
    • void EventHandler<T>.Fire(object sender, T args)
    • void EventHandler<T>.Fire(object sender, Func<T> argsFactory): Calls the factory function only if the event handler is not null. Useful when construction of the event arguments can take a long time.
  • ReaderWriterLockSlim: Allows these kind of locks to be used in a using() block, which makes code easier to read and maintain. They will also return a no-op IDisposable if a compatible lock is already held by the current thread, which makes non-recursive lock objects simpler to code with. (The return type is actually a value type that implements IDisposable, which means that usage of these methods does not incur any object allocation overhead.)
    • IDisposable ReaderWriterLockSlim.Read(): Returns an IDisposable that will release the read lock when disposed. This method returns a no-op IDisposable instead if the current thread already holds a read, upgradeable read, or write lock.
    • IDisposable ReaderWriterLockSlim.UpgradeableRead(): Returns an IDisposable that will release the upgradeable read lock when disposed. This method returns a no-op IDisposable instead if the current thread already holds an upgradeable read or write lock.
    • IDisposable ReaderWriterLockSlim.Write(): Returns an IDisposable that will release the write lock when disposed. This method returns a no-op IDisposable instead if the current thread already holds a write lock.
  • Reflection
    • T ICustomAttributeProvider.GetCustomAttribute<T>(bool inherit) where T : Attribute: Returns a typed attribute, or null if there is no attribute of type T.
    • IEnumerable<T> ICustomAttributeProvider.GetCustomAttributes<T>(bool inherit) where T : Attribute: Returns a sequence of attributes of type T present on the attribute provider.
  • Reflection.Emit
    • void ILGenerator.EmitTypeOf(Type type): Emits the IL sequence that will leave the same Type object on the execution stack.

  The amount of code is slim, but I’ve found at least one of the classes or extensions invaluable in every project I’ve worked on since starting the toolkit. It’s an interesting case where coding for a game actually winds up improving my productivity working on other software too.

  Eventually these libraries will be released under the MIT license, so stay tuned for another blog post with a link to the Git repository.

  (And yes, the above list will be converted into real documentation. Someday.)

One of the nice things that .NET 3.5 gives us is LINQ, which gives new life to the often-neglected IEnumerable generic interface. Sequence processing is now a first-class citizen in the C# world, and this is a good thing. However, it can be very tricky to design a usable API around enumerables. Today I present my solution to an annoying (but not showstopping) hurdle.

Consider the case where you have several types implementing an interface. In my case, these types all have a common ancestor, but this is beside the point. We’ll call the interface IFoo, and the classes implementing this interface ThingOne, ThingTwo, and ThingThree.

If I have a method that acts on a series of IFoo objects, it is tempting to accept IEnumerable<IFoo> as an argument. It makes sense, right? Well, sort of. Your users will not like this, because IEnumerable<ThingOne>, IEnumerable<ThingTwo>, and IEnumerable<ThingThree> are not convertible to IEnumerable<IFoo>. While it’s not too annoying, your users will have to cope by invoking Cast<IFoo>() on their enumerables for them to work with your API. This not only adds code overhead (read: more code to maintain) but a minor amount of CPU and memory overhead to create an object that is going to cast objects to an interface that they explicitly implement.

The solution is rather simple, but not very obvious at first glance. Instead of using the signature void OperateOnFoos(IEnumerable<IFoo> foos), use this instead: void OperateOnFoos<T>(IEnumerable<T> foos) where T : IFoo. It is a simple change, and the method will work exactly the same as before, except your users will no longer be required to cast their enumerables to IFoo.

This technique applies just as well to situations where you take an enumerable to a class that is designed to be subclassed.

Now depending on how generics are implemented in your runtime of choice, you’re still probably going to see a small memory hit for each different T you use when calling this method. But it’s not likely to be anywhere near the cost of creating a bunch of cast-enumerables that you really could do without. And that aside, the convenience of not having to Cast<IFoo>() enumerables is totally worth changing one line of code.

.NET 4.0 will likely render this mechanism obsolete with the introduction of covariant interfaces, but in the meantime let’s all do something nice for our users!

I’ve got this idea to code some game servers for a series of cooperative games my brother and I used to play as kids. I get similar ideas all the time… how about a game server for this card game or that board game? The problem I run into is pretty much always exactly the same: what communication protocol do I use?

I decided on a list of criteria that this protocol, whatever it is, must meet:

1. It must be portable across programming languages and runtimes. If somebody else wants to write a better client using a different environment, that should be straightforward — perhaps not necessarily easy, but at least straightforward.
2. It must be relatively efficient on the wire. Protocol chatter should be minimal in comparison to the data being exchanged.
3. The object library should be simple and elegant to code against. When writing my game, the last thing I want to worry about is silly protocol details. Just get my message to the other computer please.

And here are the existing protocols I considered:

• .NET remoting. Since I code most in C# these days, it seemed like a logical choice. But it very blatantly breaks criterion 1 when using the binary formatter, and breaks both 1 and 2 when using the SOAP formatter.
• SOAP web service. Criterion 3 is satisfied, until you get to session persistence details. Criterion 1 is satisfied, and criterion 2… not so much.
• XML-RPC. Criterion 1 is met, and 2 is somewhat met. But criterion 3 is not — XML-RPC does not define any mechanism for dealing with persistent sessions. I would have to spend time writing a session manager with expiration and whatnot. No thanks.

And I’m sure I looked at others. The point is, for something as simple as message-passing between a game client and server, there doesn’t appear to be much out there that satisfies my requirements. And this is something I’ve come back to frequently.

Well, after several years of mulling the problem over in my subconscious, I knuckled down and coded. I have a usable library after two days of development. (And we’re talking maybe a few hours per day.) Written in C#, it allows any CIL-based language to write simple message-based client/server programs in very small amounts of code. For a quick example, let’s create a server that will convert strings to uppercase, with tracing back to the client.

First, we need to create an interface library so that the client and server know what each other’s methods are:

using System;
using Cdh.SimpleRpc;

namespace ServiceTest.Interfaces {
    public interface IServer {
        [RpcMethod] string ToUppercase(string str);
    }

    public interface IClient {
        [RpcMethod] void Trace(string message);
    }
}

Now, here is the client:

using System;
using System.IO;
using Cdh.SimpleRpc;
using ServiceTest.Interfaces;

namespace ServiceTest.Client {
    public class MainClass {
        public static void Main() {
            Stream serverStream = ConnectToServer();

            var service = new RpcService<IClient, IServer>(new Client(), serverStream);

            new Thread(delegate { while(service.Read()); }).Start();

            IServer server = service.RemoteServerProxy;
            Console.WriteLine("ToUppercase result: " + server.ToUppercase("this is a test"));

            serverStream.Close();
        }

        private Stream ConnectToServer() {
            // Here is your code to connect to the server endpoint.
        }
    }

    internal class Client : IClient {
        public void Trace(string message) {
            Console.WriteLine("Server trace: " + message);
        }
    }
}

Note that the RpcService object generates a typed object that will transparently proxy calls to the remote service. The server program is almost as simple:

using System;
using System.IO;
using Cdh.SimpleRpc;
using ServiceTest.Interfaces;

namespace ServiceTest.Server {
    public class MainClass {
        public static void Main() {
            Stream clientStream = AcceptConnection();

            Server server = new Server();
            var service = new RpcService<IServer, IClient>(server, clientStream);
            server.client = service.RemoteServerProxy;

            while(service.Read());

            clientStream.Close();
        }

        private Stream AcceptConnection() {
            // Here is your code to accept a client connection.
        }
    }

    internal class Server : IServer {
        public IClient client;

        public string ToUppercase(string str) {
            client.Trace("Entering ToUppercase");
            str = str.ToUpper();
            client.Trace("Leaving ToUppercase");

            return str;
        }
    }
}

Ta-da. Some closing notes about this library:

• It should be completely thread-safe, and will allow you to place calls using the proxy objects from multiple threads. The calls will block until a response is returned from the remote service.
• Yes, you can throw exceptions in a service method, and yes, it will cause an exception to be thrown remotely from the proxy object.
• In the future it may be possible to flag service methods that return void as “no response” calls, which will cause the proxy call to return immediately. Of course, you will not be notified if an exception is thrown remotely.
• This API doesn’t do any complex serialization, and will only operate on the primitive types, excluding IntPtr. It will probably allow transmission of arrays at some point, and perhaps allow custom objects too.

Looking back on the list of criteria, this library, even in the early stages of development, easily meets all three. I’ll be hacking on it some more I’m sure, and may even publish the Git repository somewhere, when I’m confident that the code doesn’t totally suck.

Come to think of it, I can’t name any either…

I need to get into the habit of blogging more often. I haven’t even been twittering much lately…

You’ve probably noticed the visual update to my blog by now. I got tired of the default WordPress theme. I had to tweak this one a bit to get it to behave the way I want, but overall it’s pretty nice. A few weeks ago I added the live chat widget as well, which so far has attracted comments from exactly two people. Come on, I know there’s more of you out there!

My new job is going well. The current project I’m working on is a migration script to fix some datetimes that may have been incorrectly converted to GMT. If you’ve done any programming around timezone conversions, you’ll know it’s a blast! … Ok, it’s not that bad. It’s actually kind of fun, in a weird way.

On the “my projects” front, I’ve converted the OpenVP Subversion repository to two separate Git repositories, and created a third for the metadata pipeline project. Check them out on Gitorious.

I’ve started playing Black & White on my lunch break. The voice acting is a little iffy, but the gameplay is good after you figure out what you’re doing. (Which, ironically, doesn’t happen until you leave the tutorial island.) Oh, and I got the Metroid Prime Trilogy for Wii. I’ve only played the first on GameCube, but absolutely loved it. Can’t wait to tackle the two sequels.

That’s all for now.

There have been many good changes in my life recently. It’s been almost three months since I graduated college, and nearly two since I got married. Well, now I have another for the list.

For two years I’ve been working at Ontario Systems as an intern. The first year and a half were a lot of fun. I was coding PHP and Flex/ActionScript on a team of two other developers (one also my manager). However, I’ve been unable to secure a full-time job offer since graduating and have been continuing work as an intern, which pays enough to cover the bills, but not much more. Additionally, the company’s partnership with Microsoft means that pretty much all of the Linux servers are in the cross-hairs, so to speak. This is not necessarily a bad thing, but it does mean that my current role of PHP developer and interim Linux sysadmin is not exactly going to be permanent.

Last week I had the privilege of accepting a full-time job offer from Aprimo and will be starting work with them on July 27th. They are also a Microsoft shop, but I will be working on the development team and coding C#, which I’ve been hacking with for some years now. The company looks to be doing well, and the people I’ve met so far have been very friendly and welcoming. The developers I’ve talked to all seem very knowledgeable, and I’m looking forward to working with them.

I’m not sure which I’m excited about more: getting to work at Aprimo, or finally ending my job hunt. Both are pretty cool. I will miss my friends at Ontario Systems (we’ve had some pretty good times) but I am definitely looking forward to the future.