chrishowie.com

Update 2010-02-08: Jonathan Pryor has merged many of my extension methods into Cadenza. I’d strongly suggest checking it out.

It’s no secret to my friends that I love to program… even more so as I’ve been developing a FarmVille client in C# and having them test it. (As much as you might hate FarmVille, you must agree that there’s a certain awesome factor in LINQ-to-FarmVille: service.Plow(from i in service.World.Objects.OfType<Plot>() where i.State == "fallow" || i.State == "withered" select i)

Well, among this project and others I consistently find myself writing the same code over and over. I know of many programmers who have developed personal toolkits for the languages they use frequently, but for some reason I’ve been programming for about 13-15 years now and haven’t ever built my own library. This application gave me an excuse to do so, and so I’ve started on the Cdh.Toolkit suite of libraries.

Here is a summary of the classes available:

• Cdh.Toolkit.Collections: Some useful collection types, all designed to be derived.
  • ReadOnlyCollection<T>, ReadOnlyDictionary<TKey, TValue>, and ReadOnlyList<T>: Wrappers around the corresponding interface types ICollection<T>, IDictionary<TKey, TValue>, and IList<T>, throwing exceptions on all write attempts. While there is a ReadOnlyCollection<T> as part of the .NET framework, it is not designed to be derived, and the other two classes do not have a counterpart at all.
  • SynchronizedCollection<T>, SynchronizedDictionary<TKey, TValue>, and SynchronizedList<T>: Wrappers around the corresponding interfaces. All accesses are synchronized against a ReaderWriterLockSlim, allowing for multiple concurrent read operations. The enumeration behavior can be specified as either lock, which holds a read lock for the duration of the enumeration, or copy, which creates a copy of the collection and enumerates it instead.
  • ObservableCollection<T>: A collection that fires events when modified. ObservableDictionary and ObservableList are currently not provided, due to some implementation complexities. However, the interfaces IObservableCollection<T>, IObservableDictionary<TKey, TValue>, and IObservableList<T> and some EventArgs classes are provided to allow developers to implement their own observable collections easily.
  • ObservableHashSet<T>: An observable and API-compatible wrapper around HashSet<T>.
  • ReadOnlyObservableCollection<T>, ReadOnlyObservableDictionary<TKey, TValue>, and ReadOnlyObservableList<T>: Wrappers around the IObservable* interfaces mentioned above. Events from the wrapped collections are forwarded. This allows one to have a read only observable collection without sacrificing the IObservable* interface, which would happen if such a collection were wrapped in one of the normal read only classes listed above.
• Cdh.Toolkit.Extensions: Extension libraries designed to ease the use of many classes in the .NET framework.
  • Collections: Extensions specific to collection classes.
    • TValue IDictionary<TKey, TValue>.GetOrDefault(TKey key): Returns default(TValue) if the key is not present in the dictionary.
    • TValue IDictionary<TKey, TValue>.GetOrValue(TKey key, TValue fallback): Returns fallback if the key is not present in the dictionary.
  • ComponentModel: Extensions specific to the System.ComponentModel namespace.
    • void ISynchronizeInvoke.AutoInvoke(Action action): Executes the action delegate on the ISynchronizeInvoke object if required, otherwise does so on the current thread.
    • object ISynchronizeInvoke.AutoInvoke(Delegate method, params object[] args): Executes the delegate on the ISynchronizeInvoke object if required, otherwise does so on the current thread, and returns the return value of that method in either case.
    • AsyncCallback AsyncCallback.Invoked(ISynchronizeInvoke obj): Returns a wrapper around the AsyncCallback delegate that will invoke it using the AutoInvoke extension above. Useful for async callbacks that need to operate on a Winforms GUI.
  • Enumerable: Extensions to enumerable objects.
    • IEnumerable<T> IEnumerable<T?>.NotNull() where T : struct: Returns all values from the non-null nullable objects in the sequence.
    • void IEnumerable<T>.Walk(): Enumerates the sequence, discarding all values obtained. Useful as an alternative to .ToList() when you need to make sure that a query executes, but do not need to use the result.
    • void IEnumerable<T>.CopyInto(IList<T> list): Copies a sequence into a list.
  • Events: Extensions that make writing event logic easier. All of these extensions return if the event handler in question is null, making event-firing code simpler and easier to read.
    • void EventHandler.Fire(object sender): Uses EventArgs.Empty as the event arguments.
    • void EventHandler.Fire(object sender, EventArgs args)
    • void EventHandler.Fire(object sender, Func<EventArgs> argsFactory): Calls the factory function only if the event handler is not null. Useful when construction of the event arguments can take a long time.
    • void EventHandler<T>.Fire(object sender, T args)
    • void EventHandler<T>.Fire(object sender, Func<T> argsFactory): Calls the factory function only if the event handler is not null. Useful when construction of the event arguments can take a long time.
  • ReaderWriterLockSlim: Allows these kind of locks to be used in a using() block, which makes code easier to read and maintain. They will also return a no-op IDisposable if a compatible lock is already held by the current thread, which makes non-recursive lock objects simpler to code with. (The return type is actually a value type that implements IDisposable, which means that usage of these methods does not incur any object allocation overhead.)
    • IDisposable ReaderWriterLockSlim.Read(): Returns an IDisposable that will release the read lock when disposed. This method returns a no-op IDisposable instead if the current thread already holds a read, upgradeable read, or write lock.
    • IDisposable ReaderWriterLockSlim.UpgradeableRead(): Returns an IDisposable that will release the upgradeable read lock when disposed. This method returns a no-op IDisposable instead if the current thread already holds an upgradeable read or write lock.
    • IDisposable ReaderWriterLockSlim.Write(): Returns an IDisposable that will release the write lock when disposed. This method returns a no-op IDisposable instead if the current thread already holds a write lock.
  • Reflection
    • T ICustomAttributeProvider.GetCustomAttribute<T>(bool inherit) where T : Attribute: Returns a typed attribute, or null if there is no attribute of type T.
    • IEnumerable<T> ICustomAttributeProvider.GetCustomAttributes<T>(bool inherit) where T : Attribute: Returns a sequence of attributes of type T present on the attribute provider.
  • Reflection.Emit
    • void ILGenerator.EmitTypeOf(Type type): Emits the IL sequence that will leave the same Type object on the execution stack.

  The amount of code is slim, but I’ve found at least one of the classes or extensions invaluable in every project I’ve worked on since starting the toolkit. It’s an interesting case where coding for a game actually winds up improving my productivity working on other software too.

  Eventually these libraries will be released under the MIT license, so stay tuned for another blog post with a link to the Git repository.

  (And yes, the above list will be converted into real documentation. Someday.)

One of the nice things that .NET 3.5 gives us is LINQ, which gives new life to the often-neglected IEnumerable generic interface. Sequence processing is now a first-class citizen in the C# world, and this is a good thing. However, it can be very tricky to design a usable API around enumerables. Today I present my solution to an annoying (but not showstopping) hurdle.

Consider the case where you have several types implementing an interface. In my case, these types all have a common ancestor, but this is beside the point. We’ll call the interface IFoo, and the classes implementing this interface ThingOne, ThingTwo, and ThingThree.

If I have a method that acts on a series of IFoo objects, it is tempting to accept IEnumerable<IFoo> as an argument. It makes sense, right? Well, sort of. Your users will not like this, because IEnumerable<ThingOne>, IEnumerable<ThingTwo>, and IEnumerable<ThingThree> are not convertible to IEnumerable<IFoo>. While it’s not too annoying, your users will have to cope by invoking Cast<IFoo>() on their enumerables for them to work with your API. This not only adds code overhead (read: more code to maintain) but a minor amount of CPU and memory overhead to create an object that is going to cast objects to an interface that they explicitly implement.

The solution is rather simple, but not very obvious at first glance. Instead of using the signature void OperateOnFoos(IEnumerable<IFoo> foos), use this instead: void OperateOnFoos<T>(IEnumerable<T> foos) where T : IFoo. It is a simple change, and the method will work exactly the same as before, except your users will no longer be required to cast their enumerables to IFoo.

This technique applies just as well to situations where you take an enumerable to a class that is designed to be subclassed.

Now depending on how generics are implemented in your runtime of choice, you’re still probably going to see a small memory hit for each different T you use when calling this method. But it’s not likely to be anywhere near the cost of creating a bunch of cast-enumerables that you really could do without. And that aside, the convenience of not having to Cast<IFoo>() enumerables is totally worth changing one line of code.

.NET 4.0 will likely render this mechanism obsolete with the introduction of covariant interfaces, but in the meantime let’s all do something nice for our users!

I’ve got this idea to code some game servers for a series of cooperative games my brother and I used to play as kids. I get similar ideas all the time… how about a game server for this card game or that board game? The problem I run into is pretty much always exactly the same: what communication protocol do I use?

I decided on a list of criteria that this protocol, whatever it is, must meet:

1. It must be portable across programming languages and runtimes. If somebody else wants to write a better client using a different environment, that should be straightforward — perhaps not necessarily easy, but at least straightforward.
2. It must be relatively efficient on the wire. Protocol chatter should be minimal in comparison to the data being exchanged.
3. The object library should be simple and elegant to code against. When writing my game, the last thing I want to worry about is silly protocol details. Just get my message to the other computer please.

And here are the existing protocols I considered:

• .NET remoting. Since I code most in C# these days, it seemed like a logical choice. But it very blatantly breaks criterion 1 when using the binary formatter, and breaks both 1 and 2 when using the SOAP formatter.
• SOAP web service. Criterion 3 is satisfied, until you get to session persistence details. Criterion 1 is satisfied, and criterion 2… not so much.
• XML-RPC. Criterion 1 is met, and 2 is somewhat met. But criterion 3 is not — XML-RPC does not define any mechanism for dealing with persistent sessions. I would have to spend time writing a session manager with expiration and whatnot. No thanks.

And I’m sure I looked at others. The point is, for something as simple as message-passing between a game client and server, there doesn’t appear to be much out there that satisfies my requirements. And this is something I’ve come back to frequently.

Well, after several years of mulling the problem over in my subconscious, I knuckled down and coded. I have a usable library after two days of development. (And we’re talking maybe a few hours per day.) Written in C#, it allows any CIL-based language to write simple message-based client/server programs in very small amounts of code. For a quick example, let’s create a server that will convert strings to uppercase, with tracing back to the client.

First, we need to create an interface library so that the client and server know what each other’s methods are:

using System;
using Cdh.SimpleRpc;

namespace ServiceTest.Interfaces {
    public interface IServer {
        [RpcMethod] string ToUppercase(string str);
    }

    public interface IClient {
        [RpcMethod] void Trace(string message);
    }
}

Now, here is the client:

using System;
using System.IO;
using Cdh.SimpleRpc;
using ServiceTest.Interfaces;

namespace ServiceTest.Client {
    public class MainClass {
        public static void Main() {
            Stream serverStream = ConnectToServer();

            var service = new RpcService<IClient, IServer>(new Client(), serverStream);

            new Thread(delegate { while(service.Read()); }).Start();

            IServer server = service.RemoteServerProxy;
            Console.WriteLine("ToUppercase result: " + server.ToUppercase("this is a test"));

            serverStream.Close();
        }

        private Stream ConnectToServer() {
            // Here is your code to connect to the server endpoint.
        }
    }

    internal class Client : IClient {
        public void Trace(string message) {
            Console.WriteLine("Server trace: " + message);
        }
    }
}

Note that the RpcService object generates a typed object that will transparently proxy calls to the remote service. The server program is almost as simple:

using System;
using System.IO;
using Cdh.SimpleRpc;
using ServiceTest.Interfaces;

namespace ServiceTest.Server {
    public class MainClass {
        public static void Main() {
            Stream clientStream = AcceptConnection();

            Server server = new Server();
            var service = new RpcService<IServer, IClient>(server, clientStream);
            server.client = service.RemoteServerProxy;

            while(service.Read());

            clientStream.Close();
        }

        private Stream AcceptConnection() {
            // Here is your code to accept a client connection.
        }
    }

    internal class Server : IServer {
        public IClient client;

        public string ToUppercase(string str) {
            client.Trace("Entering ToUppercase");
            str = str.ToUpper();
            client.Trace("Leaving ToUppercase");

            return str;
        }
    }
}

Ta-da. Some closing notes about this library:

• It should be completely thread-safe, and will allow you to place calls using the proxy objects from multiple threads. The calls will block until a response is returned from the remote service.
• Yes, you can throw exceptions in a service method, and yes, it will cause an exception to be thrown remotely from the proxy object.
• In the future it may be possible to flag service methods that return void as “no response” calls, which will cause the proxy call to return immediately. Of course, you will not be notified if an exception is thrown remotely.
• This API doesn’t do any complex serialization, and will only operate on the primitive types, excluding IntPtr. It will probably allow transmission of arrays at some point, and perhaps allow custom objects too.

Looking back on the list of criteria, this library, even in the early stages of development, easily meets all three. I’ll be hacking on it some more I’m sure, and may even publish the Git repository somewhere, when I’m confident that the code doesn’t totally suck.

Well it’s hacking season again. With GNOME’s switch from Subversion to Git complete, which means Banshee now uses Git too, it gave me an excuse to finally learn it. This was not fun. But having toughed it out, I can definitely say that I love it.

Now that Banshee is using Git, Aaron is starting work on a branch off of the 1.4 series to incorporate my visualization patch. Wielding my new Git tool belt, I was off and hacking. Taking Gabriel’s branch allowing replacement of source widgets, I rebased that from master to stable-vis, fixed the merge conflicts, and started hacking away at Banshee.OpenVP. Unfortunately, not all the pieces I needed were there yet. So I added them and pushed them up to Gitorious. Neat.

Now Banshee.OpenVP looks like this:

I’ve abandoned my idea of creating an AJAX interface for the time being. It is a cool idea but I think I can do much better by writing a proper application.

The Linux FICS interface scene is rather weak. eboard is about the best there is in terms of usability, and it has its share of problems. xboard is there for the minimalists who want their interface to provide a chess board only. When compared with the powerful and extensible interfaces available for Windows it’s rather a shame there’s nothing similar for Linux.

So I’ve decided to take the name I was going to use for my web interface and apply it to a new Linux interface. Building on Mono.Addins, I’ve already got an interface that can be extended in several key ways. Addins can, for example, provide new text highlighting classes or classes that can manipulate the console text buffer in interesting ways.

Using a tip from jonp in #mono, I have now come up with an extensible preferences system that does not suck, based on XLinq. Addins can simply subclass PreferenceContainer, slap on a few attributes, and they have an easy-to-use set of strongly typed preferences that get automatically serialized to XML. Right now only the basic primitive types and strings are supported, but this will be extended later to include things like arrays, lists, and XLinq objects. Thanks to Mono.Addins and some more support classes I wrote, addins can also provide Gtk# widgets that get embedded into the application preferences window. By writing very little code, addin authors can persist their settings and provide the user with a convenient way to change them.

I’m still lacking a chess board though. If anyone likes writing custom widgets and feels comfortable working on this project let me know. The sources will be made public in the coming weeks after I’ve had a chance to polish them and set consistent style guidelines.

Hopefully, anyway.

I spent some time this last week (probably over 15 hours total) giving the Banshee visualization pipeline another overhaul. In the process of doing this I finally filed a bug I found in the spectrum GStreamer element that I’ve been trying to work around for a long time.

Even though Sebastian was able to fix this specific bug, another crept up, which he did fix, but it became apparent to me that using the spectrum element was the wrong approach. I won’t go into too many details, but it made disabling and re-enabling the pipeline incredibly tricky and required a buffer of spectrum slices that had to be synchronized with a mutex since it was being accessed by three threads.

Sebastian gave me some pointers on using libgstfft directly, and this has reduced the amount of code required to do spectrum analysis while making it less laggy and less of a hack.

I’m told this patch (and possibly Banshee.OpenVP) will be going into Banshee 1.6. Sweet.

Mandatory screen shot of the new code and of the new Voiceprint visualization in Banshee.OpenVP:

Update 2009-01-22: I had to revise the patch to fix a segfault caused by a race and to eliminate some timing issues with thawing synchronization. The link to the patch has been updated.

Not creative enough to think of a better title right now…

I just spent almost two hours hacking away at the Banshee visualization pipeline again and made one very important optimization: when the callback function is null (which happens when nobody in managed land is listening for visualization data) the visualization pipeline is effectively disabled. The only element that remains active is a queue, which provides a 5-second buffer so that visualization can be quickly resumed.

The patch is now over at GNOME’s Bugzilla, just waiting. Waiting for Aaron to commit it. Waiting for a chance to prove it’s all grown up now. Hoping to bring joy to audio enthusiasts all around the globe.

Yeah, I did watch Tommy Boy recently. … Why?

So apparently the visualization stuff in Banshee has been disabled since it’s a CPU hog. I don’t know why I didn’t notice it during testing (and I did check) but there seems to be a five-fold increase in CPU utilization with the visualization pipeline running. For me, this is an increase from 1-3% to 10-12% on dual-core 2.6GHz amd64.

After ruling out a few things I discovered the root cause. In OpenVP, PCM and spectrum data are represented as 32-bit floats, which means that the pipeline needs to convert whatever the audio format is into this one. Along the way it resamples the stream too, to provide a consistent frame rate of consistent-size slices. 512 samples 60 times per second is 30720 samples/sec. If anyone can show me a song found in the wild encoded at this rate I’ll give you… well, nothing, since I’m a college student and couldn’t afford to give you this pencil I have sitting on my desk.

Anywho, the conversion to float cannot be avoided but the resampling can be. By setting it to the more common 44100Hz sample rate CPU usage dropped to 4-6%. It’s pretty cheap to convert formats and throw duplicate data around, but interpolating data is a lot more expensive.

At some point the vis pipeline will be smart enough to split music into chunks of a size depending on the current sample rate. Until then this patch should be good enough. (I hope.)

After a few hours of hacking using a wigdet that Michael and a few others pointed me at, I now have a working Banshee.OpenVP extension!

Much work still remains to be done, but what I have is a pretty stable foundation. Both the OpenVP 0.0.1 branch and Banshee.OpenVP are available over at the Google Code repo. They should both be stable enough to test, but since the build system isn’t in place yet I’m not asking the general public to try it out yet. But if you know your way around MD, autotools, and Banshee you might be able to get it working. (Please don’t ask for help with that part yet. If you get it working and hit a bug please let me know though.)

Banshee.OpenVP is on hold indefinitely until a stable GTK+ widget providing an OpenGL context is available. All of the wrappers I have tried have serious issues that prevent this project from even making minimal headway.