Sorry for reopening a topic after nearly 6 months. But I cannot stay silent.
I think you got it wrong. Completely.
Although a reference might behave like “some sort” of a pointer, it is *not* a pointer. Your statement: “A reference is effectively a pointer, but this is hidden by the language.” is completely wrong.
To quote the C++ standard: “A reference is an alterantive name for an object.” It is just a new name for something that you’ve defined elsewhere. That’s the very reason why it cannot be null –> You cannot have an alternative name for an object that you do not have yet.

–Willi Burkhardt

Great, in theory. Unfortunately, none of the compilers I have used treat references as anything other than pointers. References are, on some level, supposed to guarantee non-null-ness as well as that they reference a valid object. This is not true in any compiler I have ever used.

Take this example (see it run):

#include <iostream>

static int const a_const = 5;

int const& A() {
	return a_const;
}

static int const* b_ptr = 0;

int const& B() {
	return *b_ptr;
}

int main() {
	int const& a_ref = A();

	std::cout << "Called A()" << std::endl;
	std::cout << "a_ref: " << a_ref << std::endl;

	int const& b_ref = B();

	std::cout << "Called B()" << std::endl;
	std::cout << "b_ref: " << b_ref << std::endl;

	return 0;
}

If we are to believe that references are simply another name for an object, then converting *b_ptr to a reference should have caused a runtime error. After all, we dereferenced a null pointer, right? The compiler should emit code to prevent this, right?

In an ideal world, this would cause an error — but it does not. The segmentation fault does not come until b_ref is used; indeed, we see “Called B()” in the program output, indicating that B() successfully returned a reference, which was stored in b_ref. Obviously, at runtime there was a null pointer dereference. But we didn’t use a pointer, I hear you saying. We used a reference!

Then please explain this behavior to me. On a language level, sure, references are “names for objects.” But this does not change the fact that the implementation is done using memory addresses — which is fundamentally the same thing pointers do. This helps to explain why we see the behavior of this sample. As I mentioned in my last post, when you convert an expression to a reference type, it’s treated exactly as though you had converted it to a pointer type, with an implicit address-of operator (&). So we can rewrite this function:

int const& B() {
	return *b_ptr;
}

Like this:

int const* B() {
	return &*b_ptr;
}

And it becomes immediately clear why the segmentation fault did not occur here — taking the address of a dereference expression is the same thing as taking the original expression. The & and * cancel out during compilation, and we just return the pointer. Take a look at this example, which is identical to the above example, except that A() is gone, and B() now returns a pointer, with dereferences added in the appropriate places (see it run):

#include <iostream>

static int const* b_ptr = 0;

int const* B() {
	return &*b_ptr;
}

int main() {
	int const* b_ptr = B();

	std::cout << "Called B()" << std::endl;
	std::cout << "b_ref: " << *b_ptr << std::endl;

	return 0;
}

Identical behavior.

So you can throw the spec at me all you want, but every implementation I’ve tried uses pointer-with-automatic-dereference semantics — if you convert every reference to a pointer, add an address-of operator to every assignment to a reference, and a dereference operator to every use of a reference, you will see identical behavior.

To preempt the “but the compiler can optimize local references” argument, the compiler can do exactly the same with pointers.

// With a reference
int A() {
    int a = 5;
    int& b = a;
    return b;
}

// With a pointer
int B() {
    int a = 5;
    int* b = &a;
    return *b;
}

I’ve heard the argument that the compiler can eliminate the reference in A(). Well, it can also eliminate the pointer in B(). If a local pointer is set to point at another local and the compiler can prove that it will never change, it can optimize it away just as easily as it can optimize away a local reference to a local.

So, this supports my original argument that references store memory addresses in the same way that pointers do, only with automatic dereferencing. They are effectively nothing more than syntactic sugar, allowing you to forget that you’re operating on an object somewhere else in memory.

I’ve been mulling this issue over for a few months now. There are several development projects I have where database versioning is necessary. Finally, I’ve come up with a process that is not only effective, but actually really simple!

Taking a clue from Git, we can use a digest of the current schema definition file (the SQL script that will create the database structure) as the schema identity. The schema requires a meta-table to describe the identity of the current schema — this will allow for automated schema migration, since the migration script can detect the current schema version. (I usually have a meta-table anyway, for storing site configuration in key/value pairs. The schema version fits in this table fairly well.)

So, let’s say we’re starting a new project. We design a schema, making sure to include this meta-table as well as any seed data required for the application to function. This excludes the “current database schema identity” row, since adding that row in the schema script will cause the identity to change! Then we write a migration script. This script has two functions: load the initial schema, and migrate between schema versions. When performing the initial load, it should follow this by inserting the database schema identity into the meta-table. The identity is of course obtained by digesting the schema file.

Now we are ready to make changes. Let’s say we want to add a column to the table. First, we note what the current schema’s identity is. Let’s call this $SCHEMA_ORIGINAL. We tweak the schema definition file to include this column, and then we obtain the digest of the schema file, calling this $SCHEMA_NEW. Now, we write two migration functions in the migration script: one that will migrate from $SCHEMA_ORIGINAL to $SCHEMA_NEW (an ALTER TABLE ADD COLUMN query) as well as one that will migrate in the opposite direction (ALTER TABLE DROP COLUMN). This will allow us to roll back the database if necessary.

Now, when you ask the migration script to upgrade the database schema, it only has to fetch the database’s current version, digest the schema definition file, and then find a path between those versions using a breadth-first search of the available migrations, starting at the current version.

This technique can even account for merges! When merging branches A and B together, you would resolve any conflicts that arise in the schema definition, and then construct four migration functions: A to merged, merged to A, B to merged, and merged to B. The breadth-first search during migrations means that if you are then switching from branch A prior to the merge to branch B prior to the merge, it may actually be faster to migrate the database through the merge instead of backing up until the point A and B diverged.

It may also be useful to provide a mechanism to tag certain revisions as causing data loss (such as rolling back a column addition). The migration script would then prefer longer migration paths that preserve data over shorter migration paths that destroy it.

There are some downsides to this approach. For one, migration functions will have names that provide little meaning to humans, something like migrate_0beec7b5ea3f0fdbc95d0dd47f3c5bc275da8a33_to_62cdb7020ff920e5aa642c3d4066950dd1f01f4d(). And another is that the migration script will need to construct an in-memory graph of every migration so that it can perform its search. If the only possible migration path contains a migration that will cause data loss, the script will exhaustively search the rest of the graph looking for an alternative. There’s probably some room for optimization there.

I will likely be coding up an MIT-licensed Python script to implement this algorithm in the coming days, so stay tuned.

The building block of the VPS offering is a node: a discrete unit of CPU time, RAM, storage, and bandwidth. These nodes can be deployed as separate servers, or combined together to create a more powerful server. Servers can be upgraded or downgraded by attaching and detaching nodes from them. The CPU/bandwidth changes can be applied without rebooting, but to apply RAM and storage changes, the VPS needs to be rebooted so that the RAM reassignment can happen and so the VPS root volume can be rebuilt. You can bill nodes per month or per day. The daily option is more expensive, but allows you to throw an extra node or two at a server that’s being slashdotted until the traffic subsides — much cheaper in the long haul then monthly nodes. Daily nodes are also a good choice for a throwaway server you’re going to test something on and then scrap. The math suggests that if you’re going to be using the server for longer than two or three weeks, monthly nodes are a better deal.

To further sweeten the deal, nodes are portable across any of their data centers. While you can’t easily migrate deployed VPSes, you can destroy a server in one data center and create a new one in another using the same node. I look forward to the day when we’ll be able to live-migrate running VPSes across the country. Hopefully someone at VPS.NET is working on that…

Initially I had one node and this blog was about the only thing running on the server. Since then, I’ve expanded the services I host to my personal email server (hosting a few domains for friends too), a Git-based personal software forge, some services for Wikimedia, and a few odds and ends. While not a feature unique to VPS.NET, having root on your own server is really nice. Need some software? Install it. Want to tweak the firewall? Go ahead. I’m a hacker, so it’s nice to be able to use my server to just try stuff out and see what works.

Each VPS has CPU and network usage graphs so you can pinpoint busy times and optimize accordingly. There’s also a console for each VPS that connects directly to the offline OS terminal, allowing you to recover from a botched network or ssh config without involving support. While I’d like to claim that I’ve never had to make use of this feature, that would be a lie. I don’t often need it, but it’s there when I do.

Uptime has been pretty good all-around. I’ve had a few outages here and there: a SAN failure that the staff were not properly prepared for, causing a few hours of downtime; a bootloader issue with Debian caused by upgrading to squeeze (arguably not VPS.NET’s fault, as Debian decided to change the boot volume path); and most recently, volume corruption caused by upgrading my node, which the support staff remedied within minutes, but ideally should have never happened. (This makes me a bit reluctant to upgrade again without someone over there babysitting the box to make sure it boots up correctly.) However, these occurrences were not frequent. That’s three distinct issues I can recall since late 2009, two if you attribute the bootloader problem to Debian. This isn’t 100% uptime, but it’s pretty darn close to it, and support is usually quick to respond when problems happen.

I resell a few nodes to some friends and former business partners, and they’ve had very little issues getting started. It takes about 10 minutes to go from purchasing a node to having a running server. When a friend wants another server, I can deliver them their login credentials to a running server within about 15 minutes. That’s pretty cool.

There are a few network-level filtering things one should be aware of. VPS.NET monitors outgoing SMTP (port 25) connections and applies their own spam filtering, rejecting mail if it’s too spammy. Some people might like this since it can protect them from being listed on a spam blacklist if their server gets compromised. Since I run my own mail server and do my own filtering, I wasn’t pleased when this was deployed, especially since it was done so without any communication on their part. Once I figured out what was going on, I was able to opt-out of this filtering by filing a support ticket. I would have preferred more transparency about this change. Finally, as with most hosts, IRC ports are filtered bidirectionally. As I was wanting to use my VPS to run a service that collects data from Wikimedia’s real-time change notification IRC server (essentially just a data stream delivered via IRC) this was a bit of a bummer. They will not budge on this policy, which is disappointing.

Overall, I am a happy customer. Things don’t go wrong often, and when they do support is quick to respond. I’d suggest trying a daily node or two if you want to see if they are a good fit for your hosting needs; they’re only a dollar per day.

There are bound to be things in every programming language that get in the way of productivity. In this post I’ll highlight one of those things in C++: references. References are obfuscating and mostly useless. But first, why would you use a reference?

A reference is effectively a pointer, but this is hidden by the language. You use it like it’s not a pointer, and the compiler turns direct accesses into indirect accesses. For example:

#include <iostream>

int main() {
	int a = 5;
	int &b = a;

	std::cout << "a is " << a << std::endl;

	b = 6;

	std::cout << "a is " << a << std::endl;
}

The output will be:

a is 5
a is 6

Note that I did not say *b = 6;. The reference is treated as though it were not a pointer. Cool… but what’s the benefit?

The solitary benefit I’ve heard from others is that references cannot be null. When you declare a function/method that accepts an argument typed as a reference, it’s not possible to make that reference equivalent to a null pointer.

Ok, so we trade a bit of clarity for a compile-time guarantee that we won’t be dereferencing a null pointer. That’s a good trade, right?

Maybe. Consider this excerpt:

class Foo;

void do_something() {
    Foo *foo = new Foo();
    use_foo(*foo);
    delete foo;
}

This is contrived, yes, and there’s a better way to write this code. But I’m illustrating something here. I’ve told you that Foo is a class, but I haven’t told you the prototype for use_foo(). That’s on purpose. Now, you tell me if the Foo instance is going to be copied. I’ll even tell you that Foo doesn’t overload operator*.

Do you have your answer yet? If you said yes — the logical choice — you’re wrong. If you said no, you’re also wrong. Well, actually, if you said yes or no, you might be wrong. It’s impossible to tell. If use_foo()‘s declaration is use_foo(Foo foo) then a copy will be made using Foo‘s copy constructor (if possible, otherwise it will be a compile-time error). But if the function’s prototype is use_foo(Foo &foo) then we are actually passing in the value stored in the foo variable — a memory address. The object will not be copied. In other words, while it looks like we are dereferencing foo, we are actually doing no such thing.

In C, you can tell pretty much everything you need to know from a call site. In C++, you must know how the function you’re calling is defined too, simply because you have to know when things are references and when they are not. The treatment of what is fundamentally a pointer type as a value type (at the language level) is what causes the uncertainty. You use value-type grammar around references, even though they are not really a value type.

If it weren’t for the existence references, the code above would be perfectly clear. (Well, unless you didn’t have me to tell you that Foo doesn’t implement operator*…)

Don’t get me wrong, C++ still has a lot of good features. It’s just a bit irritating that because of a bad feature (and some other features too, such as some forms of operator overloading), I must know the details of every type and function on a line of code to be absolutely sure what it’s doing.

If you don’t even know what Bitcoin is, it is a distributed and decentralized currency that operates over the Internet. There is no central bank or issuer; there are no regulators. The network as a whole decides what the rules are, and the source code for the official Bitcoin client is open. You can transfer Bitcoins to anyone, anywhere in the world, in about 10 minutes — and usually for free.

The topics I will cover in my series are:

• Basic overview: How to send and receive coins.
• Addresses: What an address actually is, and why you should care.
• The block chain: What the block chain is, and how this keeps the network secure.
• Mining: How Bitcoins are created, and why this is not “generating free money.”

If you can think of any other topics you would like me to cover, let me know and I will extend the series. If you want a bit more information right now, this video is a pretty good summary of the project:

So I was messing around with generic methods and discovered that generic constraints can be bypassed on Mono 2.6.7 and 2.8 using reflection (with the exception of the new() constraint). One of the fun results of this bug is that the String class can be made mutable without using reflection to set private members!

The following code demonstrates this; it is legal and will run on Mono up to and including version 2.8:

using System;
using System.Reflection;

public class FakeString {
    public int length;
    public char start_char;
}

public class TestCase {
    private static FakeString UnsafeConversion<T>(T thing)
        where T : FakeString
    {
        return thing;
    }

    public static void Main() {
        var a = "foo";
        var b = MakeMutable(a);

        Console.WriteLine(a);
        b.start_char = 'b';
        Console.WriteLine(a);
    }

    private static FakeString MakeMutable(string s)
    {
        var m = typeof(TestCase).GetMethod("UnsafeConversion", BindingFlags.NonPublic | BindingFlags.Static);
        var m2 = m.MakeGenericMethod(typeof(string));

        var d = (Func<string, FakeString>)Delegate.CreateDelegate(typeof(Func<string, FakeString>), null, m2);

        return d(s);
    }
}

This code outputs:

foo
boo

Smells like some fun exploits could be written taking advantage of this. Should Moonlight users be afraid? I’m not absolutely certain, but I think there might just be a way to do some damage.

I have decided to migrate away from Gmail for a variety of reasons. I figured I’d use my domain so that I have flexibility in my provider choice. For example, if I decide to change my mail provider I don’t have to get a new address. So this change should be permanent.

Now, being a tinkerer, I figured I’d set up a mail system for myself that retains the features I like from Gmail while doing away with the downsides of using Gmail. My final system is complicated, but effective. And it was a fun four days setting it up! (No, that wasn’t sarcasm. This is the kind of thing I enjoy.)

My MTA is Postfix, running on mail.chrishowie.com. It accepts mail for me and delivers mail from me. The standard security features are in place: SPF/blacklist checking and no unauthenticated relaying. I also established SPF records for my domain. I do not have a spam filter, but I might set up SpamAssassin later if I actually start seeing spam in my inbox; no spam has made it past the sender blacklist check yet.

For downloading mail, I set up Courier as a POP3 server. The observant will note that this would nullify one of the most useful aspects of Gmail: access to your email from anywhere. Once you download from a POP3 server and delete, your mail is gone from the server and lives in your mail client.

That’s where the rest of the rig comes into play. On my home LAN server I have a multi-piece system that provides me with this anywhere-access. I have a getmail4 cron job that fetches mail from my POP3 server, as well as from my Gmail account (so people can still reach me using my Gmail address), and some of my other mail accounts, and delivers the mail to my maildir using Dovecot’s delivery agent. From there, the agent processes my sieve rules, sorting my mail into various folders. (It’s like your favorite mail program’s “filters” only it runs on my server instead of my mail clients, so the configuration is centralized.)

For reading all this mail, I run Dovecot, an IMAP server. All my mail clients fetch mail from this server, and since it is IMAP, changes to messages (like moving a message between folders, or adding tags, or whatever) are actually pushed back to the server. So I can use several mail clients at once and they all have a consistent view of my mailbox.

The only major piece I have yet to set up is an LDAP server for centralization of my address book. The rest has been working quite well. I can use Thunderbird at home, or my mobile phone’s email client when I’m out of the house. No limiting or inconvenient web interfaces required.

Almost every IM client on the planet has the concept of a group. You create a group, give it a name, and populate it with your contacts. At first glance, this is useful.

Having used IM clients since the latter years of AIM’s domination, I have consistently struggled with the best way to group my contacts. To this day I have no solution. Why?

Because the group model is inherently broken in that every major IM client out there allows one group per contact. This assumes that each person I know has exactly one type of relationship to me. A contact cannot be a friend, coworker, family member, and developer. You must pick one.

This problem is exacerbated by multi-protocol clients. Not through any specific fault of their own, but when they support connecting to many different services, you suddenly wind up with an overwhelming number of groups with no coherent purpose. If you didn’t name your groups exactly the same, or you used a different grouping paradigm, you are SOL and must merge the groups somehow. This becomes an impossible mess when the service on the other end (Facebook, for example) allows contacts to be in multiple groups. Which group they wind up in on your IM client may as well be random, and may not even be the same each time you connect. (Edit: Since writing this I have discovered that the XMPP server operated by Facebook does actually indicate to clients when contacts are in more than one group. Pidgin honors this.)

What we need is a cross-service “tagging” mechanism that lets me say “this person is a coworker at X company, and worked with me on Y open-source project.” Then, whether I am looking for coworkers at X or codevelopers on Y, this person shows up. Rules could be established so that “coworker at X company” membership implies “coworker” membership. This may at first glance seem like nested groups, but it is very different. It is completely free-form, and allows me to model my contact structure after how I interact with people in real life. Not some bizarre mutually-exclusive set of too-shallow or too-specific relationships.

Multi-protocol clients should then support tag equivalency, so that service-mandated groups can be rolled into a differently-named tag. Or ignore one account’s groups entirely, with everyone being rolled into one tag, with the possibility of local memorization of extra tags.

But the way things stand now, having groups at all is actually a hindrance to my communication. “Did I put that person in friends or coworkers? Or are they still in the service-specific default friends/buddies/oxygen-converters group?”

In lieu of a real solution to this problem, I want to disable groups entirely. Except… whoops… Pidgin doesn’t even let me do that.

I welcome everyone’s thoughts on this issue.