sunfishcode's blog
A blog by sunfishcode

No Ghosts!

Posted on

This post proposes and explores a design principle for components in complex software systems:

No Ghosts!

The ideas in this post aren't new; they come from papers and blog posts such as Robust Composition, Capabilities: Effects for Free, Parse, don't validate, the nanoprocess model, and the design choices in the Wasm component model, which itself incorporates ideas from Erlang, OCaml, Rust, COM, and many others. This post is an attempt to articulate what I see as one of the themes that runs through all of these. It's not any kind of official position, and it certainly won't be the last word on any of these topics.

I'd also be very interested in feedback on what makes sense here, what doesn't, and what's missing: mastodon, zulip, email.

This is motivated by WASI and Wasm components, however the core ideas generalize to any complex software system, including those using libraries, daemons, containers, VMs, microservices, or a combination.

One of the main ways we can make complex systems manageable is to make them modular. This means being able to add, remove, change, or understand individual components (or whatever a system is composed of) without needing to consider the system around them.

It's common for systems with large numbers of components to have problems with unexpected interactions between components. A common response to these kinds of problems is to impose a level of modularity by introducing heavyweight barriers, such as sandboxes, process boundaries, firewalls, or service meshes. These comprehensively block many of the avenues for components to interact. Then, since components still do need to communicate, it's common to effectively poke small holes in the barriers, such as allowing specific HTTP connections to pass through.

However, while the barriers-and-pinholes approach fixes the immediate problems, barriers are sometimes too comprehensive. They get in the way of connecting things that we do want to connect. This then leads us to do more things through the pinholes, or poke more pinholes, which then increases the risks of unexpected interactions again.

XKCD 2044: Sandboxing Cycle

There are many reasons why we get stuck on this path, but one is that heavyweight barriers tend to focus us on limiting the mechanisms that let components interact, such as which components a component can directly talk to, and what kinds of messages it can send. However, the more fundamental problems are often in the relationships between components. One of the things we can do to avoid these problems, and promote modularity in a sustainable way, is to design component APIs that avoid ghosts.

👻? 😱!

Ghosts?

By “ghost” here, I mean any situation where resources are referenced by plain data.

And by “plain data” here, I mean strings, integers, or any other data where independently produced copies of the data are interchangeable. For example, two completely independent parts of a system may create a string with the value "Purple", and the two strings will be interchangeable.

Plain data can contain filenames, network addresses, usernames, or other forms of data which effectively reference resources.

For example, when we say that a particular string contains a filesystem path, we mean that it refers to an entity in a filesystem namespace. Filesystem namespaces are not explicitly passed as arguments in the APIs of many popular systems, so from the perspective of an API, while paths are explicit string parameters, the additional information referenced by those paths is not. In this post, we'll say this additional information is being carried by a “ghost”:

    // We're explicitly passing a path, but implicitly
    // passing the namespace to resolve it in.
    do_stuff("/tmp/data.txt");

By passing a filename, the caller here is requiring that the callee have a specific filesystem namespace, in order to interpret that filename. This is an example of a relationship between components that's difficult to control with heavyweight barriers focused on mechanisms. The actual message is just a string, which could be communicated through practically any pinhole. And once the caller can send filenames through, it can depend on the callee having a particular namespace and being able to resolve those filenames, and we have the potential to get complex relationships between caller and callee, despite whatever barriers we put between them.

As another example, suppose one part of a system sets an environment variable, and another part of the system reads it.

⬅ In one place:

export TIMEOUT=30

➡ In another:

char *timeout = getenv("TIMEOUT");

Here, two independent parts of the system both use the string "TIMEOUT" as an identifier to send a message between them. As far as these specific parts of the code know, it's as if the content of the message is carried by a ghost, from one part to the other.

IP addresses are another example of plain data that references other resources. If one part of a system listens on a socket and sends the IP address to other parts of the system for them to connect to, the address is a plain-data list of integers, while the interpretation of those integers depends on a particular network view.

Ghosts can occur in filesystems, networks, and more

Ghosts can also occur within key-value stores, registries, brokers, buses, and many other things where the identifiers are plain data. There are situations where plain-data identifiers are the only option, such as when working with external resources. But when designing component APIs, we should seek to avoid ghosts where we can, and seek to identify and encapsulate ghosts where we can't.

Granted, the way all these things work isn't literally supernatural. We can figure out how filesystem namespaces, environment-variable dictionaries, networks, and other things make our resources available if we know some things about the surrounding system. However, that goes against our goal of modularity. We specifically don't want individual components knowing about the system around them.

The trouble with ghosts

Ghosts are often convenient, in the way that duct tape is convenient. They can quickly connect two things, even in a large system, without extensive changes. And on small scales, they sometimes work well. But like duct tape, they aren't a material one wants to build complex structures from.

a boat made from cardboard and duct tape

Ghosts have four distinct problems as systems scale up in complexity:

Systems which use ghost patterns often face several challenges:

Ghosts complicate static analysis

Being unable to know where ghosts are going and where they're coming from makes it difficult and often impossible to answer questions such as:

Ghosts complicate debugging

A common way to debug complex systems is to isolate parts of the system and study how they behave independently. Ghosts create situations where components work differently when run independently than when they're run together, or work differently in different environments, making this kind of debugging more difficult.

Ghosts create hidden cause-and-effect relationships, making it harder to understand the system's behavior.

Ghosts are often a sign of over-sharing

Over-sharing happens when a component is given access to resources that it doesn't need. This often happens in namespace-oriented systems because it's difficult to precisely configure namespaces to be fine-grained and share only what's needed to each component. Even with features such as bind mounts on Linux, it can be tricky to make sure that every part of a complex system has access to all the things it needs, at the paths it expects them to be at, and nothing it doesn't need. As a result, programs are often run with more filesystem access than they strictly need.

This makes it difficult to follow the Principle of Least Authority (PoLA).

Ghosts can contribute to confused deputies

A common pattern in complex systems composed of multiple privilege levels is that some components are considered to run on behalf of specific users, which determine their privilege level. We can call components that work this way deputies of the users that own them.

When components send plain-data requests to components running as different users, senders may be able to reference resources they shouldn't be able to access. In such situations, receivers perform access control, explicitly checking requests to see whether the sender has the appropriate privileges. This is often tricky, especially when an API has a complex surface area. Receivers may get confused into doing things they shouldn't allow senders to ask them to do.

This is a form of the confused deputy problem.

A different kind of relationship

Systems which have a concept of handles—values which can be passed between components, but which are not plain data—can use them to avoid ghosts. Handles provide a way to make specific resources accessible across a component boundary without requiring any other relationship.

GER-BY-RO-Wasserburg am Inn - Brucktor (mechanische Türglocken außen)

Handles make cause-and-effect relationships clear, since they are explicitly passed between components. And, receivers can assume that any handle they are passed represents a resource that the sender is allowed to ask them to operate on. That way, receivers need less authority of their own, which reduces the risk of them accidentally misusing their authority.

Ghosts can hide inside explicit sharing

One of the tricky things about ghosts is that they're about relationships rather than specific mechanisms. Mechanisms tend to be easy to understand, and to sandbox. But, relationships that permit ghosts can pass through even the most restrictive sandboxes.

This blog post talks a lot about implicitly shared resources, however that's not the only place ghosts can hide. For example, consider our example above of caller and callee implicitly sharing a filesystem namespace, and passing strings representing paths:

    do_stuff("/tmp/data.txt");

This is a ghost pattern, with a string carrying a reference to an implicitly shared namespace. A simple way we might try to eliminate such a ghost is to replace the use of an implicit namespace with an explicit filesystem root parameter:

    do_stuff_in_root(root_handle, "tmp/file.txt");

This might be tempting, as it means that most of our code doesn't need to fundamentally change. It's a mostly mechanical change to just add root parameters in places where they're needed, and everything else about our code can stay the same.

We might then be tempted to claim that we've eliminated our ghosts here, because we now do the sharing via explicit communication rather than an implicitly shared resource. And we might indeed find that this code does afford us some added flexibility.

The problem is that this doesn't change the relationship. We're still using strings to identify specific resources within the filesystem root we're passing around. And that means we still have resources being referenced by plain data.

There's effectively a ghost, hiding inside the resource.

onigiri bento with a ghost inside (4039012309)

One way to think about it is in terms of granularity. While passing around handles to “root”, “world”, “namespace” or “registry” resources is better than implicit sharing, those kinds of resources tend to be coarse-grained. They can end up having ghosts hiding inside them. Plain-data references to specific items within coarse-grained resources can still have dynamic cause-and-effect relationships, and can still dangle, collide, or be influenced by attackers.

To avoid ghosts, it's not enough to change the mechanisms. To change the relationships, we need to switch from coarse-grained sharing to fine-grained sharing with handles. Instead of whole filesystems, we should ideally reference specific directories or even individual files, such as like this:

    // Open the file using our own privileges.
    file_handle = open(root_handle, "tmp/file.txt");

    // Instead of passing root_handle and a path, pass
    // *just* the one file handle to the other component.
    process_open_file(file_handle);

How to smell a ghost

There are some common signs that a ghost may be present.

Wrapping it up

An important property for complex software systems is that they be modular, where parts can be isolated from the whole. Ghosts, or resources referenced by plain data, create implicit relationships which must be considered when we add, remove, change, or understand individual components. They impede modularity, making complex systems less manageable.

This leads to a design principle for components in complex software systems:

No Ghosts!

👻? 🚫!