Language</th>	Has the bug</th>	Versions tested</th></tr></thead>
C</td>	Yes</td>	(all)</td></tr>
C++</td>	Yes</td>	(all)</td></tr>
Python 2</td>	Yes</td>	`Python 2.7.18</code></td></tr>`
Ruby</td>	Yes</td>	`ruby 2.7.2p137 (2020-10-01 revision 5445e04352) [x86_64-linux-gnu]</code></td></tr>`
Java</td>	Yes</td>	`openjdk 11.0.11 2021-04-20</code></td></tr>`
Node.js</td>	Yes</td>	`v12.21.0</code></td></tr>`
Haskell</td>	Yes</td>	`The Glorious Glasgow Haskell Compilation System, version 8.8.4</code></td></tr>`
</td>	</td>	</td></tr>
Rust</td>	No</td>	`rustc 1.59.0 (9d1b2106e 2022-02-23)</code></td></tr>`
Python 3</td>	No</td>	`Python 3.9.5</code></td></tr>`
Perl</td>	No</td>	`perl 5, version 32, subversion 1 (v5.32.1) built for x86_64-linux-gnu-thread-multi (with 46 registered patches...)</code></td></tr>`
Perl 6</td>	No</td>	`v2020.12</code></td></tr>`
Bash</td>	No</td>	`GNU bash, version 5.1.4(1)-release (x86_64-pc-linux-gnu)</code></td></tr>`
Awk</td>	No</td>	`GNU Awk 5.1.0, API: 3.0 (GNU MPFR 4.1.0, GNU MP 6.2.1)</code></td></tr>`
OCaml</td>	No</td>	`4.08.1</code></td></tr>`
Tcl</td>	No</td>	`8.6.11</code></td></tr>`
C#</td>	No</td>	Mono JIT compiler version 6.8.0.105</code></td></tr> </tbody></table> A more complete and current list is maintained here</a>.</p> Porting Rust's std to rustix 2022-01-04T00:00:00+00:00 Rustix</a> is a system-call wrapper library with multiple backends. It has a raw Linux syscalls backend, as well as a libc backend, and other backends</a> are in development. Rustix is designed for memory safety, I/O safety</a>, and performance.</p> And this</em></a> is a branch of Rust's std partially ported to use rustix in place of direct libc calls. Read on for why this is cool, and stay for the benchmarks!</p> Factoring out unsafe</code>, error handling, and raw pointers</h2> The first reason that porting std to rustix is cool is that rustix factors out a lot of unsafe</code> blocks from std. Talking to the OS still requires unsafe</code>, but with rustix the unsafe</code> blocks are focused on individual syscalls. That way, the unsafe</code> blocks that remain in std are the interesting ones, where std itself is doing something interesting that needs unsafe</code>.</p> Rustix also provides idiomatic Result</code> error handling for system calls. And it uses Rust references and slices instead of raw pointers. These make it easier to read std's code and focus on the important semantics of the system calls, without the distractions of libc API mechanics.</p> And, rustix simplifies some minor infelicities in syscall APIs related to C integer type sizes.</p> Putting these all together in an example, this code</a>:</p> </span>let</span> len = cmp::min(buf.</span>len</span>(), <wrlen_t>::</span>MAX </span>as </span>usize</span>) as </span>wrlen_t</span>; </span> </span>let</span> ret = </span>cvt</span>(</span>unsafe </span>{ </span> c::send(</span>self</span>.inner.</span>as_raw</span>(), buf.</span>as_ptr</span>() as </span>const c_void</span>, len, </span>MSG_NOSIGNAL</span>) </span> })?; </span></code></pre> becomes this</a>:</p> </span>let</span> ret = rustix::net::send(&</span>self</span>.inner, buf, SendFlags::</span>NOSIGNAL</span>)?; </span></code></pre> This puts the focus on the send</code> operation, without the distractions of unsafe</code>, raw pointers, wrlen_t</code> types, and cvt</code> error handling.</p> A path to a Rust on Linux without libc</h2> A second reason this is cool is that it's a step towards a Rust toolchain on Linux that doesn't depend on libc.</p> Rustix is able to make direct Linux syscalls from Rust code. And origin</a> is a Rust library which is able to startup and shutdown processes and threads (comparable to crt1.o and libpthread).</p> With these, we have all the things needed to run Rust programs on Linux. And it turns out there are two different ways to do this. The first way is mustang</a>.</p> Mustang is uses a library called c-scape</a>, which wraps rustix with libc-compatible APIs, allowing std to use rustix without modifications, including threading support</a>. This has gotten a lot of functionality up and running; mustang can run a lot of real-world code now. And mustang helps test rustix and origin. And beyond that, the c-scape libc compatibility layer has several additional uses. But, mustang in its currrent form looks like it would be awkward to upstream into Rust.</p> Fortunately, mustang's architecture of keeping c-scape as a separate layer on top of rustix and origin, with rustix and origin providing idiomatic Rust APIs, means that another way is possible as well. This blog post is about starting to port std to rustix directly. In addition to not using libc code, this path doesn't use libc APIs either.</p> Performance</h2> And another reason this is cool is that rustix-enabled std also brings several modest speedups, compared to std in upstream Rust. On machines I've tested it on:</p> std::fs::metadata</code> is about 3% faster</li> std::fs::File::open</code> is about 5% faster</li> std::fs::read_to_string</code> is about 3% faster </li> std::time::Instant::now</code> ranges from 1% to 10% faster on Linux</li> </ul> See the benchmark source code</a> to see what's being measured.</p> Rustix makes metadata</code>, open</code>, and other filesystem path operations faster by avoiding a dynamic allocation when converting from Rust strings into NUL-terminated C strings. Rustix uses stack-allocated memory for strings up to a reasonable size, which is much faster.</p> Rustix makes Instant::new</code> faster on Linux by simplifying error handling, since we know that reading the system monotonic clock never fails (and as in current std, it still does panic if a failure ever does occur).</p> And rustix contains a number of other optimizations, such as using inline system calls and avoiding the TLS errno</code> variable on Linux, but so far benchmarks confirm the common wisdom that these aren't usually very significant compared to the cost of the actual syscalls.</p> Looking forward</h2> A goal for this port of std to rustix is to eventually propose it to be merged into upstream Rust. I'm hoping this blog post will start some conversations about what this should eventually look like.</p> With rustix's libc backend, std can continue to support all the libc-using platforms that Rust currently supports. And with either backend, rustix brings the advantages of factoring out unsafe</code>, error handling, and raw pointers, and its optimizations for converting to C-style strings.</p> This project promotes several other goals as well, such as promoting I/O safety</a> concepts and APIs, helping test some of the infrastructure used by cap-std</a>, and helping set the stage for future projects related to sandboxing, WASI, nameless</a>, and other areas.</p> Thanks!</h2> Thanks to @nivkner for implementing support for child processes, to @Urgau for adding arm support and implementing several features in rustix, @cole-miller for implementing getcwd</code> and chdir</code> in mustang, and @jplatte and @ratmice and others for contributing useful patches, and @tshepang in particular for bringing up ripgrep</code> as a testcase (and it almost works</a> now!).</p> Context Brainstorming 2021-12-23T00:00:00+00:00 This is a blog post brainstorming about contexts</a>.</p> I'll us the term contexts</em> here, as tmandry is leaning to</a>, since it seems to make sense to keep capabilities distinct concepts. Idiomatic capability-based code and the Principle of Least Authority prefer fine-grained access to resources</a>, which contexts don't seem like a good fit for. So let's keep these concepts distinct for now.</p> yoshuawuyts showed me there is way we might use something like contexts to retrofit an awareness of ambient authority into Rust. Here's an attempt to sketch up more of what that might look like.</p> Automatic contexts</h2> Let's extend the contexts proposal with a concept of automatic</em> contexts, that functions would implement by default. Just like how Rust has automatic trait impls. Like automatic trait impls, you can opt out, with negative with-declarations, using !</code> syntax.</p> And let's introduce the concept of supercontexts, which are contexts that imply other contexts. Much like supertraits in Rust. This isn't strictly necessary, but it helps with granularity.</p> With those, and the observation that contexts are a way of coloring functions</a>, let's introduce some hypothetical automatic contexts:</p> global_allocator</code>, the ability to use the Rust global allocator</a>.</p> </li> ambient_authority</code>. Similar to this AmbientAuthority</code></a>, but as a context, so it can be more. This would be a supercontext which includes:</p> fs</code> - the current process' filesystem namespace</p> </li> net</code> - the current process' network namespace</p> </li> time</code> - the current process' time namespace. Preventing code from observing time entirely is hard, especially if there can be multiple threads, so maybe this time</code> would just be about the explicit time APIs rather than blocking all potential time sources.</p> </li> stdio</code> - access to the ambient stdin, stdout, and stderr</p> </li> process</code> - the ability to spawn arbitrary child processes</p> </li> mutable_static</code> - write to or</em> read from statically-allocated mutable and</em> interior-mutable state in the process. There are use cases where statically-allocated state is useful, but since we have contexts here, for maximal modularity, these cases should ideally use contexts instead of implicitly associating state with the whole process.</p> </li> and others. OS's attach a lot of miscellaneous authorities to processes. Ideally we'd make sure we have everything covered.</p> </li> </ul> </li> </ul> These being automatic is kind of a way to retroactively reinterpret existing Rust code. All code now defaults to having these contexts, and we can then opt out of them, like this:</p> fn </span>useless</span>() </span>with </span> !ambient_authority </span>{ </span>} </span></code></pre> Here, we can know immediately that this useless</code> is a useless function just by looking at its signature. It has no return value, no arguments, no ambient authority. All it could do is return, panic, or infloop.</p> Panic could unwind, and it'd be nice to add a context for that too:</p> unwind</code> - the ability to unwind the stack</li> </ul> then !unwind</code> could be used for functions that can't unwind. Maybe this could even be connected to LLVM's nounwind</code></a>. Anyway, with !unwind</code>, we could write code like this:</p> fn </span>totally_pure</span>(</span>a</span>: &A) -> B </span>with </span> !ambient_authority + </span> !unwind </span>{ </span> </span>// lots of interesting stuff </span>} </span></code></pre> I think someone told me once that the Rust compiler can know whether types have interior mutability. Let's assume it can, and that this includes types that hold I/O handles</a>. In theory, if A</code> here has no interior mutability, this should allow Rust to annotate functions like this with optimizer attributes like LLVM's readonly</code></a>, meaning calls to it could be redundant-code-eliminated.</p> Beyond just LLVM though, this could enable MIR-level redundant-code elimination of calls, even pre-monomorphization. No need to do complex alias analysis or escape analysis, because the type system just tells you what you need to know up front!</p> But it wouldn't get dead-code elimination, because of the possibility of inflooping. More on that later.</p> Pure, except where indicated otherwise</h3> By the way, if one of the arguments has a type that does have an I/O handle, including a filesystem handle, then the function can always do I/O. The fs</code> context is about the process' filesystem namespace. So with !fs</code>, you can't do File::open</code>, but you can use a Dir</code></a> you've been given as an argument to do Dir::open</code></a>, because it's resolved relative to a directory you have an explicit handle to, rather than the process' filesystem namespace.</p> Similarly, passing a &mut</code> reference into a function marked this way requires no special ceremony. Unlike "pure" keywords in languages where purity is all or nothing, the rule here is, if the signature has a &mut</code>, the callee can access it as a &mut</code>, including mutating it:</p> fn </span>pure_except_as_obvious</span>(</span>a</span>: &A, </span>m</span>: &</span>mut</span> M, </span>f</span>: &File) -> B </span>with </span> !ambient_authority + </span> !unwind </span>{ </span> </span>// lots of interesting stuff, including mutating `m` and writing to `f`. </span>} </span></code></pre> See First-class I/O</a> for more discussion of this.</p> Security</h2> It may be surprising that that I haven't talked about security in this post yet. It turns out that capability-based security really is just a special case of a deeper capability-based design philosophy. It's similar to how Rust's borrow checker is, on its face, a memory-management strategy, but also much deeper, with things to say about such seemingly unrelated areas as thread safety, pointer aliasing, iterator invalidation, and refactoring. There's a lot going on here.</p> It also turns out that security for untrusted or compromised-supply-chain code is complex. For example, if we want to completely sandbox a piece of Rust code with language mechanisms, we need to make sure it can't use unsafe</code> blocks, since unsafe Rust could trivially escape any sandbox. Security exploits are ok relying on UB if it works with enough probability in practice.</p> Getting closer: unsafe</h3> This post is all about contexts though, so let's see if we can use them to fix that problem too:</p> new_unsafe</code> - an automatic context representing the ability to introduce new unsafe contexts. This corresponds to the ability to write unsafe { ... }</code>.</li> unsafe</code> - a retroactive reinterpretation of what unsafe fn</code> desugars to. Includes new_unsafe</code> as a subcontext, or doesn't, depending on how unsafe blocks in unsafe functions</a> goes.</li> </ul> As an aside, contexts would also be a path for libraries to define unsafe-like concepts for their own invariants, which is something I occasionally see people asking for in Rust.</p> With new_unsafe</code>, we could write:</p> fn </span>untrusted_code</span>(</span>x</span>: &X, </span>y</span>: &</span>mut</span> Y) -> Z </span>with </span> !ambient_authority + </span> !new_unsafe </span>{ </span> </span>// untrusted code here? </span>} </span></code></pre> Would this be a secure sandbox? Not yet; one problem is that even if we know X</code> has no interior mutability or I/O handles, this code still exposes the address</em> of x</code> or y</code> to untrusted code, because converting a reference to a raw pointer doesn't require unsafe</code> in Rust. The address might tell an attacker something about the ASLR</a> in the process, which might make other attacks more powerful.</p> Still getting closer: raw pointers</h3> When all you're doing is writing a blog post about contexts, everything looks like a problem to be solved by adding a new automatic context.</p> raw_pointers</code> - the ability to convert references into raw pointers.</li> </ul> In addition to solving this ALSR problem, this attribute has some interesting possibilities. It's awkward how Rust allows APIs with reference arguments to observe whether two references have the same address, when this usually isn't part of the conceptual API. !raw_pointers</code> would be a way to declare that a function doesn't do that.</p> Further, with !raw_pointers</code>, it'd be possible to have Rust code that doesn't depend on a byte-addressed address space. There'd be no alignment or endianness visible. Objects could be moved at any time, just like in a moving GC. Threads could be migrated to different stacks. This might even open up a path to Rust being able to use Wasm reference types, which Rust can't otherwise hold directly since they're opaque and can't have their representation exposed.</p> Are we secure yet?</h3> No. But, to keep this blog post scoped, let's ignore side-channel attacks like Spectre, hardware attacks like Rowhammer, crypto miners, and denial-of-service attacks. And let's ignore attacks which change the behavior of the code without breaking the sandbox, such as changing an encryption implementation to emit syntactically valid but insecure data. That's a lot to ignore in reality, but the solutions to those would require radically different mechanisms, so let's put those aside for now.</p> Ok, now</em> are we done yet?</h3> What about global variables? We included mutate_static</code> in ambient_authority</code> above, so they won't be mutated, but is it a problem if the untrusted code reads any of the program's global immutable state? Could it find authentication secrets? To answer this, we'd need to start getting more specific about the threat model. But to keep things simple, let's say the program doesn't have anything sensitive in immutable global state. It's best to keep sensitive things like authorization credentials as scoped as possible in general anyway.</p> Along those lines, what about std::env::vars</code>, std::env::args</code>, std::env::home_dir</code> and others? They might contain sensitive information, or even just your username. Let's say these are disallowed by !mutable_static</code> by virtue of being mutable through libc APIs. Or, if needed, we could also add a new context to cover these.</p> Will it ever stop</h3> This is just a brainstorming post, and it's possible things are missing, but it's likely any such things can be covered by adding more contexts. For the sake of making a finite blog post, let's assume we can cover everything.</p> So can we say then, that we now, assuming all of our assumptions, finally</em> have a secure hypothetical sandbox here?</p> fn </span>untrusted_code</span>(</span>x</span>: &X) -> Y </span>with </span> !ambient_authority + </span> !new_unsafe + </span> !raw_pointers </span>{ </span> </span>// untrusted code here! </span>} </span></code></pre> Yes.</p> beat</em></p> In theory.</p> In practice, the Rust compiler isn't currently designed or intended to be used as a security surface in this way. And it's not necessarily worth it for it to try to be one. There'd be work involved, and for this to actually make sense, we'd need to look at real-world use cases and attack vectors, and we wouldn't be able to ignore any of the things we ignored above.</p> Capability-based programming</h2> However, even if we don't look to !ambient_authority</code> to be the basis of an actual sandbox, and even if the performance impacts of the aliasing, escaping, and side effect knowledge isn't compelling, this overall technique might still be useful.</p> For people reviewing code, !ambient_authority</code> could reduce the reasoning footprint</a>, because they'd be able to make more local assumptions about the side effects of calling functions.</p> And for people building large complex applications, it could give them more tools to help ensure that two parts of the application don't have unintended interactions, as explored here</a>.</p> And for people building wasm components</a>, it could give them more tools to ensure that they're only using APIs which compose cleanly with other components.</p> Potential downsides</h2> With all these colors, and with users having the ability to define their own colors, we could end up with a lot of colors.</p> Will having an ecosystem where everyone can use all these colors to enforce their requirements with extraordinary precision increase or decrease overall usability of Rust? Will it lead programmers to waste time pursuing every possible dimension of theoretical purity, regardless of what really matters in practice?</p> Will these new colors and automatic contexts prompt new rounds of users going through all their dependencies and insisting that they support new colors? If so, will it cause ecosystem churn and/or awkward workarounds, or even ecosystem fragmentation, like #![no_std]</code> sometimes does, and is that worth it?</p> I don't know.</p> What I do know is, in a vacuum, it sure is fun to think up new colors.</p> Tangent: Pretty colors</h2> Let's think about one more possible context, for fun:</p> turing_complete</code> - the ability to have loops (or tail recursion, if Rust adds that), that can't be proved to terminate.</li> </ul> The halting problem gets talked about a lot. However, how often does one actually write loop</code>, as opposed to just using for</code>? If we also had a way to assert that iterator implementations don't repeat themselves, a lot of real-world code might be able to be compatible with !turing_complete</code>.</p> One of the tricky issues for iterators would be linked lists, which would need to be guaranteed to be acyclic. But it's interesting to note that in Rust, creating a circularly linked list actually requires unsafe</code> anyway. So maybe there's something we could do here.</p> This would also address the "or infloop" case mentioned above, so we could also get dead-code-elimination of calls.</p> The Spectrum from Namespaces to Values 2021-12-22T00:00:00+00:00 In order to make large applications modular, we need to think about resources and sharing. One axis for thinking about this is the spectrum of granularity. It's a spectrum, but we can identify several notable levels:</p> Level 0: File::open</code></li> Level 1: with root: &Dir</code></li> Level 2: dir.open</code></li> Level 3: thing: &File</code></li> Level 4: thing: &StreamReader</code></li> </ul> This post describes each of these levels and considers the impact on application modularity of moving up through each of the levels.</p> As a caveat, in this post, I'll be discussing trusted</em> code. It's about modularity and flexibility. Specifically, I'm ignoring the concerns of untrusted</em> code here. If there's code in the system which is actively attempting to break out, it requires additional mechanisms to securely contain it. The concepts discussed here can be used as part of a sandbox for untrusted code, but a proper sandbox involves many additional concerns that I'm not covering here.</p> Usually when someone writes a post about capabilities and the Principle of Least Authority, they're talking about security. However, this post is about how these concepts are relevant to programming in the large in general.</p> With that in mind, let's start at the bottom and work our way up :-).</p> Level 0: File::open</code></h2> Level 0 is how most application code today uses filesystems:</p> fn </span>foo</span>() { </span> </span>let</span> thing = File::open("</span>/path/to/some/dir/thing</span>"); </span>} </span></code></pre> There's a filesystem namespace implicitly associated with the program, and filesystem paths are resolved in it. Lots of code works this way to day, and it works. But, traditional OS isolation mechanisms are not sufficient for many popular use cases today, so many use cases end up using containers</em>. A container provides a little world for an application to run in, where it can pretend it's running on a "normal computer" built specifically for it.</p> However, containers are coarse-grained. One typically needs to run an entire application within the same container. Or if applications are split into parts that run in different containers, the parts usually need to communicate over invasive RPC mechanisms. By climbing up to the next level of granularity, we can gain more options.</p> Level 1: with root: &Dir</code></h2> Using a mechanism such as cap-std's Dir</code></a> type, we could arrange to pass in a "root filesystem" as a parameter rather than using a namespace implicitly attached to the program:</p> fn </span>foo</span>(</span>root</span>: &Dir) { </span> </span>let</span> thing = dir.</span>open</span>("</span>/path/to/some/dir/thing</span>"); </span>} </span></code></pre> And if we write a lot of code like that, we might find a recent blog post about context capabilities</a> useful. That would allow us to write the above code like this:</p> fn </span>foo</span>() </span>with </span> root: &Dir </span>{ </span> </span>let</span> thing = File::open("</span>/path/to/some/dir/thing</span>"); </span>} </span></code></pre> Unlike the standard-library File::open</code>, this would use a File::open</code> that also uses context capabilities, so it would also have a with root: &Dir</code>, and it'd automatically get passed the root in which to resolve the path.</p> This approach has the appealing property that the body of the code is the exact same as it is in level 0. It would be easy to convert existing code to use this style, because it just requires changes at the high-level scopes.</p> Going from 0 to 1 means that an application could have the ability to create little dedicated filesystem worlds, and run parts of itself in these little worlds, with each part having its own root filesystem. It wouldn't have to completely split into multiple containers.</p> That's a step up. However, creating dedicated filesystems is still complex and potentially inefficient. How do we know what those little worlds need to contain? It requires us to know the set of strings that the code might dynamically pass to an open</code> function. Sometimes we can get a pretty good idea of code code needs by studying it. And sometimes we resort to running code and just dynamically recording what things it needs.</p> And, if two parts of an application want to communicate, we still have to figure out if they need to run in the same little world, or if they can run in different worlds. And if they are in the same world, to share data through it, we then need to somehow make sure they don't collide in unintended ways in other parts of their shared filesystem, such as needing different versions of a dependency installed at the same path.</p> Level 2: dir.open</code></h2> The next step up is to break up the monolithic "world" into more fine-grained handles. cap-std's Dir</code> type makes it easy to have first-class directory handles, so we can just pass those around.</p> fn </span>foo</span>(</span>dir</span>: &Dir) { </span> </span>let</span> thing = dir.</span>open</span>("</span>thing</span>"); </span>} </span></code></pre> We could use the above-mentioned context capabilities for these. However that syntax is less valuable once we start getting more fine-grained, as we do here at level 2. When we're talking about a whole filesystem, it's common for code to want to treat it like an ambiently present resource, and context capabilities approximate that. But once we can talk about specific directories, it's more natural for code to be aware of the specific directories it needs, and take those directories as explicit parameters. It's no longer about "here's a world to operate in", it's about "here's the directory you asked for".</p> Passing individual directories is a step up from level 1. With this, when an application is split into parts, we don't need to create whole filesystem views for each part. We just need to give each part its own dir</code> directory. And if they do need to share, they just share those directories, and we don't have to worry about them colliding in other areas of their filesystems.</p> But if the only thing foo</code> needs is one file in that directory, passing it a whole directory is still more complex than we need. How can we know when two parts of an application need to share a directory? As above, the only way to know how they interact is to look at the set of strings they dynamically pass into open</code> functions.</p> Level 3: thing: &File</code></h2> If foo</code> just needs a single file, it can be passed just that file:</p> fn </span>foo</span>(</span>thing</span>: &File) { </span> </span>// ... </span>} </span></code></pre> This way, an application can be split into parts, and the parts can freely operate on different files. The application doesn't need to worry about whether it needs to separate those files into separate directories. Even if the files all live in the same directory, the parts of an application accessing individual files won't collide, because they only have their &File</code>, not the whole directory.</p> Level 4: thing: &StreamReader</code></h2> The next level up is to observe that most applications don't need an actual File</code> as such. Even though Unix told us that "everything is a file", pipes and sockets aren't really files, in that they don't always have filenames, and don't support all of the File</code> operations. However, in a lot of use cases, applications would ultimately be ok reading from a pipe or a socket just as well as an actual file. In some cases, we may want them to simply read data from a buffer in memory. The thing that most application code needs is just a source of data to read from.</p> This could be impl Read</code>, Box<Read></code>, or the io-stream crate's StreamReader</code></a>. Or the async versions of any of those. But that's getting into the mechanics of how the I/O is done. It's an interesting topic, but not the focus of this post.</p> The important part for this blog post is that level 4 is where we reduce a piece of code's requirements down to just an input stream, so it can be connected to other pieces of code without having to think about what other resources that might cause them to implicitly share. Input streams can be streamed across networks. They can be buffered, or stored and replayed. If we can write code that works in terms of streams instead of files, we have a lot of flexibility, because we don't need to worry about the code implicitly depending on file-specific semantics.</p> And beyond</h2> The next step up from byte streams would be typed streams, iterators, or generators. A big sequence of u8</code>s requires the producers and consumers of the data to have some implicit agreement about how the bytes are interpreted. An API providing a sequence of values of some type T</code> would allow producers and consumers to be self-describing. The mechanics of these APIs are another interesting topic, and also not the focus of this post :-).</p> The focus is that once we have an abstraction level where bodies of code can communicate typed values, these bodies of code can communicate precisely what they need, without incompatibilities among things they implicitly share.</p> At each step up, we got closer to "what does the code actually need?", and found ways to give it what it needed, while reducing the amount it implicitly shares with other parts. This reduced the potential for conflicts with other code, and simplified the task of breaking up applications into parts.</p> An analogous concept in computing is false sharing</a>, except here, instead of performance and sharing parts of cache lines that we don't need, it's about modularity and sharing parts of monolithic resources that we don't need.</p> By applying the Principle of Least Authority, and passing around fine-grained resources to code that advertises specific needs, we improve modularity.</p> Thread support in Mustang 2021-10-14T00:00:00+00:00 Mustang</a>, a system for running Rust programs entirely written in Rust, has make a lot of progress since the last blog post:</p> New targets: riscv64 and arm, joining x86_64, aarch64, and x86; thanks to @Urgau for arm support in rustix!</li> Threading support, including TLS, TLS destructors, and detaching</li> Panic and unwind support, thanks to the unwinding</a> crate!</li> Math library support, thanks to the libm</a> crate!</li> A proper allocator, thanks to the dlmalloc</a> crate!</li> No more debugging messages on stderr by default</li> DNS (ToSocketAddrs</code>) support</li> Much smaller code size</li> Lots more test coverage</li> </ul> Mustang's thread library is a chance to explore the role of a thread library, the special syscalls and registers that only a thread library uses, and the interaction between a thread library and other system calls. The rest of this blog post takes a closer look.</p> I'd like to thank Steed's pthread implementation</a> for the initial inspiration here, and demonstrating how to use clone</code>, futex</code>, and the platform thread register.</p> So, what does a thread library do?</p> Create threads</h2> First, what is a thread?</p> A thread consists of an OS thread, which on Linux is just a special kind of process, and some data: metadata, user TLS data, stack memory, and a stack guard. Creating a thread involves allocating memory for the data, and then creating an OS thread configured to use them.</p> Mustang's origin</a> crate allocates all the memory for a thread in a single contiguous anonymous mmap</code> allocation. Most platforms have a special "Thread Pointer" register which is used to point to thread-specific data, %fs</code> on x86_64, %gs</code> on x86, tpidr_el0</code> on aarch64, tp</code> on RISC-V, and so on, and we use this to point to this allocation. Compiled code uses this register to locate TLS data, and origin uses this register to locate its metadata.</p> </p> The specific layout differs between architectures</a>, for example with the metadata being located before or after the user TLS data. And things would be much more involved if we were talking about dynamic linking. But this covers the basics.</p> Once the data is allocated and initialized, it's time to create the OS thread.</p> The clone</code> system call</h3> Origin creates an OS thread using the clone</code></a> system call. Linux considers this to be similar to fork</code></a>, which creates new processes, but with more options. A new thread is a new execution context, with its own CPU register state, similar to a new process. The main thing that makes it a thread instead of a process is that it happens to use the same virtual address space and other parts as its parent.</p> </p> Like fork</code>, clone</code> is invoked by one execution context, and when it returns there are two execution contexts, one for the parent and one for the child.</p> Unlike fork</code> though, when the child shares virtual memory with the parent, it can't continue on the same stack, so we pass clone</code> the pointer to the stack we allocated above for the child to run on.</p> The child starts execution by returning from the clone</code> call, and the return value of the clone</code> call is 0 in the child, and the child's thread id in the parent, so the code after the clone</code> call can test whether it's running in the child or the parent.</p> The clone</code> system call doesn't have a built-in way of passing any arguments to the child to tell is what to do. To do that, we place additional arguments in registers that we know the system call doesn't clobber, effectively "passing" them to the child.</p> To illustrate, here's an illustrated version of the RISC-V code for this. Other architectures do equivalent things, though eg. x86 has more ABI details in play.</p> // Place syscall arguments in a0, a1, ..., </span>// and `__NR_clone` in a7. </span>// </span>// And place the function pointer for the child to call, and </span>// the argument to pass to it, in a8 and a9, which are not </span>// used by the syscall, but not clobbered either. </span>... </span> </span>ecall </span>// Do the `clone` system call. </span>bnez a0, </span>0</span>f </span>// If we're in the parent, branch to 0: </span> </span>// We're in the child! Move the argument and function </span>// pointer into calling-convention registers for the call to </span>// `entry`. </span>mv a0, a8 </span>// `arg` </span>mv a1, a9 </span>// `fn_` </span> </span>// Zero out the frame address and return address. The call </span>// below will never return. </span>mv fp, zero </span>mv ra, zero </span> </span>// Call into our Rust code which will call the callee, </span>// passing it the argument. The Rust code will terminate </span>// the thread and not return here. </span>tail </span>entry </span> </span>// We're in the parent! Return from the system call in the </span>// normal way. </span>0</span>: </span>... </span></code></pre> origin's philosophy is to do as little as possible in assembly, so this code just sets the frame pointer and return address to null, to ensure that we never try to return back to our little assembly fragment, moves the arguments into place for the platform calling-convention argument registers, and calls into Rust code, specifically the threads::entry</code> function</a>, to do the rest.</p> As an aside, this is very similar to the code origin uses for the start of the process, _start</code></a>, which calls program::entry</code> function</a>. The OS doesn't start a process in a manner compatible with the platform calling-convention, so we use a minimal amount of assembly to set up a call into Rust code, also including setting the return address and frame pointer to null.</p> In both threads and processes, we always exit by calling the exit</code> (for threads) or exit_group</code> (for the process) system calls, and never by returning to assembly code.</p> Join threads</h2> When one thread wants to wait for another to exit, it performs a "join".</p> Once the other thread exits, the joining thread is woken up, and it frees any resources associated with the other thread, and collects the return values (though Rust's std::thread</code> implementation doesn't use thread return values as such, so the return value part isn't implemented in Mustang yet).</p> In the clone</code></a> system call that we called in create_thread</code> above, one of the flags we pass is CLONE_CHILD_CLEARTID</code>. This tells Linux to clear the child thread id and wake up any futexes that are waiting on that memory location when the child exits. It's important that it does both, so that the parent's futex call can avoid waiting if the child has exited first.</p> If the child hasn't exited yet, join_thread</code>'s futex</code> call waits until the child exits and Linux wakes it up. If the thread wasn't detached (more on that below), joining then frees the thread's memory—its stack, TLS data, and metadata.</p> In the code, this is join_thread</code></a>, which calls wait_for_thread_exit</code>, and then free_thread_memory</code>.</p> Run thread destructors (registered with __cxa_thread_atexit_impl</code>)</h2> Variables declared with the thread_local</code></a> macro can have Drop</code> implementations, and those drop</code> functions are called on a thread's copy of the data when the thread exits.</p> __cxa_thread_atexit_impl</code> is the C ABI function to register a cleanup function to call when a thread exits. It simply pushes the function onto a Vec</code>, and then the thread exits, it calls call_thread_dtors</code></a> to call the functions in the Vec</code>, in reverse order of their registration, to ensure that object initializations and finalizations are nested.</p> Detach threads</h2> "Detaching" a thread declares that it will never be "joined" by any other thread, so it needs to release its resources on its own when it exits.</p> This is a fairly straightforward matter of just doing bookkeeping to keep track of whether a thread is in the detached state, and freeing resources when it exits if it is, but there are a few catches.</p> One is that we create threads with clone</code> with the CLONE_CHILD_CLEARTID</code> flag, which tells Linux to zero out our tid field when the child exits. That helps join_thread</code>, but for detached threads, it means that if we free our memory before we exit, the kernel will still think it needs to clear the tid field. Something else in the program could reuse our free'd memory, and it could get corrupted if Linux clears out what is no longer our tid field. So when a thread is marked detached, we call the set_tid_address</code> system call, passing it a null pointer, which effectively disables the CLONE_CHILD_CLEARTID</code> for the thread.</p> The other is that a thread's memory is freed by an munmap</code> system call, but if we call the usual munmap</code> function that wraps the system call, it will free the current thread, including the stack we're running on out from underneath us. The munmap</code> wrapper will then access freed memory trying to read its return address for the return jump.</p> To fix this, we need more assembly code. We need a special code sequence to make a munmap</code> system call, and then make an exit</code> system call to exit the thread, without touching the stack after the munmap</code>. Fortunately, this can be done in just a handful of instructions</a>.</p> Note that on 32-bit x86, we normally don't make syscalls directly, because the usual int 0x80</code> mechanism is very slow, and Linux provides a much faster way to make syscalls through the vDSO</a>. However, we can't use that here because the vdso is effectively another syscall wrapper that expects to store a return address on the stack. So we use int 0x80</code> so that we can completely prevent touching the stack. Fortunately, threads don't exit all that often, so the performance loss here isn't that important.</p> Synchronize threads</h2> It's common for threading libraries to provide synchronization primitives such as mutexes and reader-writer locks.</p> Mustang's main strategy is to use parking_lot</code></a> to implement these, since parking_lot</code> is a widely-used library, and there's not a lot to add here.</p> Except there's one place Mustang can't yet use parking_lot</code>: the Rust global allocator uses a Mutex</code>, because it's allocating memory for multiple threads from a shared heap. But the twist for Mustang is, parking_lot</code>'s Mutex</code> does global allocation.</p> Uh oh.</p> Allocator tries to acquire a lock, which tries to perform an allocation, which tries to acquire a lock, which tries to perform an allocation...</p> 💥</p> So what do we do? For now, Mustang has its own Mutex implementation</a>, built on top of atomics and Linux's futex</code>. It's nothing fancy, it's unfair, it's inefficient in a bunch of ways, and it's certainly not proven correct. But it's pretty simple and it passes all the tests! This is your regular reminder that simply implementing everything in Rust does not make anything automatically safer. Mustang is still experimental at this point, with lots of unsafe</code>.</p> But not cancellation</h2> The other really big piece of functionality that a libpthread would provide is cancellation</a>. Cancellation is very complex, and requires hooks in libc and very special handling of many system calls. However, Rust doesn't support cancellation, and it's relatively rare even in C code, so origin and mustang don't implement it.</p> Mustang organization</h2> The origin crate provides low-level but still somewhat Rust-idiomatic interfaces to process startup and shutdown, and now also threads. This accompanies the rustix</a> crate which provides Rust-idiomatic interfaces to system calls.</p> The c-scape crate provides libc and libpthread ABIs as wrappers around rustix and origin. Right now, this allows existing code, such as Rust's std</code> to run on origin and rustix without any extra porting work.</p> But also, code that wants to can bypass the c-scape compatibility layer, and call into rustix and origin directly. This eliminates some overhead, but more importantly, it offers greater safety and simplicity, because there are fewer raw pointers, raw file descriptors, and raw error return values.</p> And beyond!</h2> Of course, even rustix and origin are still very low-level. Most users should of course continue to use std</code>, and high-level libraries such as cap-std</a>. But perhaps someday there could be a way of using rustix in std</a>, simplifying the code by factoring out raw pointers, raw file descriptors, and raw error handling.</p> And perhaps someday, one could even imagine, an official Rust target</a> for Rust programs built entirely in Rust, with I/O safety down to the syscalls</a>.</p> Rust programs written entirely in Rust 2021-09-07T00:00:00+00:00 mustang</code></a> is a system for writing Rust programs entirely in Rust, meaning they don't use libc</code>, crt1.o</code>, or any C code. It's experimental, but it's complete enough to run a std</code>-using Hello World and other simple programs on Linux on x86-64, x86, and aarch64.</p> See mustang</code>'s README</a> for information on how to use it. This post walks through the major steps leading to this point.</p> System calls</h2> The first building block for mustang</code> was rustix</code></a>, a crate providing a syscall-like API, but which can be configured to use different backends, currently either libc or raw Linux syscalls. rustix</code> has been a place to help prototype the io-lifetimes</code></a> crate and I/O safety feature, support Wasmtime</a>'s WASI implementation, support cap-std</a>'s filesystem sandbox implementation, and to push the boundaries</a> of low-level performance while maintaining safe and relatively idiomatic Rust APIs.</p> These continue to be the main motivators for rustix</code>, and that's an important point: while mustang</code> isn't particularly practical, it isn't all a one-off effort. Some of the major pieces have real-world practical use cases.</p> The first part of the inspiration for mustang</code> was the realization that rustix</code> could be used in this way.</p> Process startup</h2> crt1.o and process startup has always had an aura of mystery about it, at least to me. There are parts implemented in assembly, parts that use obscure C compiler features, parts that are heavily conditionalized with ifdefs, and support for dynamic linking requires several particularly tricky features.</p> The other part of the inspiration for mustang</code> was the realization that simple Rust programs don't need most of what goes into crt1.o and libc startup code The minimum needed is just a little bit of assembly</a> to translate from the initial process state that the OS sets up into the Rust calling convention. Once Rust code is running, it just needs to compute argc, argv, and envp</a>, call any .init_array</code> functions</a>, and then call main</code></a>.</p> This code is in mustang's origin</code></a> crate; it's very small.</p> To tell the linker to avoid linking in crt1.o so that it uses origin</code>'s startup code instead, mustang</code> defines custom targets</a> named -mustang</code>. These primarily add -nostdlib</code> to the link command so that crt1.o</code> is not linked in.</p> Cargo's build-std</a> mode allows mustang</code> to easily compile the standard library with its custom targets, so it doesn't need to make any upstream standard library changes (for now...).</p> C-scape</h2> If we can start a process with Rust code, and make system calls with Rust code, what's left that's not Rust?</p> As a practical matter, Rust's standard library depends on libc</code> and uses many libc APIs. One option would be to build a runtime library that runs on no_std</code>. However, in the limit, that would essentially require a reimplementation of std</code>, which would be a lot of work, and a huge burden to maintain.</p> So mustang</code> instead defines C-compatible ABIs</a> providing functions like write</code>, strlen</code>, memcpy</code>, mmap</code>, and other things used by Rust's standard library, using rustix</code> internally to do actual system calls. These implementations are currently very minimal, but they are enough to support Hello World, running with std</code>.</p> And going forward, it should be relatively straightforward to add additional libc</code> APIs as needed by std</code>.</p> That said, the goal here is to just implement enough of libc to get std</code> working on top of it. Mustang</code> doesn't need to implement a whole new libc, which would be a lot more work, and much of it wouldn't be needed in Rust programs.</p> This code is in mustang's c-scape</code></a> crate.</p> The global_allocator</code></h2> Rust allows users to configure their global allocator with the global_allocator</code></a> attribute. The default global allocator ultimately uses libc malloc</code>. Writing a new malloc</code> is a lot more work that I was looking to do here, so the question here was, is there a nice Rust allocator crate that can be used with #[global_allocator]</code>?</p> It turns out, there are a few options, but one which turned out to be particularly easy to integrate was wee_alloc</code></a>. Built for reducing code size on WebAssembly, wee_alloc</code> doesn't do fancy optimizations, but it does plug directly into #[global_alloctor]</code> with no additional hassle, and it is portable, so it was very easy to set up. And it only uses a few things from libc</code> which c-scape</code> is easily able to support.</p> Putting it all together</h2> The mustang</code> crate pulls all these pieces together to make it easy to use them all at once. But note that since mustang</code> is aiming at supporting regular std</code>-using Rust code, it isn't something you explicitly call yourself. As such, it currently needs an extern crate mustang</code> to link it into the program.</p> A note about safety (and a lack thereof)</h2> There is often a temptation whenever one is working with C code to Rewrite It In Rust, however it's not always a good idea. In mustang</code>'s case, existing libc implementations are mature, robust, and well optimized. Libc interfaces are relatively portable, and on some operating systems, libc interfaces are the only stable interfaces to operating system functionality. There's no urgency to stop using libc or to rewrite it.</p> mustang</code> is inspired by the realization that some of the major pieces were already available, and the remaining pieces to get basic examples working could be written with a relatively quick effort. It's new code, it's experimental, it has a lot of unsafe</code> code, it lacks important optimizations, it's overall far less safe to use than just using libc, and it would take a lot of work to change this. But, it's fun, and educational.</p> It's also possible that in the future, mustang</code> could be part of a path to designing new low-level system features like command-line arguments, environment variables, or OS error codes are handled with more safety or other ideas such as first-class I/O</a>. This is also part of why the goal isn't to just build a Rust implementation of libc: many libc APIs consist of a small amount of code behind a function signature that uses raw pointers; just putting Rust behind those kinds of APIs isn't going to make them significantly safer. But that's another post; for now...</p> Hello, World!</h2> See mustang</code>'s README</a> for step-by-step instructions for how to build an run mustang</code>.</p> If you're interested in following the story, helping out, or even just asking questions, come say hi in the chat channel</a>, or file an issue</a>!</p> Broken Encapsulation 2021-09-02T00:00:00+00:00 What kinds of bugs should safety</em> in Rust protect against?</p> Rust clearly wants to say that safety is about protecting programs against some kinds of bugs, but not all possible</em> bugs. Where should the boundary be?</p> Safety should at the very least mean protection against memory corruption through dangling and out-of-bounds pointers. To do that, it's necessary to protect against all Undefined Behavior, because if behavior of a program is undefined, anything could happen, including arbitrary memory corruption.</p> There are also categories of bugs that we don't expect Rust's safety to protect against. For example, Eq</code></a> implementations must be reflexive, symmetric, and transitive, however it would likely be impractical to enforce all the invariants of all such APIs. Fortunately, violating these invariants doesn't cause Undefined Behavior; it just causes some algorithms to do the wrong thing.</p> So, Undefined Behavior is a very practical place to put the boundary.</p> However, there are several potential situations which do not necessarily involve Undefined Behavior, but which are still arguably within the spirit of Rust's safety:</p> I/O safety</a>: Raw file descriptors have the same fundamental properties that make raw pointers unsafe: They can dangle and they can be be forged. One crate operating on a dangling or forged file descriptor can end up doing I/O on file descriptors held in other unrelated crates. Without I/O safety, it's impossible to characterize the I/O of a crate without considering the behavior of all other crates it might be linked with.</p> (I/O safety does intersect with memory safety through mmap</code>, but using mmap</code> safely is non-trivial in any case, and this is not the only motivation for I/O safety.)</p> </li> POSIX's munlock</code></a> function: This function takes a raw pointer and length, but it's defined to fail gracefully if given invalid pointers. It doesn't mutate any memory, or cause any subsequent memory access to behave differently with respect to Rust language semantics, so it arguably never causes Undefined Behavior. However, if a crate is internally using locked memory to protect sensitive data, exposing munlock</code> as a safe function would mean that a wayward munlock</code> call in another crate could bypass the first crate's encapsulation and munlock</code> the memory, compromising the sensitive data.</p> </li> POSIX's write</code></a> function: this function also takes a raw pointer and a length, and also guarantees to not segfault or mutate any memory. POSIX isn't clear on whether write</code> has Undefined Behavior in the presence of data races or provenance violations or other infelicities with its buffer, but for the sake of this post, let's assume it doesn't. The memory is read by the OS, which one could argue isn't bound by the same rules as userspace. In that case, one can argue that write</code> never has Undefined Behavior. But making it safe would mean safe code in any crate could read encapsulated memory in any other crate, which seems outside the spirit.</p> </li> </ul> So, instead of Undefined Behavior</em>, a slightly more expanded concept that covers these cases might be described as Broken Encapsulation</em>. This is a superset of Undefined Behavior, because anything that causes Undefined Behavior can break any language-level encapsulation boundary. And like Undefined Behavior, it still excludes things like Eq</code>'s invariants.</p> Language-level encapsulation boundaries help in maintaining Reasoning Footprints</a>, especially in programs that contain many crates. It's what lets us look at an individual crate and understand its behavior in isolation, without having to think about whether any other crate in the program could accidentally observe the crate's internal data, do I/O on its internal file descriptors, or cause its internal secrets to be swapped out of memory and potentially compromised.</p> In practice, thinking about Broken Encapsulation is only slightly different from thinking about Undefined Behavior, but it also reflects a broader observation: Guarding against Undefined Behavior is about ensuring that certain kinds of bugs don't happen, while guarding against Broken Encapsulation is also about helping users build large programs out of smaller parts.</p> (renamed) 2021-07-12T00:00:00+00:00 Posish has been renamed to Rustix. Here's the new URL for this page</a>!</p> (renamed) 2021-07-12T00:00:00+00:00 Rsix has been renamed to Rustix. Here's the new URL for this page</a>!</p> I/O safety and speed: Why not both? 2021-07-12T00:00:00+00:00 Rustix: I/O Safety in practice</h2> The I/O Safety RFC</a> is now merged! But it's all fairly abstract, so what will this look like in practice with real APIs?</p> One of the ways is rustix, the fastest POSIX-ish, Unix-ish, Linux-ish, and libc-ish API for Rust!</p> And it's also I/O-safe! And memory-safe! These are actually the bigger motivators for this crate, and it helped guide the development of the I/O safety APIs. But it's also fast, which is fun, and shows that Rust's "abstraction without overhead"</a> is preserved. More on that later though. First, what's all this about safety, you say? I'm glad you asked 😉!</p> Safety</h2> Memory safety is of course an established concept for Rust. Memory-safe abstractions for POSIX/libc functions is a well-established idea, with Rust references, slices and return values in place of raw pointers, and rustix provides all this as well.</p> I/O safety is a newly-introduced concept</a> and is about treating file descriptors similar to pointers, with concepts of ownership</a> and borrowing</a>. See the I/O Safety RFC</a> and the io-lifetimes crate</a> for details. Rustix uses io-lifetimes types for all its APIs that work with file descriptors.</p> A tour of a function in rustix</h3> Let's take a quick look at the openat</code> function, which demonstrates several of the things rustix does.</p> The libc version of openat</code> looks like this</a>:</p> </span>pub unsafe extern </span>"</span>C</span>" </span>fn </span>openat</span>( </span> </span>dirfd</span>: c_int, </span> </span>pathname</span>: </span>const</span> c_char, </span> </span>flags</span>: c_int, </span> ... </span> ) -> c_int </span></code></pre> Rustix's looks like this</a>:</p> </span>pub fn </span>openat</span><P: Arg, Fd: AsFd>( </span> </span>dirfd</span>: &Fd, </span> </span>path</span>: P, </span> </span>oflags</span>: OFlags, </span> </span>create_mode</span>: Mode </span> ) -> Result<OwnedFd> </span></code></pre> Here's a breakdown of the differences:</p> pub fn openat<P: Arg, Fd: AsFd</span>>( dirfd: &Fd</span>, path: P, oflags: OFlags, create_mode: Mode ) -> Result<OwnedFd> </pre></code> Rustix's dirfd</code> accepts any type which implements the AsFd</code></a> trait, so you can pass in a &File</code> or &TcpStream</code> or other things conveniently and safely instead of having to call .as_raw_fd()</code> and pass in the result. AsFd</code> works like a borrow, and similar to Rust references, it's prevented from escaping and dangling.</p> AsFd</code></a> is similar to AsRawFd</code></a>, except that AsFd::as_fd</code></a> returns a BorrowedFd</code></a> instead of AsRawFd::as_raw_fd</code></a>'s RawFd</code></a>, which is an integer, it returns a BorrowedFd</code>, which is a special type scoped to the lifetime of the borrow.</p> pub fn openat<P: Arg</span>, Fd: AsFd>( dirfd: &Fd, path: P</span>, oflags: OFlags, create_mode: Mode ) -> Result<OwnedFd> </pre></code> Similar to nix</a>, rustix's path</code> is also generic and accepts any type of string that implements the Arg</code></a> trait, which includes &str</code>, &Path</code>, &CStr</code>, &OsStr</code>, and other string-like types in Rust's standard library. Actual POSIX-ish system calls expect NUL-terminated strings, so there are optimized paths that avoid dynamic allocation in most cases where you pass in something other than a &CStr</code>. And rustix avoids making any assumptions about PATH_MAX</code></a>, so the lengths of strings you can use is between you and the individual syscalls.</p> pub fn openat<P: Arg, Fd: AsFd>( dirfd: &Fd, path: P, oflags: OFlags</span>, create_mode: Mode</span> ) -> Result</span><OwnedFd> </pre></code> Similar to other libc-wrapping creates, rustix's oflags</code> argument takes an OFlags</code>, which uses the bitflags</code> crate</a> to create a type-checked flags type instead of a raw integer type. Rustix uses an explicit create_mode</code> argument instead of ...</code> so it avoids the unsafety of varargs. And rustix reports errors via a Result</code> instead of a plain c_int</code>, and returns a Error</code></a> inside the Result</code> instead of setting errno</code>.</p> pub fn openat<P: Arg, Fd: AsFd>( dirfd: &Fd, path: P, oflags: OFlags, create_mode: Mode ) -> Result<OwnedFd</span>> </pre></code> On success, rustix returns an OwnedFd</code></a> instead of a raw file descriptor, modeling the file descriptor's lifetime in the type system, so similar to a Rust Box<T></code> or other owning type, it's prevented from dangling.</p> OwnedFd</code> is very similar to std::fs::File</code></a>. From a user perspective, the main difference between OwnedFd</code> and File</code> is that OwnedFd</code> doesn't suggest any file-like purpose or behavior, it's just a generic owned file descriptor for any kind of resource. Most use cases will likely want to wrap OwnedFd</code> in higher-level types. When OwnedFd</code> is added to the standard library, I expect File</code> itself will be implemented in terms of OwnedFd</code> in this way as well.</p> pub fn</span> openat<P: Arg, Fd: AsFd>( dirfd: &Fd, path: P, oflags: OFlags, create_mode: Mode ) -> Result<OwnedFd> </pre></code> And finally, the openat</code> system call itself is documented</a> to not have any other side effects, so it has first-class I/O</a>, though at a fairly coarse granularity since the path</code> can contain ..</code> components and can thereby reference any path in the filesystem (if you're worried about that kind of thing, see cap-std</a>).</p> So with all these together, rustix's openat</code> is a safe function, both in memory safety and I/O safety</a>.</p> Cool cool what about speed though?</h3> On Linux on x86-64, x86, and aarch64, rustix uses avoids libc</code> entirely, avoiding errno</code> and pthread cancellation checking. On stable Rust it uses out-of-line asm to perform syscalls; on Rust nightly, it uses inline asm and, since it adds very little code itself, it can fully inline the syscall instructions into the user callsites. On all other platforms, it currently uses libc</code> and errno</code>, but it's factored to facilitate additional backend implementations in the future.</p> It also uses an optimized error type and the string argument optimization mentioned above. In microbenchmarks, at this moment in time, filesystem syscalls with string path arguments are about 15% faster than nix, and about 3% faster than libc with allocating temporary CString</code>s. And it uses the vDSO</code> to make clock_gettime</code> really fast on Linux (and to avoid using the super-slow int 0x80</code> mechanism on x86), just like Linux libc implementations do, but without depending on libc.</p> That said, the bigger picture is that most of the time for syscalls is spent in the OS itself, and rustix makes most syscalls only a few percentage points faster at most. But these days, with system calls often getting slower, it's still fun to have some gains in performance in this space, even if they are relatively small. And it demonstrates that the new abstraction of I/O safety doesn't entail new overhead.</p> Is that all?</h3> Rustix provides a few additional niceties, including:</p> It always uses 64-bit file offsets, so users don't need to juggle off64_t</code> vs. off_t</code>, or remember to pass O_LARGEFILE</code> when needed.</p> </li> On 32-bit Linux platforms that use direct Linux syscalls, it uses a 64-bit time_t</code> type when the underlying kernel supports it, so clocks won't wrap around in the year 2038</a>.</p> </li> And as a fun bonus, some names have been made more human-friendly and less historic-accidental, with names such as accept_with</code> and dup2_with</code> to take extra flags arguments instead of accept4</code> and dup3</code>, and seek</code> instead of lseek</code>. The code also uses doc aliases</a> so that you can still find things via the traditional C names.</p> </li> </ul> What about portability?</h3> Rustix isn't a fully-fledged portability layer; it doesn't support Windows and some of its APIs are OS-specific, and even OS-version-specific. Rustix doesn't do non-trivial emulation of features, and prefers to let higher-level layers provide that kind of portability when needed.</p> Many POSIX-ish APIs are very low-level, and not a suitable abstraction layer for efficiently implementing on different kinds of platforms. See Rust's standard library for an illustration of this: For example, std::fs::Metadata</code></a> is more abstract than struct stat</code></a>, and this gives platforms more options when implementing functions like std::fs::File::metadata</code></a>.</p> Other interesting cases are epoll</code></a> and io_uring</code></a>. Rustix doesn't yet have APIs for these, and it's an open question whether it's worth it. These APIs are very low-level, they have complex file descriptor ownership, and they are most often wrapped in higher-level APIs. It's not yet clear whether I/O safety makes sense at the abstraction level of these APIs, or whether they should just continue to use RawFd</code> and introduce safety at higher levels of abstraction.</p> What about close</code>?</h3> It's called drop</code></a> 😃.</p> Rustix doesn't currently have a close</code> function in its public API. It would be straight-forward to add it, and the signature would either have an OwnedFd</code> argument or a T: IntoFd</code> generic argument, to express that the file descriptor ownership is being consumed. However it'd also be redundant for most users. OwnedFd</code> calls close</code> in its Drop</code> implementation, so in most cases, the way to close a file descriptor is to simply drop it.</p> (Drop</code> doesn't have a way to report errors, so this may not be the last word on this subject, but this is a separate and much more complex topic involving things like NFS in async mode.)</p> How complete is rustix?</h3> It has everything that cap-std</a> and wasmtime</a>'s WASI implementation need, as well as a few other projects, though it doesn't have everything. If you find it's missing something you need, please file an issue!</p> Where does I/O safety go from here?</h3> Now that the I/O Safety RFC</a> is merged, my next step in this wing of the story is to prepare a PR adding OwnedFd</code></a>, BorrowedFd</code></a>, and friends to the Rust standard library. Once that's done, I'll update the io-lifetimes crate</a> to use the new standard library types and traits instead of defining them itself.</p> io-lifetimes defines more functionality than will be in the initial standard library PR, such as views</a>, from_into_fd</code></a>, and the AsFilelike</code></a>/AsSocketlike</code></a> Windows/Unix portability layer, so it'll remain useful for some time, but it'll get thinner as the standard library takes over more functionality.</p> And for the Rust ecosystem as a whole, the I/O safety RFC outlines the path forward</a>.</p> Introducing cap-std, a capability-based version of the Rust standard library 2021-06-14T00:00:00+00:00 Introducing cap-std</code></h2> cap-std</code> is a project to create capability-based versions of Rust standard library and related APIs.</p> Capability-based here means that the APIs don't access files, directories, network addresses, clocks, or other external resources implicitly, but instead operate on handles that are explicitly passed in. This helps programs that work with potentially malicious content avoid accidentally accessing resources other than they intend, and does so without the need of a traditional process-wide sandbox, so it can be easily embedded in larger applications.</p> Background</h2> Some of the most devious software bugs are those where the code looks like it does one thing, and usually does that thing in practice, but sometimes, under special circumstances, does something else. Here's a simple example using Rust's filesystem APIs:</p> </span>fn </span>hello</span>(</span>name</span>: &Path) -> Result<()> { </span> </span>let</span> tmp = </span>tempdir</span>()?; </span> fs::write(tmp.</span>path</span>().</span>join</span>(name), "</span>hello world</span>")?; </span> } </span></code></pre> The expected behavior of this function is to write "hello world" to a file within a temporary directory. The code looks like it will do this. And indeed, it will usually</em> do this. But if the path passed in is ../../home/me/.ssh/id_dsa.pub</code>, then the behavior of this function could be to corrupt the user's ssh public key 😲. That's... not remotely within what we said the expected behavior is. It usually doesn't do that, but under the right circumstances, it could.</p> And since name</code> is just a string, if the string is computed in a way that could be influenced by an attacker, the right circumstances could easily be made to occur in practice.</p> The cap-std</code> project</a> provides Rust crates with lightweight ways to avoid such problems. In particular, the cap-std</code> crate</a>'s Dir</code> type represents a directory, with methods</a> corresponding to Rust's std::fs</code> functions, for opening and working with files within the directory, that ensure that all paths stay within that directory. For networking, the Pool</code></a> type represents a set of network addresses, ensuring that all network accesses made through the API are to addresses in the pool.</p> In contrast to conventional sandboxing, cap-std</code> doesn't have any global state, so using it in one part of an application doesn't require using it in the rest of the application. Library crates can use cap-std</code> internally without imposing any sandboxing constraints on their users.</p> What can Dir</code> do?</h2> cap_std</code> is just a library, so by itself, it isn't a sandbox for arbitrary Rust code—it can't prevent arbitrary Rust code from using std::fs</code>'s path-oriented APIs. Instead, it protects against malicious content</em>, when filesystem paths can be influenced by untrusted inputs, and malicious concurrent modifications</em>, when another program running at the same time has the ability to remove, rename, or create files, directories, symlinks, or hard links in ways that could cause a program to inadvertently access unintended resources.</p> Revisiting our example above, with cap-std</code> we might write:</p> </span>fn </span>hello</span>(</span>name</span>: &Path, </span>tmp</span>: &Dir) -> Result<()> { </span> tmp.</span>write</span>(name, "</span>hello world</span>")?; </span> } </span></code></pre> In this code, if the passed-in path uses ..</code> to access directories outside of the one passed in, tmp.write</code> returns an error.</p> One key difference from before is that instead of creating the temporary directory itself, this function requests a directory be passed in. The Dir</code> type here serves as a "vocabulary" type, allowing the function to declare that it wants a directory to be passed in, and that it intends to access resources within that directory, rather than accessing arbitrary locations in the filesystem. These kinds of declarations can help reduce the reasoning footprint</a> of a function call.</p> The Dir</code> crate also makes it much easier to write this code robustly. There's no need to think about ..</code> or absolute paths at the application level, and no need to handle symlinks specially, which with the Rust standard library today isn't even possible to do robustly without platform-specific code</a>.</p> It also gives callers increased control. The caller gets to choose how and where to create the directory, and when to remove it. Callers could choose to use something like cap-tempfile</code></a>'s tempdir</code> function to easily create a temporary directory in a conventional location and automatically remove it afterwords, however they could also opt to create the directory somewhere else and manage it manually.</p> Note that Dir</code> is passed by immutable reference, even though it's being used to mutate external filesystem state. This follows Rust's conventions, for example in std::fs::File::set_len</code></a>, and it reflects an underlying truth about filesystems. &mut</code> in Rust is sometimes called an "exclusive" reference, because when someone has a &mut</code>, they're the only one which can access the underlying object. However, this is generally not a safe assumption when working with filesystem objects, because other programs could concurrently access or even mutate files or directories without Rust's type system having any say in the matter. Consequently, it makes sense to think of filesystem state as being external to the program, with File</code> and Dir</code> objects being just handles that are themselves typically immutable.</p> Dir</code> can also be combined with other security techniques. In a project which is written to carefully avoid using untrusted paths, it can add an extra layer of defense in depth.</p> And in the Wasmtime</a> project, the next step we described earlier</a> is now finished, and we're now using cap-std</a> in combination with with our WebAssembly sandbox to implement WASI, providing sandboxed access to system resources.</p> A simple example</h2> The main pattern for filesystem operations using the cap-std</code> crate is to obtain a Dir</code> and use methods on it, which closely resemble the functions in std::fs</code>.</p> One of the ways to obtain a Dir</code> is to use the cap-directories</code></a> crate to request a Dir</code> for a standard directory (similar to the directories-next</code></a> crate, but returns a Dir</code> instead of a Path</code>). For example, to obtain the data directory for an example program:</p> </span>let</span> project_dirs = </span> cap_directories::ProjectDirs::from( </span> "</span>com.example</span>", </span> "</span>Example Organization</span>", </span> "</span>`cap-std` Key-Value CLI Example</span>", </span> cap_directories::ambient_authority() </span> ) </span> }; </span> </span> </span>let</span> data_dir = project_dirs.</span>data_dir</span>().</span>unwrap</span>(); </span></code></pre> Then in place of fs::read</code> and fs::write</code> to read and write files, one can use data_dir</code> here to do data_dir.read(key)</code> and data_dir.write(file_name, value)</code>.</p> Note the use of the ambient_authority()</code> function here, which is a no-op that returns an instance of the opaque AmbientAuthority</code></a> type, and serves to mark a place in the code where ambient authority is being invoked. cap-directories</code>, and related crates have an overall invariant that functions don't create their own absolute filesystem paths, and always rely on resources being passed in as handles. Functions which don't uphold this invariant, such as cap_directories::ProjectDirs::from</code></a>, take an AmbientAuthority</code> argument to advertise their ability to open resources given only a string.</p> This makes it easy to search a codebase to find all the places where a non-sandboxed cap-std API is being used. It can also be scanned for with Clippy using a clippy configuration file</a>.</p> To see all this put together in a complete example, see the kv-cli example</a> in the cap-std</code> repository. This program implements a simple key-value store, using filesystem paths as keys, and using cap-std</code> ensures that it only accesses paths within its own data directory. Attempts to escape the directory with ..</code> fail gracefully. This is true even if a concurrently running program renames directories on the path or changes symlinks—something that's very hard to get right using std::fs</code> APIs.</p> $</span> cargo run</span> --quiet --example</span> kv-cli color green </span>$</span> cargo run</span> --quiet --example</span> kv-cli color </span>green </span>$</span> cargo run</span> --quiet --example</span> kv-cli temperature cold </span>$</span> cargo run</span> --quiet --example</span> kv-cli temperature </span>cold </span>$</span> cargo run</span> --quiet --example</span> kv-cli color </span>green </span>$</span> cargo run</span> --quiet --example</span> kv-cli /etc/passwd </span>Error:</span> a path led outside of the filesystem </span>$</span> cargo run</span> --quiet --example</span> kv-cli ../../../secret_cookie_recipe.txt </span>Error:</span> a path led outside of the filesystem </span></code></pre> Another useful crate is cap-tempfile</code></a>, which creates temporary directories and provides a Dir</code> to access them.</p> It's also possible to create a Dir</code> by opening a raw path, using Dir::open_ambient_dir</code>. Note that this function takes an AmbientAuthority</code> since it does not uphold the sandboxing invariant that the rest of the API does.</p> A real-world example</h2> Web servers often need to serve files from a given directory, and it'd be nice to have a guarantee that they don't accidentally stray outside that directory.</p> tide-native-static-files</a> is a fork of a real-world Web server project built on the Tide framework, ported to use cap-std instead of directory paths.</p> The port is very straightforward, mostly consisting of passing around a Dir</code> instead of a string holding a base directory name. And in many cases, working with a Dir</code> is actually simpler than working with a string. The complete set of changes needed for this port can be seen here</a>.</p> Implementation Landscape</h2> One of the reasons that Rust doesn't already have a Dir</code> type, when it does have a File</code> type, is that popular OS filesystem APIs don't make this as efficient or idiomatic as just using paths to name directories. However, this is changing.</p> One of the inspirations for cap-std</code> is the CloudABI project</a>, which among other things developed a technique of using a sequence of openat</code> calls to emulate path lookup in userspace in a way that's robust in the face of concurrent renames. cap-std</code> uses a variant of this technique, optimized to use fewer intermediate system calls, to implement a portable sandboxed path lookup algorithm.</p> And, Linux recently</a> added a system call, openat2</code>, which has the ability to restrict path lookup</a> so that it stays within a given directory, which is exactly the behavior we want here. It doesn't require a process-wide mode, and it avoids the overhead of doing multiple system calls. cap-std</code> uses this in place of its portable algorithm whenever it can. On systems which support openat2</code>, most functions in the API perform only one or two system calls.</p> Linux and other operating systems are also exploring adding more such features, and as these features become available, it will become increasingly practical to not just implement a Dir</code> type, but to implement it with WASI-style sandboxing protections built in.</p> Philosophy</h2> cap-std</code> came about because we were looking to generalize the filesystem sandboxing techniques we were using in Wasmtime's WASI implementation to make them more broadly applicable, and we were particularly inspired by async-std</code>'s philosophy</a>:</p> the best API is the one you already know.</p> </blockquote> Rust already has a standard library API. It's very good overall, and a lot of care has gone into ensuring that it's implementable on many platforms. It's used by a lot of code, and well known to a lot of developers.</p> cap-std</code> is an approach that takes advantage of this. Developers who know std</code> can easily learn cap-std</code>. Applications using std</code> can be ported to cap-std</code>, with the main concern being about how to ensure that directory handles are available to all the places that need them, rather than with dealing with differences in the API or in filesystem behavior.</p> The close alignment between cap-std</code> and std</code>, combined with the close alignment between async-std</code></a> and std</code>, also make it straightforward to do both at the same time, producing cap-async-std</code></a>.</p> If you're familiar with using std::fs</code>, you should be familiar with cap-std</code>'s APIs without any surprises. Similarly, if you're familiar with async-std</code>, cap-async-std</code>'s APIs should work as expected.</p> Current status</h2> Cap-std works on Linux, macOS, FreeBSD, Windows, and more, with stable Rust. On Linux, cap_std::fs</code></a> is optimized to use new system calls including openat2</code>, when available, which significantly reduces the sandboxing overhead.</p> Support for compiling to WASI is under active development.</p> Speaking of WASI...</h2> The sandboxing performed by cap-std</code> is the same as what's provided by WASI APIs.</p> While cap-std</code> is designed so that it can be used as a library within otherwise unsandboxed native applications, WASI applies the same kind of sandboxing to all filesystem accesses, so that it serves as an extension to the core WebAssembly sandbox.</p> This means that when the cap-std</code> library is compiled for the WASI platform, it will be able to bypass its own sandboxing techniques and simply call into the WASI system calls directly, achieving smaller code size and tighter integration with the underlying WASI platform.</p> The Future!</h2> We're continuing to add more testing, fuzzing, and optimization. A port to WASI is underway.</p> We're also starting to think about extending the capability-based model to more parts of Rust's API. The most obvious next step is std::net</code>, which is in a very early state right now, but this is a space we're thinking about for the future! Other areas that may be interesting include std::env</code> for information passed in by the host environment, std::process</code> for launching sandboxed processes, and anything else that allows programs to interact with the outside world.</p> And as we're doing with cap-directories</code></a> and cap-tempfile</code></a>, we're also interested in ways that we can do more than just translate the standard library API into a capability-based model, but also make the capability-based model easy to use.</p> First-Class I/O 2021-04-06T00:00:00+00:00 @withoutBoats makes this observation about Rust in "Notes on a smaller Rust"</a>:</p> Pure functional programming is an ingenious trick to show you can code without mutation, but Rust is an even cleverer trick to show you can just have mutation.</p> </blockquote> A particular aspect I'd like to explore here is: Can we apply this observation to I/O?</p> Haskell also has an ingenious trick to do first-class I/O without mutation. So, can we have first-class I/O in Rust that just uses mutation?</p> What is First-Class I/O?</h2> As in Haskell, first-class I/O in Rust would mean functions that do I/O would do so through values which can be passed around the program as arguments or return values:</p> fn </span>do_some_io</span>(</span>f</span>: &File) -> io::Result<()> { </span> ... </span>} </span></code></pre> File</code> here is an example of a first-class value which represents a resource that supports I/O. But unlike Haskell, instead of using monads, File</code> here just has side-effecting operations like read</code> and write</code>.</p> Of course, Rust already has lots of these kinds of types, including in the standard library with types like File</code> and TcpStream</code>. And a lot of Rust code already follows this pattern. Strictly speaking, many of the operations don't take &mut</code> references, but these types conceptually use interior mutability</a>, which isn't unique to I/O.</p> The remaining piece that Haskell has that Rust doesn't here is that in Haskell, all</em> I/O is done through values which are passed around through the program. Any function which does I/O says so in its signature. In Rust, when a function has a File</code> argument in its signature, you know it's going to do I/O using that File</code>, but a function which doesn't have File</code> or any other I/O type might still access files.</p> Of course, monads in Haskell have many other uses too. This blog post is just looking at I/O operations in function signatures, which has implications for how two otherwise unrelated pieces of code interact, as well as for one piece of code can redirect the I/O of another.</p> Example: Stdout</h2> An example in Rust of code that uses global I/O is std::io::stdout</code>. Any code can call this and obtain a Stdout</code> value that can do I/O without having any mention of it in the surrounding function's signature. Stdout</code> uses a builtin mutex, even though many use cases don't need that, and is a common performance pitfall. Diligent users can of course use StdoutLock</code> to reduce the performance impact in some cases, but what if we could eliminate the builtin mutex altogether?</p> What if stdout was a value that a program would acquire once, and then pass around to all functions that want to print to it? Using Rust's usual ownership and borrowing rules, it wouldn't need a mutex for many use cases. And users could of course still explicitly wrap it in a Mutex</code> or similar in cases where they really want shared access to it, just like anything else in Rust.</p> The io-streams</code> crate</a> has an implementation of this. The StreamWriter::stdout</code></a> function returns an output stream which writes to the process' standard output. It implements the standard Write</code></a> trait so it's easy to use, and it conceptually owns its resource, so it doesn't need a builtin mutex.</p> Behind the scenes, this function actually acquires a StdoutLock</code> to prevent accidental mixing of std::io::stdout</code> usage with StreamWriter::stdout</code> usage, to uphold its exclusive ownership assumption.</p> Example: Files</h2> As another example, any code can do File::open(...)</code> and pass it a string, to open any file in the process' filesystem namespace, without declaring it in a function signature.</p> This means that if you want to run piece of code that does this in a different directory, the only way to do so is to create a new process, with a new filesystem namespace. This is a very heavy-weight operation, both in terms of performance and memory usage, but also in terms of portability and complexity.</p> Some codebases have a convention of having a "root" path that is passed in that everything is relative to, which helps, but doesn't enforce that all paths are relative to the root, or that paths don't lead outside the root using ..</code>.</p> The cap-std</code> crate</a> has a Dir</code></a> type, which represents a directory, which can make filesystem access a first-class part of a function's signature, and allow callers to specify a different directory for the I/O to happen in. And, it also performs sandboxing, ensuring that paths relative to the Dir</code> stay within the Dir</code>.</p> Filesystems are effectively pools of aliased mutable state, and with relatively weak synchronization primitives. While cap-std</code> doesn't address all the problems that can arise from this, it can be one tool for helping manage this state with standard Rust idioms.</p> Tangent: Rethinking "pure" functions</h2> Rust doesn't have a way to declare functions as "pure", having no side effects.</p> fn </span>this_is_pure</span>(</span>x</span>: </span>i32</span>, </span>y</span>: </span>i32</span>) -> </span>i32 </span>{ </span> x + y </span>} </span></code></pre> (As an aside, this function could panic on overflow; in what follows, I assume "pure" permits panics.)</p> Features to enable this have been proposed</a> a few times, but they haven't been added to the language, in part because the need for such features tends to be lower in Rust than other languages. From an optimizer perspective, this property can often be inferred, at least in simple cases.</p> And from a programmer perspective, programmers don't need "pure" to know whether arguments are mutated or not, because in Rust, these things are already declared, reliably, in the signature. The presence or absence of &mut</code> or types with interior mutability tells you everything you need to know about which arguments could be mutated. As such, much of what "pure" would mean would be redundant with information which is already there:</p> /// Can you guess which state this function mutates? </span>fn </span>increment</span>(</span>x</span>: &</span>mut i32</span>) { </span> </span>// You're right! </span> x += </span>1</span>; </span>} </span></code></pre> The things that aren't covered by the signature are global mutable state and I/O. Global mutable state tends to be less important in Rust than other languages because Rust pretty strongly discourages global mutable state.</p> However, Rust doesn't outright prohibit global mutable state, and doesn't really discourage global I/O, which is present even in the standard library.</p> So instead of a "pure", that prohibits all mutations, the more interesting property would be what I'll call "explicit", which would mean "no global mutable state or I/O". When used on a function with no &mut</code> or interior-mutable types in the signature, "explicit" would be the same as "pure", and could be declared as "pure" to optimizers. However, "explicit" could be used in functions that do have mutable arguments too, where it would indicate that those are the only things that are mutated.</p> /// All mutations and I/O are accounted for! </span>fn </span>this_is_explicit</span>(</span>f</span>: &File, </span>x</span>: &</span>mut i32</span>) -> io::Result<()> { </span> writeln!(f, "</span>hello world</span>")?; </span> x += </span>1</span>; </span> Ok(()) </span>} </span></code></pre> The "explicit" property would also serve to indicate a lack of ambient authority</em>, from a capability-oriented security</a> perspective.</p> Would it make sense to add an "explicit" keyword to Rust then? Probably not as such; it's likely that the majority of functions in Rust would qualify as "explicit", so adding it as a function attribute would add a lot of clutter. A more ergonomic approach might be to give crates a way to declare that all their functions are explicit, with a way to opt out for individual functions.</p> A possible direction for future exploration would be to use the rustc driver API</a> to create a custom static analysis tool that could recognize some form of syntax for this and then check that "explicit" functions don't accidentally call "non-explicit" functions without explicit overrides, much like unsafe</code>.</p> Conclusion</h2> First-class I/O points to a way of thinking about software where "the machine" it's running on or "the process" it's running in aren't the focus. Instead of ad-hoc conventions for coordinating access to a shared filesystem namespace or other process-associated resources, code using first-class I/O can pass values around to manage its I/O resources similar to how it already manages other program resources.</p> First-class I/O can be a useful concept to apply broadly, such as how capability-oriented security</a> helps enable shared-nothing linking</a>. It can also be useful to apply incrementally, such as how the io-streams</code> crate</a> or the cap-std</code> crate</a> can help parts of a program cooperate with each other efficiently and idiomatically.</p> Thanks to Pat Hickey</a> and Luke Wagner</a> for feedback on this post!</em></p> Canonicalization 2018-10-22T00:00:00+00:00 Canonicalization and canonical forms are one dimension of organizing the work of an optimizing compiler.</p> Intro</h2> A lot of code constructs can be written in multiple ways. For example:</p> x + 4 </span> 4 + x </span> (x + 2) + 2 </span></code></pre> Canonicalization</em> means picking one of these forms to be the canonical form</em>, and then going through the program and rewriting all constructs which are equivalent to the canonical form into the canonical form.</p> In the case of add, it's common to pick the form that has a constant on the right side, so we'd rewrite all these constructs to x + 4</code>.</p> Why is canonicalization useful?</h3> The goal of canonicalization is to make subsequent optimizations more effective</em>. This is a key point, and we'll get into some of the subtleties below. But there are a lot of cases where it's just obviously a good thing to do. It means that subsequent optimizations that look for specific patterns of code only have to look for the canonical forms, rather than all forms.</p> Another way of saying this is, having a canonicalization pass is a way of factoring out the parts in a compiler that know all the different forms x + 4</code> could take, so that most optimization passes don't have to worry about this. They don't have to look for 4 + x</code>, because they can assume that looking for x + 4</code> covers that. Handy!</p> How do we choose a canonical form?</h3> Sometimes it's easy. The canonical form for 2 + 3</code> is 5</code>, because that's clearly simpler in every possible way.</p> Sometimes it's an arbitrary choice. It often doesn't matter than much whether one picks 4 + x</code> over x + 4</code>, but it is helpful to pick one or the other, so sometimes it's just human aesthetics.</p> It's tempting to pick whatever form would be fastest, or optimal</em>, on the target machine. And indeed, sometimes what's fastest aligns with what's simplest. 2 + 3</code> canonicalizing 5</code> is typically such a case. But sometimes it doesn't.</p> Yeah so what about x 2</code>...</h3> x * 2</code> is equivalent to x + x</code>; which of these should be the canonical form? It might seem like we might want to say: pick whatever's optimal for the target architecture. Addition is generally faster than multiplication, so that would suggest we pick x + x</code> as the canonical form.</p> But, x + x</code> can actually make things harder for subsequent optimizations, because it means that now x</code> has multiple uses. Having multiple uses makes some optimizations more complex -- in terms of the dependence graph, this is a DAG rather than a tree, and trees are generally simpler to work with. So maybe x * 2</code> is actually a better canonical form, even if it's a worse optimal form.</p> That said, we might consider canonicalizing this to x << 1</code>, which has only one use of x</code>, and has the nice property of making it as obvious as possible that the least significant bit of the result is zero. That way, we don't have to have as much random knowledge of multiplication by powers of two strewn throughout the compiler.</p> Efficiency on the target machine still matters, but we can defer thinking about that until codegen, where it's no longer as important to enable subsequent optimizations. At that point, we're going to start caring about picking between +</code>, </code>, and <<</code> based on which one executes fastest.</p> The basic philosophy of canonicalization says that canonical forms should be translated into optimal forms toward the back of the compiler, after all mid-level optimizations which benefit from canonical form are done. It's also worth noting that codegen itself benefits from having the code coming into it be in canonical form, so that it doesn't have to recognize all the ways to write x << 1</code>, and can just recognize one pattern for that and emit the optimal code for it.</p> This is often a source of confusion: Is canonicalization the same as optimization? It's often done as part of the "optimizer", and many of the things it does produce more optimal code directly. But ultimately, in its purest form, canonicalization just focuses on removing unnecessary variation so that subsequent optimizations can be simpler.</p> Canonical form, canonical form Canonical form hates optimal form They have a fight, canonical wins Canonical form…</p> </blockquote> (sung to the tune of "Particle Man" by They Might Be Giants)</p> Sometimes it's ambiguous: redundancy elimination</h3> Is redundancy elimination a canonicalization or an optimization?</p> It's certainly simpler to compute a given value once and reuse the value, rather than compute it twice. But does that aid subsequent optimizations? It depends.</p> A case where it does aid subsequent optimizations is when it eliminates redundant memory accesses. That way, it's essentially saying that no later passes have to even ask what the dependencies are for a given memory access, because the memory access has been eliminated.</p> A case where it doesn't is where it can take expression trees where everything has a single use and give some values multiple uses. Some kinds of optimization passes are harder to do on DAGs than on trees, so this can result in pessimizations in some cases, depending on what kinds of things the rest of the compiler is doing.</p> However, we typically do think of redundancy elimination as being a canonicalization. It's trivial to convert multiple-use values into single-use values by duplicating code, while going the other direction requires some analysis.</p> Even more ambiguous: inlining</h3> Inlining can act like canonicalization, especially in cases where the inlined function body can be optimized away. However, thinking of inlining in terms of canonicalization doesn't lead to a natural threshold. Should we maximally inline everything as far as possible? Or should we do the reverse and maximally outline and deduplicate outlined functions? Neither extreme is particularly practical, so most compilers use heuristics in practice rather than having a rigid definition of canonical form relative to calls.</p> That said, even though there's no clear boundary, we can imagine a rough guideline. Think of old-school C code, written at a time when "C is a portable assembly language" was more true than it is today, where calls that were important to inline were written as macros. In many ways, one of the jobs of compilers for higher-level languages is to compile them down to roughly this level, so that they can be optimized using optimization techniques which work well at this level. In theory, we could define the task of a canonicalizing inliner to just be to inline code down to what it would have looked like if that same code had been written in old-school C (at least with respect to inlining). That's not very pure, and it's difficult to precisely describe, but it's an intuitive and relatively practical compromise between extremes.</p> Canonical form isn't just for arithmetic expressions!</h3> Everything in an IR may be subjected to canonicalization.</p> For example, a control flow canonicalization might involve sorting the basic blocks of a program into Reverse Post-Order (RPO). Doing this is a simple way to ensure that the code is optimized the same regardless of how the user organizes the code inside their functions.</p> Dead code elimination is also a kind of canonicalization. The canonical form for dead code is no code.</p> In aggressive loop-transforming compilers, another form of canonicalization is to maximally fission loops into as many parts as possible, and then assume subsequent passes will fuse them back together into optimal loops that effectively utilize available registers and cache space.</p> Canonicalization as compression</h3> It's often the case that smaller forms are preferred over longer forms, so canonicalization tends to make code smaller, making it a form of compression. Also, since it effectively reduces non-essential entropy, it can make subsequent general-purpose compression more effective as well.</p> If we could conceptually perform all theoretically possible canonicalizations on a program, we'd end up with something related to its Kolmogorov Complexity</a> (it may not be identical, since canonicalization puts the needs of subsequent optimizations first, rather than absolute compression). Maximal canonicalization is frequently impossible in practice, because of the halting problem, but also because even in cases where it's theoretically possible, it can require impractical amounts of computation.</p> That said, it is pretty fun to think that for any given program, there is a theoretical "maximally canonical form" for that program, that all equivalent ways of writing that program could be reduced to. It is tempting to think of this as a kind of pure essence of the program.</p> Excessive canonicalization</h3> Canonicalization discards inessential information. However, sometimes that information can be useful to preserve.</p> An example arises in instruction scheduling: Say a user writes code like</p> x = a + b; </span> y = c d; </span></code></pre> Assuming there's no aliasing going on here, there's no reason why one of these statements has to be ordered before the other. Their order in the user's source code is inessential information. "Sea of nodes" style compilers may canonicalize to the point where there is no inherent ordering between these two statements.</p> The compiler backend ultimately has to produce machine code, which on conventional architectures requires the compiler to pick some</em> ordering. Compilers can be pretty smart, and can take into consideration many things, such as available execution resources before and after these statements to know what order a CPU would prefer to see them in. However, as smart as they can be, compilers can't always find the optimal answers. Optimal instruction scheduling is NP complete, but also, it may come down to runtime factors that ahead-of-time compilers don't have. And CPU hardware performance characteristics aren't always fully documented.</p> "Do No Harm"?</h4> Most software has no idea how it'll be mapped on to CPU pipelines. But some does. And it's these cases where getting the mapping right is most important.</p> On such software, optimizations that make actual improvements are fine, however it can be important that compilers not make anything worse</em> than if they had translated the code naively. Aggressively canonicalizing compilers have a risk that they will throw away information and at the end reconstruct a form which is worse than if they had just simply translated the code as it was written.</p> This, along with the ambiguous cases above, suggests that canonicalization be used in practical rather than rigid ways.</p> The theoretical shape of optimization.</h2> A compiler typically starts with human-written source code.</p> Typical human-written code follows various human-oriented sensibilities. Different humans may have different aesthetic sensibilities, and this can lead to writing the same code in different ways. But of course this isn't interesting to optimizers.</p> So the first then we typically do when the code hits the optimizers is to start canonicalizing. Throw away useless fluff that humans imbue their code with.</p> Canonicalizing optimizations are cascading; often doing one canonicalization will enable more. Folding an expression to a constant may allow other expressions to be folded to constants. Replacing a load by forwarding a value through a prior store may create a direct edge between two expression trees and introduce opportunities to simplify further.</p> So ignoring the practical concerns we mentioned above, we can imagine a theoretical compiler that does all the canonicalization it knows how to do up front. And while we don't realistically approach Kolmogorov Complexity levels, we do lift the program closer to what we might think of as its essence, the simplest form that does what it needs to do.</p> And then, the compiler can begin to optimize, rewriting canonical forms into optimal forms, as it lowers the code all the way down to assembly code.</p> For practical reasons, it isn't always possible to build compilers in terms of a purely canonicalizing phase and a purely optimizing phase, however this can be a useful reference point for understanding compilers in practice.</p>

sunfishcode's blog

Writing into uninitialized buffers in Rust

2025-03-11T00:00:00+00:00

Uninitialized buffers in Rust are a long-standing question, for example:</p>

https://internals.rust-lang.org/t/reading-into-uninitialized-buffers-yet-again/13282/4</a></li> </ul>

Recently, John Nunley</a> and Alex Saveau</a> came up with an idea for a new approach, using a Buffer</code> trait, which is now in rustix 1.0</a>, which I'll describe in this post.</p>

Update: This idea is now available in a standalone published library: buffer-trait</a>.</p>

Introducing the Buffer</code> trait</h2>
The POSIX read</code> function reads bytes from a file descriptor into a buffer,
and it can read fewer bytes than requested. Using Buffer</code>, read</code></a> in
rustix looks like this:</p>
pub fn </span>read</span><Fd: AsFd, Buf: Buffer<</span>u8</span>>>(</span>fd</span>: Fd, </span>buf</span>: Buf) -> Result<</span>Buf::</span>Output>
</span></code></pre>
This uses the Buffer</code> trait to describe the buffer argument. The Buffer</code> trait
looks like this:</p>
pub trait </span>Buffer<T> {
</span>    </span>/// The type of the value returned by functions with `Buffer` arguments.
</span>    </span>type </span>Output;
</span>
</span>    </span>/// Return a raw pointer and length to the underlying buffer.
</span>    </span>fn </span>parts_mut</span>(&</span>mut </span>self</span>) -> (</span>*mut</span> T, </span>usize</span>);
</span>
</span>    </span>/// Assert that `len` elements were written to, and provide a return value.
</span>    </span>unsafe fn </span>assume_init</span>(</span>self</span>, </span>len</span>: </span>usize</span>) -> </span>Self::</span>Output;
</span>}
</span></code></pre>
(And thanks to Yoshua Wuyts</a> for feedback on this trait and encouragement
for the overall idea!)</p>
(Rustix's own Buffer</code> trait</a> is sealed and its functions are private, but
that's just rustix choosing for now to reserve the ability to evolve the trait
without breaking compatibility, at the expense of not allowing users to use
Buffer</code> for defining their own I/O functions, for now.)</p>
Buffer</code> is implemented for &mut [T]</code>, so users can pass read</code> a &mut [u8]</code>
buffer to write into, and it'll return a Result<usize></code>, where the usize</code>
indicates how many bytes were actually read, on success. This matches how
read</code> in rustix used to work</a>. Using this looks like:</p>
let mut</span> buf = [</span>0_</span>u8</span>; </span>16</span>];
</span>let</span> num_read = </span>read</span>(fd, &</span>mut</span> buf)?;
</span>use</span>(&buf[..num_read]);
</span></code></pre>
Buffer</code> is also implemented for &mut [MaybeUninit<T>]</code>, so users can pass
read</code> a &mut [MaybeUninit<u8>]</code>, and in that case, they'll get back a
Result<(&mut [u8], &mut [MaybeUninit<u8>])></code>. On success, that provides a pair
of slices which are subslices of the original buffer, containing the range
of bytes that data was read into, and the remaining bytes that remain
uninitialized. Rustix previously had a function called read_uninit</code></a> that
worked this way, and in rustix 1.0 it's replaced by this new Buffer</code>-enabled
read</code> function. Using this looks like:</p>
let mut</span> buf = [MaybeUninit::<</span>u8</span>>::uninit(); </span>16</span>];
</span>let </span>(init, uninit) = </span>read</span>(fd, &</span>mut</span> buf)?;
</span>use</span>(init);
</span></code></pre>
This allows reading into uninitialized buffers with a safe API.</p>
And, Buffer</code> also supports a way to read into the spare capacity of a Vec</code>.
The spare_capacity</code></a> function takes a &mut Vec<T></code> and returns a
SpareCapacity</code></a> newtype which implements Buffer</code>, and it automatically
sets the length of the vector to include the number of initialized elements
after the read</code>, encapsulating the unsafety of Vec::set_len</code>. Using this looks like:</p>
let mut</span> buf = Vec::<</span>u8</span>>::with_capacity(</span>1024</span>);
</span>let</span> num_read = </span>read</span>(fd, </span>spare_capacity</span>(&</span>mut</span> buf))?;
</span>use</span>(&buf);
</span></code></pre>
In rustix, all functions that previously took &mut [u8]</code> buffers to write into
now take impl Buffer<u8></code> buffers, so they support writing into uninitialized
buffers.</p>
Under the covers</h2>
read</code> is implemented like this</a>:</p>
let</span> len = </span>unsafe </span>{ backend::io::syscalls::read(fd.</span>as_fd</span>(), buf.</span>parts_mut</span>())? };
</span>unsafe </span>{ Ok(buf.</span>assume_init</span>(len)) }
</span></code></pre>
First we call the underlying system call, and it returns the number of bytes it
read. We then pass that to assume_init</code>, which computes the Buffer::Output</code> to
return. The output may be just that number, or may be a pair of slices reflecting
that number.</p>
What if T</code> is not u8</code>?</h2>
Buffer</code> uses a type parameter T</code> rather than hard-coding u8</code>, so that it can be
used by functions like epoll::wait</code></a>, kevent</code></a>, and port::get</code></a> to return event
records instead of bytes. Using this can look like this:</p>
let mut</span> event_list = Vec::<epoll::Event>::with_capacity(</span>16</span>);
</span>loop </span>{
</span>    </span>let</span> _num = epoll::wait(&epoll, </span>spare_capacity</span>(&</span>mut</span> event_list), None)?;
</span>    </span>for</span> event in event_list.</span>drain</span>(..) {
</span>        </span>handle</span>(event);
</span>    }
</span>}
</span></code></pre>
This drains the Vec</code> with drain</code> so that it's empty before each wait</code>, because
spare_capacity</code> appends to the Vec</code> rather than overwriting any elements.</p>
There are no dynamic allocations inside the loop; SpareCapacity</code> only uses the
existing spare capacity and only calls set_len</code>, and not resize</code>.</p>
Alternatively, because Buffer</code> also works on slices, this code can be written
without using Vec</code> at all:</p>
let mut</span> event_list = [MaybeUninit::<epoll::Event>; </span>16</span>];
</span>loop </span>{
</span>    </span>let </span>(init, _uninit) = epoll::wait(&epoll, &</span>mut</span> event_list, None)?;
</span>    </span>for</span> event in init {
</span>        </span>handle</span>(event);
</span>    }
</span>}
</span></code></pre>
Error messages</h2>
One downside of the Buffer</code> trait approach is that it sometimes evokes error
messages from rustc which aren't obvious. This happened enough that we now have
a section</a> in rustix's documentation about them, and an example</a> showing examples
where they come up.</p>
Using Buffer</code> safely</h2>
Rust's std currently contains an experimental API based on BorrowedBuf</code></a>, which
has the nice property of allowing users to use it without using unsafe</code>, and without
doing anything hugely inefficient, such as initializing the full buffer. To
achieve this, BorrowedBuf</code> uses a "double cursor" design to avoid re-initializing
memory that has already been initialized.</p>
The Buffer</code> trait described here is simpler, avoiding the need for a "double cursor",
however it does have an unsafe</code> required method. Is there a way we could modify it
to support safe use?</p>
A Cursor</code> API like BorrowedCursor</code></a> could do it. That supports safely and
incrementally writing into an uninitialized buffer. And a key feature of
BorrowedCursor</code> is that it never requires the full buffer to be eagerly
initialized.</p>
With that, the Buffer</code> trait might look like:</p>
pub trait Buffer<T> {
</span>    // ... existing contents
</span>
</span>    /// An alternative to `parts_mut` for use with `init`.
</span>    ///
</span>    /// Return a `Cursor`.
</span>    fn cursor(&mut self) -> Cursor<T> {
</span>        Cursor::new(self)
</span>    }
</span>}
</span>
</span>impl<T, B: Buffer<T>> Cursor<T, B> {
</span>    /// ... cursor API
</span>
</span>    fn finish(self) -> B::Output {
</span>        // SAFETY: `Cursor` ensures that exactly `pos` bytes have been written.
</span>        unsafe { self.b.assume_init(selff.pos) }
</span>    }
</span>}
</span></code></pre>
This way, a user could write their own functions that take Buffer</code> arguments
and implement them using cursor</code>, without using unsafe</code>.</p>
Why parts_mut</code> and a raw pointer?</h2>
The parts_mut</code> function in the Buffer</code> trait looks like this:</p>
fn </span>parts_mut</span>(&</span>mut </span>self</span>) -> (</span>*mut</span> T, </span>usize</span>);
</span></code></pre>
Why return a raw pointer and length, instead of a &mut [MaybeUninit<T>]</code>? Because
a &mut [MaybeUninit<T>]</code> would be unsound in a subtle way. We implement Buffer</code>
for &mut [T]</code>, which cannot contain any uninitialized elements, and exposing it
as a &mut [MaybeUninit<T>]</code> would allow uninitialized elements to be written
into it.</p>
With a raw pointer, we put the burden on the assume_init</code> call to guarantee
that the buffer has been written to properly.</p>
Looking forward</h2>
A limited version of this Buffer</code> trait is now in rustix 1.0</a>, so we'll
see how it goes in practice.</p>
This idea is now also available in a standalone published library: buffer-trait</a>.</p>
If it works out well, I think this Buffer</code> design is worth considering for
Rust's std, as a replacement for BorrowedBuf</code></a> (which is currently unstable).
It's simpler, as it avoids the "double cursor" pattern, and it has the fun
feature of supporting the Vec</code> spare capacity use case and encapsulating
the unsafe Vec::set_len</code> call.</p>

Errors and Pipelines, a diversion

2024-09-10T00:00:00+00:00

Consider the following Unix shell command:</p>

$ a | b | c > d
</span></code></pre>
This runs program a</code>, pipes the output to program b</code>, pipes that output to
program c</code>, and finally redirects that output to file d</code>.</p>
If we think of the process tree, we could draw a diagram for it like this:</p>
             a   b   c
             ↑   ↑   ↑
             └─shell─┘
</pre>
The shell spawns processes for a</code>, b</code>, and c</code>, wires up their file descriptors
to connect their I/O in the desired way, and then waits for them to complete.</p>
From another perspective, this is a kind of call graph, in which the shell "calls"
a</code>, b</code>, and c</code>. They're on separate stacks and have separate threads of control,
so they can coexist, but the shell waits for them to complete and collects their
"return values".</p>
Anyway. A very common question from newcomers in Unix is, "what's the difference
between |</code> and ></code>?" It's easy, we tell them. |</code> pipes to a command, while ></code>
redirects to a file. So easy, we say. But is it really easy?</p>
A wild error appears</h2>
First let's consider the normal case. What happens if b</code> encounters an error?</p>
                 😱₀
             a   b   c
             ↑   ↑   ↑
             └─shell─┘
</pre>
Well, each of these programs has been invoked by the shell, which is wait</code>ing
for them to exit so that it can collect their error status. So b</code> will exit with
an error code, back to the shell.</p>
Unix shells will silently ignore such errors. Oops. So ok let's back up and remember
that if we're using Unix shell scripting for anything, we should
add set -euo pipefail</code> at the top of all our scripts</a>.</p>
Now the shell will notice the error.</p>
And then, the other commands in the pipeline will be terminated with SIGPIPE</code>.
Or if they ignore the SIGPIPE</code>, they will be given an EPIPE</code> error next time
they try to write output, which tells them to exit quietly. So the error
propagates to the shell and a</code> and c</code> quietly exit:</p>
            ❌  😱₀ ❌
            a   b   c
            ↑   ↑   ↑
            │   😱₁ │
            └─shell─┘
</pre>
Great.</p>
A wild error appears somewhere else</em></h2>
What happens if d</code> fails?</p>
Or wait, where even is d</code> in that diagram?</p>
Oh oh oh! That's the wrong question! Remember, we used ></code> for d</code>, so it's not
a program. It's just a redirection to an output file. So it can't fail!</p>
Or wait, it kindof can? Filesystems can run out of space. Disks can have errors.
Things can happen. But in that case, why can't these errors be reported in the
same way?</p>
Oh that's right. It's because unlike the file descriptors for a</code>'s and b</code>'s
outputs, the thing on the other side of c</code>'s file descriptor is just the operating
system.</p>
And operating systems can't fail. Or I should say, they can't
report a failure back to the shell, because the shell isn't thinking of the
OS as a process it's managing. The operating system can't push the error
directly to where it needs to go, so it instead pushes the error backwards
through the pipeline into c</code>, and obliges c</code> to report the problem to the
shell for it.</p>
          ❌  ❌  😱₁  😱₀
          a   b   c   OS
          ↑   ↑   ↑
          │   │   😱₂
          └─shell─┘
</pre>
It wasn't c</code>'s error. c</code> is just a stream transformer and really shouldn't
care about where the output is going or what ultimately happens to it, just like a</code> and b</code>.
But c</code> is the unlucky one that happens to be at the end of the pipeline, so it
gets the error and the job of ensuring that the error makes it back to the shell.</p>
So here's a question. What would be different about this process if
we inserted what might seem like a no-op into our command line:</p>
$ a | b | c | cat > d
</span></code></pre>
cat</code> defaults to reading from stdin and writing to stdout, so it's just
a no-op passthrough here. But inserting it does subtly change how the error
handling works. With a cat</code> in there, instead of the error being reported to
c</code>, the error is reported to cat</code>. That means that c</code> can now be simply
terminated with SIGPIPE</code> or told to exit quietly with EPIPE</code> in exactly the same
way as a</code> and b</code>.</p>
         ❌  ❌  ❌   😱₁  😱₀
         a   b   c   cat  OS
         ↑   ↑   ↑   ↑
         │   │   │   😱₂
         └─shell─┴───┘
</pre>
Which is kind of nice. And simple. It might look a little busier, but look closely
at what this says. a</code>, b</code>, and c</code> are all handled in the same way. c</code> is just a
stream transformer just like a</code> and b</code>. Because why should it be different?
Why should it have to know anything about error reporting for errors that happen elsewhere? It
doesn't need that if it's just writing to a pipe, because things like SIGPIPE</code> or
EPIPE</code> automatically handle it, just like in the others.</p>
What if we built a system that always did that for programs in pipelines, so that they
don't need to worry about doing this extra thing? Do what we already do for all the other
processes in a pipeline? And which would make it easier to write small programs
that do one thing and do it well.</p>
But we wouldn't want to always create an extra cat process. That would slow things down.
What we'd really want to do is design a system that just behaves like that without
needing an extra process. And perhaps such a system could explore making other things about
a pipeline more efficient as well. All those processes and context switching have a cost.</p>
That's it for now. I hope you enjoyed this diversion!</p>



Thinking about streams in WASI
2024-09-10T00:00:00+00:00
Streams are an essential feature of any large modular system. Whether they're in
the form of channels, pipes, sockets, iterators, async coroutines, or other
things, streams represent sequences of data over time, and can connect two pieces
of software with minimal coordination.</p>
WASI is being designed to support large modular systems, so WASI needs a
solid stream design. This is a blog post about streams in WASI.</p>
WASIp2 has been out a while, so we can now look at some experience with it
in practice. WASIp3 is a ways in the future yet, so we have some time to think
about what we want to do differently. So let's take a look!</p>
WASIp2 streams</h2>
WASIp2 streams have overall worked fairly well, and are used in APIs like
wasi-filesystem</a>, wasi-sockets</a>, and wasi-http</a>, providing a common
interface for streaming functionality. However, some of their big ideas
are limited in practice by the lack of integrated async.</p>
To block or not to block</h3>
WASIp2 has input-stream</code> and output-stream</code> types, in wasi-io</a>.
These have pretty ordinary bytestream APIs, although they do do one thing
that's different from what people may expect: Instead of being simply
blocking, or simply non-blocking, or even blocking-or-not depending on a
dynamic flag (O_NONBLOCK</code>) as in POSIX, WASIp2 streams have separate
functions for doing blocking and nonblocking I/O. The goal of this is to
fix the composition problem.</p>
A lot of people have heard of the "what color is your function?"</a>
blog post. In a sense, streams that are blocking or non-blocking also have
different "colors" which can present barriers to composition. Pass a
non-blocking stream in as the stdin of grep</code> in Unix and you get this:</p>
grep: (standard input): Resource temporarily unavailable
</span></code></pre>
So Unix's grep</code> needs a stream with the "blocking" color. Pass it the wrong
color and it doesn't work.</p>
The observation behind WASI's streams is that one typically has to write
different code to work with using blocking versus non-blocking streams. It's
not something that can be meaningfully switched at runtime based on a flag.
So WASI has separate API calls for blocking and non-blocking, so programmers
can just request what they want, and the stream can work in that way without
"color" being a property of the stream.</p>
It's a good idea in theory, though in WASIp2 it is somewhat hampered by two
issues. One is that the output-stream</code> API ended up being a little more
cluttered than one might have wanted. The flush</code> concept, in particular, is
something I hope we can improve in WASIp3.</p>
The other issue is that composability in WASI for anything involving I/O is
limited by WASI's lack of async</code> features. The poll</code> function</a> needs to know
about all types of things that might be polled for, which means that virtualizing
any API that involves I/O requires virtualizing the poll</code> function, which then
requires virtualizing everything else that uses I/O at the same time.</p>
It is possible to virtualize things, and tools like WASI-virt</a> do so, but the
virtualization ends up having to completely wrap everything, rather than being
able to virtualize individual APIs individually.</p>
And, input-stream</code> is a different type from output-stream</code>, so it's not possible
to directly connect stream output from one place to stream input to another.
And the reason they have to be separate types is that WASIp2 has no way to suspend
the writer of a stream while the reader is reading. One can connect readers and
writers in various ways using custom code in the host, but it can't be done
from within WASI.</p>
Fixing all these issues properly really needs async features built into the platform.
Fortunately, it was already clear from the beginning that async features would be
critical for many use cases, so there's already a plan to follow up WASIp2 with WASIp3
to add async as a headline feature. The idea is to replace poll</code>, input-stream</code>,
output-stream</code>, and all the rest of wasi-io.</p>
Streams versus files</h3>
From a POSIX perspective, the other big difference between WASIp2 streams and
POSIX streams is they WASIp2 doesn't have the "file is-a stream" property. Streams
are separate resources from files. You can request streams for reading from or
writing to files. And similar for sockets and other things. Libraries like libc
may hide this, but this is what's happening at the WASI level.</p>
I blogged about some of the reasons for doing this in "What does everything is a file do?"</a>.
Long story short, decoupling streams from files means better composition because
we can compose anything that works in terms of stream types without worrying about
whether they actually need to be file streams.</p>
These ideas will likely be more important in WASIp3, once we get more powerful
composition with async. Having them in WASIp2 has made things like supporting
libc more complex, and has made it less familiar to people familiar with POSIX.
At the same time, it has also been a step helping us get ready for WASIp3 where
these ideas will have more advantages.</p>
Several people have observed that these changes aren't particularly useful if
everyone is just going to use libc, which hides them. But the reason they make
sense for WASI is that in Wasm, not everything will want to use libc. In the "native" world, people
are often accustomed to thinking of C as being the only thing that really
matters, with all other languages simply being ways of producing the kinds of
programs that C could produce, and communicating with each other and the OS
as if they were C programs. But Wasm from the beginning has been on a path of
supporting other languages, including adding features like Garbage Collection
to the underlying platform. So to meet the needs of Wasm, WASI needs to address
languages that don't use libc. That does make some things more complex for people
who only care about C, but the tradeoff is we get to aim for a more unified
ecosystem.</p>
Also, even for C and C-like languages, WASI covers many areas not covered by
traditional libc APIs, such as HTTP</a>, and in these areas, users will more often
be using new APIs or new frameworks.</p>
WASIp3 streams</h2>
The biggest feature of WASIp3 is expected to be the addition of async support</strong>.</p>
Luke Wagner's talk on async in the Component Model</a> is a good introduction,
and discusses how WASI can fix the function coloring problem. Because Wasm VMs
are in control of the callstack, they'll be able to do stack switching as needed to do
things like bridge between async code calling sync functions. And this is
also the way they'll be able to bridge between blocking and non-blocking streams.
That talk goes into detail on a lot of material; here, I'll just highlight a
few aspects of how it affects streams.</p>
When async is a builtin feature of the platform, it enables a lot of different
possibilities. Imagine being able to link together code written in different
languages, in the same process on the same thread and using a single underlying
async event loop. That's powerful. That's where this is going.</p>
With my standards hat on, none of what I'm describing here is in an official
proposal yet, but these general ideas are shared by several involved parties.
If you have thoughts or questions or concerns or suggestions, please reach out
and let's talk!</p>
A unified stream</code> type</h3>
Some of the ideas can look surprising at first, especially if you're like me
and have an existing intuition based on existing platforms that are not designed
around async. For example, with async we can have a single stream</code> type, rather
than having separate input-stream</code> and output-stream</code> types. It's still
unidirectional, but it works as an input stream when it appears as a function
argument, and works as an output stream when it appears as a function return
value. In effect, streams</code> are less like first-class handles that can be
passed around anywhere, and more like coroutine enables, that simply mediate
between a caller and a coroutine callee that coexist and have a stream of values
flowing between them.</p>
For example, consider this signature for a hypothetical gzip function:</p>
gzip: func(input: stream<u8>) -> stream<u8>
</span></code></pre>
The argument is a stream<u8></code>, which is a bytestream. Because it's an argument,
it's an input stream that the function can read from. The return type is
also stream<u8></code>. Being a return value, it's an output stream that the function
can create and write to.</p>
Because it's the same type on input and output , it could compose with other
functions using ordinary function composition. For example, if we also have this:</p>
gunzip: func(input: stream<u8>) -> stream<u8>
</span></code></pre>
Then we could do compositions like gzip(gunzip(data))</code> to decompress and
recompress the data (as a silly example).</p>
These ideas aren't new. For example, gRPC also has a single conceptual
stream</code> type that works as input when it's an argument and output when it's
a return value.</p>
Nevertheless, these ideas are things I've had to take time to understand, as I'm
more accustomed to thinking of input and output streams as being independent
handles (file descriptors) that can be passed around, and these kinds of streams
work in different ways. I've explored questions like
how does error handling work</a>
and should closing an output stream be allowed to fail?</a>
and what happens if someone passed you an output-stream and then ...</a>
and where do output-stream errors go?</a>.
One big-picture takeaway, every time I think "what if someone passes you an
output stream and then...", I find that the answer is that I need to step
back a level and think about the system in a broader way.</p>
My posts here are a start on exploring these topics, though clearly much
more will be needed as WASIp3 progresses. And WASI will need bridges to
connect with existing languages and libraries that work in terms of
channel-like concepts, or other streaming concepts.</p>
But put all these pieces together, and we get the most plausible path
I'm aware of having been proposed to an ecosystem with a single stream</code> type
that connects all the streaming things, which seems like a valuable goal
to pursue for WASI.</p>
Stream of T</code></h3>
Another advantage of being a builtin type is that the stream</code> type can
more easily be parameterized on the data type. Bytestreams can do a lot,
but with a rich type system with things like resource handles, being
able to pass handles over a stream will be a powerful connecting primitive.</p>
From a Unix perspective, streams-of-handles is much like passing file
descriptors over a Unix-domain socket, but hopefully with a much less
complex API.</p>
From a higher-level language perspective, streams of typed values look
a lot like iterators.</p>
Summary</h2>
WASIp2 streams are here today. They are a transitional step between
WASIp1-style Unix streams and WASIp3's coroutine-style streams.
But because WASIp2 lacks integrated async, they have some limitations.</p>
WAISp3 will add integrated async, and make streams much more flexible
and powerful.</p>


Errors from `close`
2024-09-09T00:00:00+00:00
A while ago I wrote a blog post about "Bugs in Hello World"</a>, about how a lot
of programming languages' default way of printing to stdout silently swallow
errors.</p>
This led to a repo</a> for maintaining lists of languages that do and don't have
this bug, and some examples of how to fix the bug. For example, in C, the
fixed Hello World looks like this:</p>
#include </span><</span>stdio.h</span>>
</span>#include </span><</span>stdlib.h</span>>
</span>
</span>int </span>main</span>(</span>void</span>) {
</span>    </span>printf</span>("</span>Hello, World!</span>\n</span>");
</span>
</span>    </span>if </span>(</span>fflush</span>(stdout) != </span>0 </span>|| </span>ferror</span>(stdout) != </span>0</span>) {
</span>        </span>return</span> EXIT_FAILURE;
</span>    }
</span>
</span>    </span>return</span> EXIT_SUCCESS;
</span>}
</span></code></pre>
It's a little more verbose than the hello world code that's usually presented
to newcomers, but still manageable.</p>
So that's it, right? Do we now have a completely</em> bug-free version of Hello World?</p>
Well for completeness, we might observe that this version doesn't make any provision
for internationalization or accessibility. Not everyone will be able to read its output.
That's worth thinking about. However the rest of this blog post is about a different
topic, let's set this aside here.</p>
And if we dig further, we notice that this version doesn't call fsync</code></a>.
That means that if its output is redirected to a file, and it runs, and exits, and reports
success, then data might still be lost. If the power goes out, then once power is restored
and the machine boots up again, users may log in to find corrupted or even entirely missing
output. Even though the program reported succeeding. Uh oh!</p>
However, this is where we need to talk about expectations</em>. Simple command-line
programs that just read from stdin and write to stdout aren't expected</em> to
fsync</code> their output. The reason fsync</code> exists in the first place is so that
applications can chose not to call it when they think they don't need to, and
skip the overhead.</p>
Simple command-line programs are often just one step of a larger logical program,
and it would add useless overhead if every program in a pipeline called fsync</code>.
So the expectation is, programs only call fsync</code> when they are operating at a
scope where they know it's needed.</p>
For example, something like vim</code> does call fsync</code> when saving a file. If the
power goes out, you really want that file you were editing to have all your edits.
But something like grep</code> doesn't call fsync</code> when writing its output. If the
power goes out halfway through some script, it'll take down the script too.
So you can often just rerun the whole script from the beginning. And if the
script's output was really important, it can do a sync</code> itself. So it all kind
of works out.</p>
Our Hello World program here is a simple command-line program, so it isn't expected
to call fsync</code>. So we'll say that's not a bug.</p>
Ok great. That means we really are done, right?</h2>
Maybe.</p>
what</h2>
There's something of a longstanding debate.</p>
seriously?</h2>
So back in the day, when the Network File System (NFS) was first released, it was a triumph of
Unix design. The mighty filesystem could abstract over everything</em>, even the network.</p>
But back in the day back in the day, when Unix was being designed, even though disks and
tapes and things were slow, CPUs were also really slow. Moore's law was still just getting
started. And the Internet didn't exist yet. It would be decades before anyone would start thinking
about stuff like C10K</a>.</p>
And in that environment, filesystem APIs were designed to have lots of simple operations that
each only did a small amount of work, and each operation blocked until the operation was complete.
This made everything very simple.</p>
In the following decades, Unix systems would add various asynchronous I/O APIs, and some of these things do
make some things better for some applications in some settings. But most regular applications
on Unix platforms continue to use synchronous APIs for most things.</p>
Now, when NFS was introduced, all these little synchronous messages were going over the
network to a server, and applications were always waiting for the server to respond over
the network to each message before continuing. So NFS had a lot of waiting going on.</p>
To speed things up, the NFS designers introduced an "async" mode. This mode still presents
the appearance of synchronous I/O to applications, but under the covers it does some of the
work asynchronously. In particular, write</code>s appear to succeed, before the actual I/O is
completed on the server. This makes things faster, but also means that if an error occurs when
the data is sent, it's too late to have the write</code> fail with an error code. So write errors
end up getting deferred, and reported in later write</code> or other calls. That's different from
what applications previously expected, but in practice it's close enough that most things
basically work.</p>
And to be sure, we're still not talking about use cases that should be doing fsync</code> here.
It's not about ensuring the data reaches the actual persistent storage on the server, so that
it's safe from power failures. It's just about ensuring that the data reached the server and
the server didn't run out of storage space or quota, so that it's safe from being corrupted
even when there isn't a power failure.</p>
But there is always one write</code> that isn't followed by another write</code>: the last one. Errors
that happen during the last write</code> on a stream would then be reported in the final close</code>
call. From an OS perspective, this is fine. close</code> is just a syscall that can fail, like any
other syscall.</p>
However, despite it being nice and tidy from an OS perspective, userspace often doesn't check
for errors from close</code> in practice. For a lot of different reasons.</p>
For example, if we look at the system calls performed by our Hello World program above, there isn't
even a close</code> call:</p>
...
</span>write(1, "Hello, World!\n", 14)         = 14
</span>exit_group(0)                           = ?
</span></code></pre>
If the output of this application is redirected to a file on an NFS filesystem, then
that 14</code> returned from write</code> might just be the NFS subsystem pretending everything is
synchronous, despite not having sent the data to the server yet. If the server runs out of
storage space, the user runs out of quota, the network gets overloaded, or
other failures happen, our hello world program will silently swallow the error. strace
won't see it, because it'll happen implicitly when the process exits and the OS cleans up
all its open file descriptors.</p>
And to be sure</em> sure, we're still still</em> not talking about applications that are expected
to fsync</code> here. If you run out of storage space or quota, your output could be truncated,
and other applications could therefore see incorrect output, even if there isn't a power
failure.</p>
Is it a bug that we silently swallow errors here? Yes. Silently swallowing errors (unless
explicitly silenced) is always a bug</a>.</p>
But whose bug is it?</p>
Who will it be? Who will it be?</h2>
From the perspective we've approached it here, it first seems like this is the
application's bug. Linux's close</code> documentation</a> agrees, notably observing that
"careful programmers" should check for errors from close</code>.</p>
Fingers pointed.</p>
And it certainly is nice and tidy to just end the story there. The designers of Unix
intended for us to check errors from all syscalls, and close</code> is a syscall, so we
should check errors from it. And in C, failing to check errors from a function is
associated with careless programmers, so it's easy to just blame the programmers for
any problems that come up.</p>
And there are some programs that do manage to do this. Notably GNU command-line utilities
like grep</code> do close their stdio streams. So perhaps everyone should do this, if they
wish to think of themselves as "careful programmers".</p>
But it's not so simple. POSIX's own close</code> documentation</a> says that
"The close() operation itself need not block awaiting [...] I/O completion". So
programs written to the POSIX spec can't rely on getting errors from close</code> anyway.</p>
And in the real world, lots of other real-world applications, written by careful
conscientious programmers, don't check for errors from close</code>.</p>
Furthermore, even in applications which do close stdout</code>, it can cause problems. Some
debugging tools inject code that uses stdout</code> for output, which can break if stdout</code>
gets closed. Some C++ libraries use static destructors that produce output to stdout</code>.
These are real-world concerns; for example, at one time, LLVM tools were made to close
stdout</code>, to catch errors just as we're discussing here, and it caused so many problems
that this code was eventually reverted, and they no longer do.</p>
As an aside, sometimes when this topic is discussed, people bring up the idea that
instead of closing stdout, one could just dup</code> it and then close the newly-created
file descriptor. However, this isn't guaranteed to work, because file descriptions
are reference-counted, so a close</code> of a dup</code> may just decrement the reference
count without doing any extra work, so that doesn't seem to be a reliable answer.</p>
So not only do we not tell application programmers that they need to close stdout,
they face complicated problems when they do. Is it really their fault if they aren't
doing it?</p>
Maybe the real bug is in the design of the system that creates this situation.</p>
Pointing the finger elsewhere</h2>
Perhaps we can blame the designers of NFS for inventing this "async" mode and not
fully implementing the expected local filesystem semantics. Or blame their users
who demanded faster NFS performance. Or blame the Unix designers for selling us on
the promise of "everything is a file" and encouraging the computing world to build
up an entire ecosystem of software around synchronous I/O.</p>
Or perhaps we can blame system administrators who choose to use NFS.</p>
Or perhaps we can blame the causes of the I/O errors. If a goat chews a network
cable, we blame the goat for any data loss that results. When a user uses too much
space, we blame the user for using too much space. If those things hadn't happened,
none of this would have been a problem.</p>
But as in so many things in life, it's worthwhile to keep a question in mind: is the
goal just to find someone to blame? Or is the goal here to make systems more reliable?</p>
FUSE</h2>
And besides, it's not just NFS. Or other network filesystems that do similar things.</p>
This problem of errors from close</code> also comes up in FUSE filesystems too. The FUSE documentation</a> does
say that FUSE modules shouldn't return errors from close</code> if it's important that
applications see them. And on some platforms, these errors aren't even reported</a>.</p>
However nothing stops modules from ignoring this documentation, or in just ignoring
platforms other than Linux, so in theory they can also be having this problem.</p>
What if we say that checking for errors from close is optional?</h2>
What if we said that it's ok to ignore errors from close in general, and that applications
only have to do it if they have a special need to. If they're handling sensitive data,
or if they're talking to an unreliable filesystem, then they should check, but otherwise
they don't have to?</p>
But the problem is, most programs don't know either of these things. Most programs don't
know the context in which their users are using them. They don't know what importance
their users assign to their data. And, while it's possible to check for NFS, it's not
possible to check for "is this an unreliable filesystem" in general.</p>
Most applications don't have any way to be aware of such things.</p>
Who should</em> be responsible?</h2>
When I started this journey, I believed that applications were responsible. That's what
major platforms say in their documentation, so it's right by definition. Several years ago,
I was the one who made LLVM tools close stdout</code> originally, and defended that code for
years as waves of reports of problems with it came in.</p>
But eventually I stopped defending that code, and the problem reports kept coming in, and
the code was removed.</p>
Partly as a result of this experience, I came to believe that the platform documentation
was misguided. It's too easy. Platform developers can just hide behind the simple
stance that close</code> is just another syscall that can fail, and pointedly document that
"careful</em>" programmers will check for errors. If users have difficulty doing so, blame the
users.</p>
Instead, I came to believe that we needed to renegotiate this relationship.
The responsibility should be with some combination of the people who chose to use NFS
or FUSE, and the causes of the I/O errors. Users working with NFS should</em> monitor their
free space. Administrators should</em> be monitoring their networks and their hardware. And
everyone should watch out for goats. Because
so many applications don't handle these errors gracefully, users and administrators should
be doing this anyway. Users using FUSE modules should</em> be wary of modules that ignore the
FUSE API documentation's recommendations. And if they're doing those things anyway, from there,
it's not a huge leap to say that users' responsibility</em> to do them.</p>
And besides, network filesystems like NFS are less popular today than they once were. Computer
labs have been disappearing in favor of bring-your-own-device. Data is increasingly stored in
the cloud rather than on departmental servers. NFS is still out there, but it's far less
prevalent than it once was.</p>
And besides besides, programming languages which make async programming more convenient
are becoming more popular. If more programs were written in a way that calling write</code>
didn't block them from doing other work, then this whole situation might be avoided entirely.</p>
So these days, it's difficult to justify taking a strong stance that programmers
everywhere need to take on all the burdens that come with calling close</code> on stdout.</p>
So where are we now?</h2>
Platforms still blame applications for not checking errors from close</code>. And these
days, platforms have decades of inertia to justify not making any major changes.</p>
A great many applications don't check for errors from close</code>. And they also have
decades of inertia too.</p>
Neither side can be easily fixed.</p>
So mostly, this issue just persists through time. When problems in practice do happen,
there are usually other things around that can be blamed. I'm sorry your data got
silently corrupted. It's those goats. Eating our network cables. You know how goats are 😉.</p>
What I can say at this point is that I personally am not going to embark on a quest
to get application programmers to check for errors from close</code>.</p>
And so, my conclusion here is that, no, our hello world program does not have a bug.
It's fine. It's just fine.</p>


Capabilities and Filesystems
2024-07-05T00:00:00+00:00
When we first designed the filesystem API in WASI, we had a lot of questions.
When a Wasm module has an import</code>, what does that mean? How are import names
resolved? What do we know about the state of the world outside of what the spec
calls the "store"? Capabilities are pretty cool; what's the best way to
incorporate capabilities into filesystem APIs?</p>
At the time, we mainly wanted to keep our options open and avoid hard-coding
ambient authority anywhere, because we knew from other systems that retrofitting
capability-based security onto an ecosystem designed without it can be
prohibitively difficult. So we followed the example of CloudABI, which in turn
draws a lot from Capsicum, which combine filesystem APIs and capabilities in
practical systems.</p>
Today, we've learned a lot about how WASI is being used, and about what
imports are and how they're resolved in the component model, and how the overall
security model of the component model works. And with this knowledge, it's worth
taking a high-level look at WASI's filesystem API to see if we can make something
that works with the tools we have today and fits with the way people want to use
it today.</p>
WASI's current design</h2>
In WASI's current design, all filesystem API functions require a handle value to
be passed to them as an argument. Instead of having a plain open</code> function which takes
a string and resolves it in some implied namespace, there's just an openat</code>-like method
in the descriptor resource, which takes a descriptor</code> as its receiver parameter.</p>
    resource descriptor {
</span>        ...
</span>
</span>        open-at: func(
</span>            /// Flags determining the method of how the path is resolved.
</span>            path-flags: path-flags,
</span>            /// The relative path of the object to open.
</span>            path: string,
</span>            /// The method by which to open the file.
</span>            open-flags: open-flags,
</span>            /// Flags to use for the resulting descriptor.
</span>            %flags: descriptor-flags,
</span>        ) -> result<descriptor, error-code>;
</span>
</span>        ...
</span>    }
</span></code></pre>
The implied "self" argument handle identifies a directory tree, and the path</code>
string is resolved as a relative path within that tree. Programs are then given
an initial set of handles to work with via the "preopens" mechanism.</p>
Unlike in POSIX, this openat</code>-style function enforces sandboxing, preventing
absolute paths, or ..</code> or symlinks which would reference files or directories
outside of the handle's namespace.</p>
All these ideas come directly from CloudABI</a> and indirectly from Capsicum</a>,
and similar ideas were being adopted in Google's Fuchsia OS</a>.</p>
And to be sure, there are reasons why this approach makes sense, for security,
modularity, and portability. The areas around the roots of filesystems tend have
higher concentrations of security and portability hazards. Roots are where many "interesting"
directories like /etc</code>, /tmp</code>, /proc</code>, and C:/Windows</code> live. If we desire secure
and portable programs, we need to avoid them being especially aware of those directories.
Prohibiting absolute paths at the WASI level strongly resonates with that desire.</p>
And while we knew that banning absolute paths would break a lot of things, we had the
idea to add compatibility back in, using the techniques of libpreopen</a>. The basic idea
is to emulate absolute paths in libc, while still avoiding them at the WASI level.</p>
The cool thing about this approach is that, in theory, it achieves the best of both worlds.
We get compatibility for existing code and toolchains when we want that, and we also get
the option to have programs that don't use those compatibility layers and in return get
stronger security guarantees.</p>
Another theoretical win of this approach is that it avoids baking the concept of a "current working directory"
into the system. The current working directory is basically a file descriptor for a directory, except
that unlike normal file descriptors, this file descriptor is implicit. In Unix, this means it
needs its own special system calls like chdir</code> to do what open</code> does, and fchdir</code> to do what dup2</code>
does. So it has arbitrary restrictions, anywhere the specialized system calls don't cover things that
the general system calls can do. And, this implied file descriptor is per-process, making it inflexible
for multi-threaded applications. So instead, we kept the current working directory concept out of WASI
and aimed for an approach of emulating it in libc instead.</p>
This overall design means that Wasm toolchains and libraries can chose for themselves
whether to use the compatibility bridges if they need that functionality, or to lean
into the capability model and avoid having a filesystem root or current working directory
imposed on them if they don't.</p>
How it's going</h2>
This approach basically works, but it has several papercuts.</p>
It's less familiar. Practically everyone is familiar with traditional
filesystems and absolute paths and current working directories. And even though
Unix and Windows have concepts of directory handles, they're not widely used, so
not everyone is familiar with them.</p>
And it's less compatible. Practically every source language has a standard
library with an open</code> function that takes a string and opens a file in an
implied namespace. WASI's current approach requires all source languages to use
libc, or to do their own emulation. And lots of programs end up depending on
absolute paths, or symlinks to absolute paths, or the ability to canonicalize
paths into absolute paths, and WASI's current approach makes all that awkward.</p>
And, there still are places where things don't work quite right. In theory some
of these are fixable by doing more work in libc, but some are not.</p>
And unlike on Unix where one can often get away by assuming that everyone will
be using libc, on Wasm, many toolchains are not using libc. And indeed, we don't
want</em> every toolchain to have to use libc.</p>
New tools</h2>
When we started looking at capability-based security, one of the big ideas is to
avoid ambient authority</em>.</p>
But what we know now is, that there is no such thing as purely ambient authority.
Any system can be virtualized, such that any ambient authority can be sandboxed
and redirected. It's not always cheap or easy to do so, but it's always doable.
So fundamentally, ambient authority is always a question of degree and
granularity rather than being an absolute.</p>
So instead of just saying "ambient authority bad", we need to look at systems and
see the degree to which they achieve the Principle of Least Authority (PoLA) in
practice, and analyze what forces help or hinder it.</p>
In Wasm, let's imagine a component that imports a function named do-stuff</code>, with no
arguments. This function has the appearance of using ambient authority, because it's
just a function with no handle arguments. It would seem that anyone can call it
anywhere.</p>
   do-stuff: func();
</span></code></pre>
However, Wasm components don't have a global scope. It's not true that anyone can
call it anywhere. It has to be linked to something, at link time. And it doesn't
need to be linked to the same thing in every instance that needs it. So whoever
controls the linking process can chose to link this do-stuff</code> import to a do-stuff</code>
export of their choosing. They can even link it to a do-stuff</code> wrapper around
another do-stuff</code> implementation. That means it can be virtualized or attenuated
or refused entirely.</p>
The fact that there are no handle values being passed in as arguments just means
that instead of a runtime capability, this do-stuff</code> function provides a
link-time</em> capability. That might have consequences for granularity, but it'll
depend on how it gets used. So instead of calling it ambient authority and prohibiting
it, we can consider it, and think instead about how it'll get used in practice.</p>
Type safety and granularity</h2>
One of the reasons for wanting to avoid filesystem roots is that rooted
filesystem namespaces tend to bundle up many different logical resources
into a single namespace, which means the granularity of access granted
by that namespace is very coarse.</p>
We could push for finer-grained filesystem capabilities, however there are
options at the conceptual level. Instead
of exposing new capabilities through filesystem APIs, we'd really rather be
exposing new capabilities through dedicated APIs using resources, because
that's much easier to make fine-grained, typed, and robust.</p>
So if we push toward more typed APIs, then we can let WASI's filesystem API
just focus on providing access to files, and not be burdened with the
pressure to be finer-grained.</p>
Link-time versus runtime in the future</h2>
And there's one additional piece of this puzzle. What if make something like
do-stuff</code> use link-time authority, and then some time in the future decide that
we really need it to use runtime authority? For example, suppose some program needs
to have the ability to call two different do-stuff</code> backends, deciding between
them at runtime. If we were designing a new API, that'd be a use case for handles.
But if we didn't use handles at the start, are we stuck, at least until we can
do a semver break?</p>
This is likely a ways off in the future, but we now at least have an idea for
a future component-model feature, where we allow component instances to implement
resources. We could then have handles to component instances, and the instance exports
could satisfy the require methods of the resource type. If we had that feature,
then any interface with link-time authority functions could be converted into an
interface with runtime authority functions. This possible future path gives us
more confidence that we'll be able to evolve in the future when we need to.</p>
What does this look like in wasi-filesystem?</h2>
We could add a plain open</code> function, which takes a path argument and resolves it relative
to a filesystem root that comes with the instance that the function is imported from.
That makes this open</code> function use a link-time capability. It could support absolute
paths, symlinks to absolute paths, and all the rest. And, it could support a current
working directory concept.</p>
And then, we could add similar non--at</code> versions of all the -at</code> functions.</p>
This new API could live in a new interface that could coexist with the current
interface.</p>
Now, I believe we still don't want to be defaulting to exposing the entire
host filesystem to WASI programs. So we'll still want to have a simple VFS
system in Wasm engines, with roughly the same level of virtual "mounting"
that today's preopens provide, just using a link-time authority namespace
instead of actual preopen handles.</p>
What about ghosts?</h2>
Ghosts</a> are when logical resources are identified by integers or strings
or similar things, across code boundaries, and "ghosts" conceptually relay
the resources needed to resolve the references. In the case of filesystem APIs,
ghosts are present in the form of filesystem paths being passed around, which
requires any code accepting such a path to have access to the same filesystem
namespace.</p>
If a component ecosystem grows up where it's common to use filesystem paths to
identify resources between components, rather than handles, the ecosystem becomes
less composable and, has weakened security properties, and requires more
expensive virtualization.</p>
If we add an open</code> function, there is a greater risk that the WASI ecosystem
will acquire some ghosts, in the form of components passing around filenames
and assuming a common filsystem namespace, making them more expensive and
complex to virtualize.</p>
Now that we have resources though, we can see that they're relatively easy to use,
which makes me more optimistic that we don't have to worry as much about
people using filesystem paths in place of handles. This is something we can
continue to monitor as the ecosystem grows. For now, I think we can be
confident enough that we don't need to preemptively worry about it.</p>
Conclusion</h2>
With the new tools and understandings in Wasm components, I think we can
re-evaluate how wasi-filesystem works, to make it more familiar and more
compatible with existing tools, libraries, and applications.</p>


Bridging between source languages, in Wasm
2024-05-23T00:00:00+00:00
One of the core design goals for Wasm is to support code compiled from many
different programming languages.</p>
To this end, core Wasm's type system is very low-level. It's designed in view of
the fact that programming languages all have their own ways of doing things. Even
seemingly simple things like strings or dictionaries can have different semantics
or performance tradeoffs between different languages. Instead of imposing one
answer on every language, Wasm gives languages the flexibility to make their own
choices.</p>
That also means that whenever we have a Wasm program written in one language, and
we want it to be able to talk to another Wasm program, or a host, written in another
language, we need a way to bridge between those different languages.</p>
What are the options?</h2>
Looking outside of Wasm at how other systems provide for cross-language
interfaces, there are roughly three different categories of approaches:</p>

Point-to-point</li>
Language-family</li>
All-to-all</li>
</ul>
Point-to-point</h3>
Point-to-point</em> means connecting one specific language to another specific
language. For example, PyO3</a> is a system for connecting Rust and Python. With
PyO3, one might write Rust code, that talks to Python code, like this:</p>
use </span>pyo3::prelude::*;
</span>
</span>/// Formats the sum of two numbers as string.
</span>#[</span>pyfunction</span>]
</span>fn </span>sum_as_string</span>(</span>a</span>: </span>usize</span>, </span>b</span>: </span>usize</span>) -> PyResult<String> {
</span>    Ok((a + b).</span>to_string</span>())
</span>}
</span></code></pre>
In point-to-point systems, at least one of the two sides knows</em> what language
it's talking to. With PyO3, it's Rust code that knows it's talking to Python code.
That Rust code wouldn't be able to talk to any other language. The upside of that
tradeoff is that by being specialized to specific languages, point-to-point systems
can provide rich integration between languages. PyO3 can make much of the
expressivity of Python available to the Rust code, and it can do so efficiently.</p>
But the downside from a Wasm perspective is that we specifically don't want
interfaces where one side knows the source language of the other. The source language
of the guest shouldn't have to be aware of the source language of the host.</p>
Language-family</h3>
Language-family</em> systems connect a family of languages that all share something in
common with each other. For example, on the Web, many different JavaScript-like languages
can talk to each other by having both sides pretend to be JavaScript. On Unix-like operating
systems, many different C-like languages can talk to each other by having both sides pretend
to be C. And so on.</p>

</a>
</center>
Language-family systems tend to provide somewhat less rich integration than
point-to-point systems. You can't access every feature of Python if you don't
know that you're talking to Python, for example. Any unique feature of any
language that isn't common to the family as a whole tends to be hidden.</p>
For example, ClojureScript code can talk to any JavaScript-family language by pretending
to be JavaScript. JavaScript doesn't have the same types as ClojureScript, so
ClojureScript provides a function named clj->js</code> to convert ClojureScript values to
JavaScript values, and it's lossy:</p>
(clj->js [</span>:red </span>"</span>green</span>" 'blue])
</span>;;=> #js ["red" "green" "blue"]
</span></code></pre>
This uses a feature of ClojureScript called symbols, which JavaScript doesn't have. When
symbols get converted to JavaScript, they need to be converted into something JavaScript
does have, such as strings. This allows ClojureScript programs to talk to any other
language that thinks its talking to JavaScript. But, it means that ClojureScript can't
expect other languages to pass it back symbols. There's less integration.</p>
And at the same time, languages that aren't in the language family can't fully
participate.</p>
All-to-all</h3>
All-to-all</em> systems are designed to connect any language to any other language.
Each side is completely unaware of the language of the other side. Often this
is done using an Interface Description Language (IDL), so that the interface between
languages can be described in a language-independent way.</p>


</center>
All-to-all systems are common in RPC protocols, such as network protocols, where it's
especially desirable to be able to implement clients and servers in different languages.</p>
IDLs also have a lot of similarity with database schema languages. Both need to define
datatypes, and both typically have a strong need to keep the data independent of the
programming languages that will produce or consume the data. And, both have a need for
the data to be meaningful without existing within a particular address space or a GC
heap with an arbitrary reference graph.</p>
What Would Wasm Want?</h2>

</a>
</center>
First of all, there is no one answer that's best for all situations. There are tradeoffs
in each of these three categories, so no single cross-language interface system will work
best for all situations. All three should, and can, coexist within the Wasm ecosystem.
C developers can use .a</code> archives containing C code that links to other C code via C
ABIs. As Wasm GC matures, perhaps there will be conventional ways to link Java libraries
compiled to Wasm with other Java libraries, allowing any language in the Java language
family to be linked together with Java-level integration.</p>
At the same time, there is a need for an over-arching all-to-all system. Point-to-point
and language-family approaches can be nested inside of the over-arching system to provide
greater integration where needed. The all-to-all approach is the only way to ensure that
every language can participate in the overall ecosystem, without having to be a member of
the right language family, without having a blessed language that every language has to
pretend to be.</p>
This is one of the unique opportunities for Wasm. In contrast, Unix, the JVM, the CLR, and
JavaScript are all language-family platforms. They each start with their respective blessed
language, and oblige all other languages to talk to each other by pretending to be that
blessed language.</p>
Wasm, on the other hand, doesn't have an inherent blessed language.</p>
Ok wait. Wait just a minute. Everyone knows that C is the blessed language</h3>
The Wasm MVP</a> explicitly focused on C/C++. Many people's first introduction to Wasm
was filled with pointers and offsets and struct layouts. It gave a lot of people the
impression that Wasm was settling into its place within the grand Unix tradition of
using C as its blessed language of communication. Not everyone may be happy about
that, but a lot of people aren't surprised by it. If you had asked me before I started
working on Wasm where it would go, C ABIs are what I would have guessed it would use.</p>
And in many places in the computing world today; this was seen as inevitable, as C
is seen as the universal language. After all, some people observe, all information on a
computer is ultimately just bytes, and C pointers can point to any bytes in memory,
and that means C can talk to anything, in a way that most other languages cannot.
That makes C uniquely suited to be the universal glue between all languages.</p>
Except that it isn't.</p>
😱</p>
There are several wrinkles in that story when it comes to Wasm. One small wrinkle is
memory64</a>, which is linear memory with 64-bit
pointers. So we don't just have one C ABI; we have
at least two, because linear-memory addresses can be either 32-bit or 64-bit, and
programs compiled for one can't directly interoperate with programs compile for the
other, even if they're both pretending to be C.</p>
But we also have a big wrinkle: Wasm GC.</p>
This whole idea about how all data is just bytes doesn't work in Wasm.</p>
Wasm GC values are not accessible as just bytes. Wasm GC types can't</em> be pointed to by C
pointers. This means that C is not the fundamentally universal language on Wasm in the
way that it can be within the realm of Unix processes.</p>
And even beyond that, there are more wrinkles, such as data lifetimes. Nothing prevents
the data indexed by a C pointer from dangling when data is deallocated. For decades, C
got away with saying that Undefined Behavior was unavoidable, but today, many users are
demanding different answers.</p>
Clearly GC is the answer</h3>
As obvious as it seems in some circles that C ABIs are the answer, it is equally obvious
in other spaces that some form of Wasm GC-based ABIs are the answer.</p>
After all, if you look at the JVM, the CLR, or JavaScript, which are all very
popular platforms that run wide varieties of programming languages,
they all provide a set of GC types provided by the platform that everyone on
those platforms just uses. This is simple, efficient, and proven. And unlike C,
it doesn't have scary memory safety hazards. So it might seem to be the obvious
answer for Wasm.</p>
Except, there are winkles with that approach too.</p>
😱</p>
One is that even though C won't be the blessed language, linear-memory languages
still do matter, and they can't easily interop with GC types. GC types can't easily
point to linear memory, and linear-memory languages can't easily hold GC references,
so GC types aren't universal either.</p>
Another is that, in the spirit of Wasm as a whole, Wasm GC is being designed to be
as language-independent as possible, and this has led it away from attempting to
provide one-size-fits-all opinionated GC types for "string", "list", "dictionary",
and so on. One person's list is a growable array, while another's is a cons list,
and another's is a rope. In practice, programming languages have very different
needs, and Wasm instead aims to provide primitive constructs that programming
languages can use to build higher-level types.</p>
Yet another is that when people are using systems that don't need a GC, they often
don't want to have to use a GC. If we make GC be the universal glue, then we
add GC dependencies in places that don't want them.</p>
Do re me, RPC</h3>
Wasm isn't the first place in computing to have a need to connect different
languages without having a single obvious blessed language. Networking protocols
in particular are an area where no single language took hold, in part because
most languages' type systems have things like pointers to mutable data, which
is awkward to share over a network. Popular network protocols have often turned
to IDLs, such as OpenAPI, Protobufs, or others, which make them all-to-all
systems.</p>
Should Wasm use one of these existing RPC-based cross-language systems?</p>
Just as language-family cross-language systems have a place in the Wasm
ecosystem, RPC systems do too. And just as before, there is also an over-arching
need in Wasm for a common system.</p>
When we scale up software systems, they tend to become distributed systems,
so using an RPC protocol is tempting, as it would mean we'd be ready to go
distributed, out of the box. On the other hand though, one of the lessons from
CORBA is that making everything network-aware makes everything harder.</p>
And, encoding calls into bytes and decoding them on the callee side has
overhead, and it's overhead that would be difficult to optimize away in the
case where we have two components running on the same computer.</p>
So what we'd ideally want is a system that uses an IDL to achieve the same kind
of all-to-all cross-language properties that RPC systems have, but which is
isn't tied to either bytestream serialization or network awareness.</p>
The Wasm component model</h2>
The Wasm component model</a> is an all-to-all cross-language system. It has an IDL</a>,
and connects languages to each other without either side being aware of the other. And it
isn't tied to bytestream serialization or network awareness.</p>
There's a lot in the component model, but to get a taste of how it works, consider
a type like string</code>. There is no string</code> type in core Wasm, so string</code> is just a
type in the interface</em> type system. That means it doesn't have a fixed representation
or even a fixed set of operations. It's just a set of logical values, which for
string</code> is the set of all sequences of Unicode Scalar Values.</p>
Bindings for individual language work by encoding descriptions of how the Unicode
Scalar Values are represented within their languages. This avoids either side
of an interface knowing how the other side represents its values. And, it provides
enough information to linkers to insert whatever adaptation code is needed:</p>


If Wasm code is passing a string to the host, the host can just read
the string data straight from the Wasm code's memory. No copying is needed
in many cases!</p>
</li>

If Wasm code using UTF-8 strings is passing a string to Wasm code using
UTF-16 strings, the linking process can transparently insert UTF-8 to UTF-16
transcoding between then, so that strings can be passed without either side
knowing the encoding of the other side.</p>
</li>

If Wasm code is passing a string to Wasm code using the same encoding,
the data can be copied. A copy may sound expensive to some ears, but keep
in mind that this doesn't happen between component and host, it only happens
between two components. And in today's C-like ABIs, there isn't a way to
link two modules at all, so this isn't a regression of anything. And in GC land,
there are ideas for how even this copy could get optimized away in the future.</p>
</li>
</ul>
And because there is no serialization, and no object request brokers or network
awareness baked in, as compilers continue to optimize, component-model interfaces
will be able to be inlined, because everything compilers need to do inlining is
exposed up front.</p>
So there's a lot more to it than this, but hopefully this gives a taste of how
the system works.</p>
For more information about using the component model, see the component model
documentation</a>.</p>
Wrap up</h2>
There's a lot more to the component model, such as how the anticipated async
support avoids the function "coloring" problem, though see here</a>
for a preview. This blog post is just about
the cross-language aspects of the design.</p>
Wasm needs an over-arching cross-language interface system, if it's to avoid
long-term language-based fragmentation. The component model works differently
from what people expecting it to be just C ABIs expect to find, and also different
from what people expecting just GC types expect to find, but it has the properties
that a unified ecosystem needs.</p>


WASI 0.2 Launched
2024-01-25T00:00:00+00:00
The WASI Subgroup has just voted to launch WASI 0.2, also known as WASI Preview 2 or WASIp2!
This blog post is a brief look at the present, past, and future of WASI.</p>
The present</h2>
The Subgroup voted to launch WASI 0.2!</p>
This is a major milestone! We made it! At the same time, the journey is
only just beginning. But let's talk this moment to step back and look at
what this means.</p>
Most immediately, what this means is that the WASI Subgroup officially
says that the WASI 0.2 APIs are stable. There is still a lot more
to do, in writing more documentation, more tests, more toolchains, more
implementations, and there are a lot more features that we all want to add.
This vote today is a milestone along the way, rather than a destination in
itself.</p>
It also means that WASI is now officially based on the Wasm component model</a>,
which makes it cross-language and virtualizable. Figuring out what a
component model even is, designing it, implementing it, and building APIs
using it has been a huge effort with involving many people, and it's now
officially in WASI. Yay!</p>
WASI 0.2 includes two worlds</em>:</p>


wasi-cli</em>, the "command-line interface" world, which roughly
corresponds to POSIX. Files, sockets, clocks, random numbers, etc.</p>
</li>

wasi-http</em>, an HTTP proxy world, organized around requests
and responses.</p>
</li>
</ul>
There are more worlds in development, but for now, the important thing is
that we do have multiple worlds included. This means wasi-cli world isn't
the only world, or even the primary world. It's just one world, among
multiple.</p>
One things I'm looking forward to that's enabled by having multiple worlds is
Typed Main</a> (some details in that presentation are out of date by now, but
the big ideas still make sense), because we can have new worlds with new
entrypoints, with new signatures. It's also a part of how we can grow WASI to
fit into new non-traditional computing environments. And this is just the
beginning.</p>
Looking back</h2>
All the way back since the beginning of WebAssembly, people have been talking
about using it outside of browsers. Node.js had famously made JavaScript popular
on servers and more, and if it made sense for JavaScript, it was pretty natural to
imagine WebAssembly doing similar things. But beyond the basic outline, there
were a lot of questions, and a lot of ideas.</p>


How can we build a coherent ecosystem without fragmentation? And how do
we prevent an NPM-like situation with one company gaining control over the
ecosystem?</p>
</li>

To some people, POSIX was the obvious place to start on a non-JS-based API.
POSIX is a C API, so how do we ensure that our C ABI</em> is future-proof?
C ABIs tend to get baked in if one is not super careful, and bugs can
take decades to fix</a>.</p>
</li>

As much as POSIX was the obvious starting point for a lot of people, it was
just as much an obvious non-starter for a lot of other people. WASI being based
on POSIX made it very</em> biased toward C, with raw pointers everywhere, implied
data structures, and so on. And would there need to be a fork</code> function, with
all its ecosystem-wide implications? Building an ecosystem with C ABIs as its
primary connective tissue is very undesirable from a security perspective. Also,
what's the plan when Wasm GC arrives?</p>
</li>

Should we always export linear memory, to ensure that we can pass pointers
around? But if we do that, we lose the benefit of the sandboxing that Wasm is
otherwise doing between modules—if you can corrupt a C program's memory,
its game is over. But if we don't export linear memory, how else do we pass
around complex data structures?</p>
</li>
</ul>
Looking at use cases, there are the "port existing code" use cases, and the "do things
that other systems can already do, but do them IN WASM</em>" use cases, and these are
important. But there are also new use cases that we can imagine Wasm can do, and we
wanted to be sure we didn't define the system in terms of compatibility and exclude
these new use cases. One of the big themes that came up repeatedly was virtualization</em>.
All I/O in Wasm goes through its imports and exports, so we can completely virtualize a
Wasm module's view of the outside just by controlling what the imports and exports are
linked to.</p>
And, Wasm has a trusted stack, which is what makes it possible to call from
Wasm into JS and back within JS engines. In theory we should be able to use this
property to enable calling from Wasm into some other mutually untrusted Wasm.</p>
Also, early on, some security-minded folks told us that we should look at something
called "capabilities", which they said were really great. And avoiding global state
sounded like a good direction to go in. CloudABI combined capability-based security and
some impressive work to simplify the Unix platform down to a relatively small set of
carefully-designed primitives, which made it especially appealing. Consequently, much
of WASI 0.1 was closely derived from CloudABI.</p>
That helped get us started, and helps us build systems that people could write
code with and get things working, but we still had all these open questions, and
big ideas to figure out.</p>
Component Model enters the chat</h3>
So while WASI 0.1 was out in the world, people were starting to think big
thoughts about how to answer all these big questions and how to fit all the
big ideas into a coherent design. There were these early proposals called
"interface types" and "module linking", which seemed to be pointing toward
something, but had been through numerous iterations and hadn't quite settled
in yet.</p>
When I first heard about the idea for a component model, which would
subsume interface types and module linking, I didn't know what a "component"
meant. My knowledge of COM was "that's some Windows thing, right?". And I knew
only slightly more about CORBA. Mostly, I knew just barely enough to react
"surely we don't want to do that</em>". But as I got into it, I learned that
the Wasm component model was a chance to both learn from those existing systems
which had solved many of the problems we needed to solve, and also to learn
from those systems and do some things differently.</p>
We didn't want to force everyone to think about distributed computing just
to link libraries together. And we didn't want to lose the advantage of Wasm
sandboxing each module independently. And when we looked at where that leaves
us, the common theme that remained was composition</em>. It's about building
things that can be easily put together to make larger things.</p>
And then from there, composition ties together an entire family of ideas.
Even Mark Miller's PhD thesis</a> about capability-based security is literally
titled "Robust Composition". It's right there. It's what this is all about.
How do we put together parts to make a whole, without all kinds of complexities
creeping in?</p>


Cross-language interop is needed to compose components written in
different languages.</p>
</li>

Component isolation supports composition without the fear of components
having unexpected conflicts.</p>
</li>

No global namespace at runtime, so that composed components don't
collide in or have differing requirements of the namespace.</p>
</li>

Any interface can be virtualized, which is to say, composed with
any implementation of that interface.</p>
</li>

The output of linking two components is a component, so composition
can happen incrementally.</p>
</li>

Capabilities are a way to compose two components together, in which
they share some parts of themselves with each other and not others.</p>
</li>
</ul>
There's a lot more to these topics, but this is hopefully enough to paint
the picture that there are some themes that all fit well with each other,
within the conceptual framework of composition.</p>
And so, WASI 0.2 represents not just a new API, but a new approach to
APIs that gets us out of worrying about C ABI fragility, or C pointer hazards,
and that has reasonable paths forward for supporting many different
programming languages without hard fragmentation. It even allows us to not
worry so much about getting WASI 0.2 itself perfectly right for all time,
because we know we can virtualize WASI 0.2 itself in terms of future APIs.</p>
This is why the Subgroup voting to launch WASI 0.2 is such a big deal.</p>
Looking ahead</h2>
Ok, that's a high-level view of how we got here. What's next?</p>
In terms of standards, there are two activities that people are already
looking forward to. With WASI 0.2.0 finalized, it'll be time to start planning
for things we want to add in a WASI 0.2.1 soon after it. That'll be another
milestone: the first update to WASI 0.2, which will establish how to do updates,
and set up the rhythm for doing incremental updates. WASI 0.2 is small still;
we had to scope down a number of features that people really wanted, so we're
looking forward to doing many updates in 0.2.1 and beyond.</p>
At the same time, work towards WASI 0.3 will be getting underway. The
major banner of WASI 0.3 is async</em>, and adding the future</code> and stream</code> types
to Wit. And here again, the theme is composability</em>. It's one thing to do
async, it's another to do composable</em> async, where two components that are
async can be composed together without either event loop having to be nested
inside the other. Designing an async system flexible enough for Rust, C#,
JavaScript, Go, and many others, which all have their own perspectives on async,
will take some time, so while the design work is starting now, WASI 0.3
is expected to be at least a year away.</p>
And when we do get there, the transition from WASI 0.2 to WASI 0.3 should be much smoother
than WASI 0.1 to WASI 0.2. The component model's virtualizability means it
should be easier to polyfill WASI 0.2 in terms of WASI 0.3. And, it should be
easier for engines to support both WASI 0.2 and WASI 0.3 at the same time if
they wish to, because WASI 0.2 will essentially use a subset of WASI 0.3's
language features.</p>
Wrapping it up</h2>
WASI 0.2 is a major milestone that many people have contributed to, and
it's now launched! It's an API, but it also represents a new way to define
APIs for Wasm.</p>


(renamed)
2024-01-25T00:00:00+00:00
WASI Preview 2 is also known as WASI 0.2. and I've gotten feedback that that
name makes more sense to people, so I've renamed my post accordingly.
Here's the new URL for this page</a></p>


Eyra does the impossible
2023-11-15T00:00:00+00:00
Eyra</a> is challenging ideas about what it means to be a libc. In doing so, it's
doing a few things often considered to be... impossible</em> 😉.</p>
Fixing Rust's set_var</code> unsoundness</h2>
rust-lang/rust#27970</a> is a soundness bug in Rust. It is a way that Rust programs
can segfault without using any unsafe</code> code. The bug was opened 8 years ago,
and it's widely believed to be impossible to fully fix.</p>
One of the reasons it's so difficult is that it isn't enough to do locking,
even inside libc, because getenv</code> returns pointers to memory that need to stay
valid after the getenv</code> call returns. And, getenv</code>/setenv</code>/etc. can be called
by arbitrary C code, so any scheme that relies on all callers of getenv</code>
following a particular protocol isn't reliable.</p>
Eyra solves this by having setenv</code> etc. just leak the old memory. That ensures
that it stays valid for as long as any thread needs it. Granted, leaking isn't
great, and Eyra makes it configurable with the "threadsafe-setenv" cargo feature,
so it can be disabled in favor of the thread-unsafe implementation. However that
said, leaking is not unsafe</a>, and for some use cases, it'll be less bad than
the possibility of undefined behavior from dangling pointers.</p>
Eyra currently only supports Linux, and has other limitations, so it isn't a full
solution for all of Rust, but it does solve the problem within its range.</p>
Full host NSS and DNS support without dynamic linking</h2>
In glibc, a statically-linked binary, despite being statically linked,
heroically includes the ability to dlopen</code> NSS libraries as needed in order
to implement the lookup rules defined in "/etc/nsswitch.conf", and this means
that statically-linked binaries end up depending on the versions of the NSS
libraries that match the glibc version they were built with.</p>
In musl, a statically-linked binary just hard-codes the basic NSS and DNS
resolution strategies. It knows how to read "/etc/passwd", "/etc/resolv.conf",
and other files, and do the main things that one does with those files, so it
doesn't respect "/etc/nsswitch.conf" at all, and doesn't use the same name
lookup logic as other programs on a glibc-based distribution.</p>
These have long been the only options, but Eyra does something different.</p>
In Eyra, NSS functions are implemented by executing the external getent</code></a>
program, and parsing its output. getent</code> is present on both glibc</code>-based
and musl</code>-based Linux distributions, and has a stable command-line interface,
so Eyra programs do not depend on a specific version of libc being installed,
and it follows the system NSS and DNS configuration.</p>
(And to be sure, using getent</code> like this won't work for all</em> use cases, so
Eyra may add other options in the future.)</p>
Compiling whole programs with a single cargo build</code>.</h2>
Eyra is not the only system capable of whole-program optimization, however to
my knowledge, it is the only system that can compile a whole program, entirely
from source, in a single cargo build</code> invocation.</p>
This is achieved by using cargo's -Z build-std</code> option, which builds libcore,
liballoc, libstd, and other Rust libraries from source, by using Eyra which
builds the libc implementation from source, and, for completeness, by disabling
Eyra's "use-compiler-builtins"</code> feature, which tells it to use its own
implementations of memcpy</code> etc. instead of linking to Rust's prebuilt
compiler_builtins</code> library. Thay way, everything down to the OS boundary is
just a cargo dependency built from source. See the all-from-source example</a>
for more details.</p>
This may be useful for doing whole-program static analysis, for using
non-standard calling conventions, or anything else that requires that the whole
program be compiled together.</p>
What's Eyra?</h2>
Eyra</a> is currently a side project that I'm building for fun. If you think it
sounds interesting, please reach out!</p>


Embrace the Kinda
2023-05-31T00:00:00+00:00
So yeah, what...</p>
What is</em> Wasm?</p>
</a></p>
Well, I guess, one can find various one-sentence descriptions out there. The
webassembly.org website</a> leads with:</p>

WebAssembly (abbreviated Wasm</em>) is a binary instruction format for a stack-based virtual machine. Wasm is designed as a portable compilation target for programming languages, enabling deployment on the web for client and server applications.</p>
</blockquote>
Wikipedia's WebAssembly article</a> leads with:</p>

WebAssembly (sometimes abbreviated Wasm) defines a portable binary-code format and a corresponding text format for executable programs as well as software interfaces for facilitating interactions between such programs and their host environment.</p>
</blockquote>
The WebAssembly reference manual</a>
I wrote a few years ago leads with:</p>

WebAssembly, or “Wasm”, is a general-purpose virtual ISA designed to be a compilation target for a wide variety of programming languages. Much of its distinct personality derives from its security, code compression, and decoding optimization features.</p>
</blockquote>
And these are all like fine. But at the same time, these have all been out
there for years and here we are today and people are still trying to figure
out what this whole thing, just, like, is</em> is.</p>
Hang around Wasm spaces, and you may hear someone drop the quip:</p>

WebAssembly is neither Web nor Assembly!</p>
</blockquote>
It's a scintillating one. It pops. It packs that irresistible combination
of classic meme ancestry, hot new subject, apparent absurdity, and a real kernel of
truth all in one scrumptiously crunchy package.</p>
</a></p>
“Wasm is not Web”, the quote goes, “because it's carefully factored so that nothing
in the core spec has any dependency on browsers. And we're using it in servers
and embedded devices and stuff!” And it's right!</p>
At the same time, Wasm does come from a Web context, and many of the forces
shaping it have relationships with the Web. Wasm's concept of program isolation,
with a trusted callstack enabling it to call out into code that it doesn't trust
and doesn't trust it, and its system of imports and exports, come from the need
to embed Wasm within very complex browser environments. Wasm has a standards body which is
a W3C Community Group. And in many ways, “the Web” itself has also
grown beyond its original scope, into an interconnected ecosystem of scopes,
which Wasm participates in in multiple ways. Is ActivityPub</a> Web? Well... yeah.
Is Web of Things</a> Web? Still yeah. And of course, Wasm also does run in actual
browsers! So if we want to fully</em> understand why Wasm works like it does, both
on a technological and social level, we can't ignore</em> the Web side of the story
either.</p>
So is Wasm “Web”?</p>
Perhaps the shortest way to say it would be... well...</p>
Kinda</a>.</p>
“...or Assembly”, the quote goes on, “because it has a lot of structure and a
type system”. And this is also true; assembly languages as we know them today
all basically look alike if you squint a little. Wasm looks distinctly
different.</p>
But here too, Wasm's instruction set is also very low-level, retaining many of the
design characteristics of a typical assembly language, like having a single
i32</code> type that's neither signed nor unsigned, leveraging the magic of two's
complement, just like most machine ISAs do. And in some of the major compilers
that have been ported to target it, Wasm is emitted by the part of the compiler
that's built to emit assembly code. It's the “architecture”, in compiler parlance.
And this is one of the major forces behind Wasm's overall design, so if we ignore
it, we aren't seeing the whole picture.</p>
So is Wasm “Assembly”?</p>
Well...</p>
</a></p>
Kinda.</p>
Kinda???</h2>
Kinda! Our existing mental categories are important, because they're how we
approach a new system, and they're how we port existing code to new systems.
So we do talk about Wasm in terms of other systems, and we do build bridges
to enable concepts and code to be ported over.</p>
At the same time, attempting to understand Wasm as categorically one existing
category or another will limit our potential to understand it and to take
advantage of its strengths.</p>
Cool story</h2>
Is a Wasm instance a process</em>? Kinda. It can have an address space, like
a process does. And it has some forms of isolation from other instances,
somewhat like processes do from other processes. But it can also import its
entire address space from another instance, or not have an address space at
all! And it doesn't carry around all the state that Unix attaches to its
processes. And the address space doesn't contain things like the call stack
or the program's executable code.</p>
So we do sometimes use the word “process”, in contexts where that word fits.
We value compatibility with existing software, and existing software often
expects to have a process, so we do things to present the illusion that an
instance is such a process.</p>
Sometimes we use the word “nanoprocess”, which highlights how Wasm processes
perform isolation without relying on heavyweight host process boundaries,
enabling it to scale to very many instances runnining on the same machine at
the same time, like little nanogears.</p>
</a></p>
But instances also have some superpowers that processes don't have. Instances
can call</em> each other. On the same callstack. This is difficult to even
contemplate if we limit ourselves to a Unix perspective, and it creates
interesting new opportunities for what Unix would call “Inter-Process
Communication”, but with very little ceremony. It's just a call.
Compatibility is important. Taking full advantage of powerful new tools is
also part of the big picture here.</p>
</a></p>
Look I think I've got a pretty good picture</h2>
Is WASI an OS? Kinda. It's a framework for defining APIs, which could represent
an OS, and plays the role of an OS from the perspective of compatibility with
a lot of existing software.</p>
But it doesn't have</em> to be an OS. WASI APIs could be implemented as an adapter
library on top of other APIs, WASI or otherwise, allowing a program written for
one environment to run in another. So it's kinda?</p>
Is Wasm a container? Kinda. It can perform isolation, protecting
the inside from the outside and vice versa, and it can be a single file that
contains everything needed to run a program. And as the tools mature I expect
it'll be adding a lot of the features found in container systems. But at the
same time, it's also kinda different. As in our discussion of processes above,
Wasm programs can straight-up call functions in other programs.</p>
Very interesting now lets...</h2>
Is Wasm a stack machine? Kinda. It has a stack-machine instruction encoding,
where most operands don't need to be explicitly named, as they can be
implicitly “pushed” and “popped”, which compresses the binary encoding. But
it's a heavily constrained stack machine. Wasm's validation will check that
at every point in the code, the exact size of the stack, and the types of
everything on it, are known at compile time.</p>
This enables optimizing Wasm engines to translate the stack machine into
a register-machine IR, with virtual registers that then get register-allocated.
And of all the Wasm producers I've seen, they all either internally work in
terms of registers, or an abstract syntax tree, and just translate into
pushes and pops at the last moment. So almost nothing in the overall ecosystem
actually operates at the pushes-and-pops abstraction level, except for the
binary encoding and a few tools that operate on it. So the stack-machine
personality is only a surface appearance.</p>
Is Wasm a Reduced Instruction Set Computer (RISC)? Kinda. It does have
explicit load and store instructions, making it a load-store architecture,
and most of its arithmetic operations are very simple RISC-like instructions.
But it also does have fairly complex instructions like call</code> and memory.grow</code>.</p>
Is a Wasm component an executable, or a library? Kinda... both? In the
component model, those aren't fundamentally different things. There are
components that can be used in the manner of commands, but these same
components can also be linked to like a library.</p>
... can we wrap this up?</h2>
Is Wasm a bytecode? Kinda. Or, well, yes. Yes it is. But the kinda here is
because if we reduce it down to just</em> “a bytecode”, it's easy to miss what
the big deal is. We already have multiple popular bytecode-based systems
out there. What makes Wasm different?</p>
Wasm has a different approach to isolation, which makes it easier to embed
within existing systems and compose with itself without compromising its
security model. Wasm doesn't have a single favorite source language,
and aims to be language-neutral from the beginning. And, Wasm isn't strongly
associated with a single large corporation. It's defined by a standards
body, and in practice it has active participation and meaningful consensus
process involving a wide variety of organizations.</p>
Is Wasm a platform? Kinda? It's kinda more like a substrate on which platforms
can be defined, along with a shared ecosystem that can be used on such
platforms. WASI has a concept of worlds</a> which are each like their own
platforms, defining all the APIs available to programs within them. And
embedders can define their own worlds, making their own platforms. So in a
sense, we're building a meta-platform, plus a number of platforms inside.</p>
But at the same time, with virtual platform layering</a>, programs written for
one world can be adapted to run on another. And many ecosystem tools and
libraries will be able to be shared. So there is also a sense in which
all worlds are part of a single overarching meta-“platform”.</p>
Maybe stop now?</h2>
Is Wasm a programming language? Kinda. It does have a type system, and a
syntax. And function calls with arguments and return values instead of a
“put things in certain registers and memories” calling convention. Tail
calls have to be built in rather than being just compiler cleverness. It even
has an if</code>/else</code> construct and eschews goto</code>. But it's also too
low-level to be really written by hand on a regular basis.</p>
Ok, ok. Here. Is Wasm a Lisp?</h2>
Well... ok, fine. You got me there. One could try to make the case for Wasm being
a Lisp with the S-expression text syntax, and like, being Turing-complete?
And it's gaining tail calls and GC, and it is</em> getting increasingly interesting
to compile Lisp-family languages to. But ultimately, it's not recognizably a
Lisp.  There are no builtin concepts of cons</code> lists, eval</code>, macros, and so on.</p>
But here we are. A sea of kindas. And we can, and do, simplify them into the
familiar concepts and categories that people know and that existing programming
languages and existing code is often assuming. Giving people onramps and
compiling existing code and running existing applications are all critically
important for Wasm to succeed.</p>
At the same time,</p>
Let's have fun, and embrace the kinda!</p>


The Filesystem Namespace
2023-04-17T00:00:00+00:00
This post is in a series about "Everything Is A File":</p>

Is Everything A File?</a></li>
Measuring System Interface Complexity</a></li>
What does Everything Is A File do?</a></li>
The Filesystem Namespace (this post)</li>
</ul>

Blog post content coming soon!</p>


What does Everything Is A File do?
2023-04-13T00:00:00+00:00
This post is in a series about "Everything Is A File":</p>

Is Everything A File?</a></li>
Measuring System Interface Complexity</a></li>
What does Everything Is A File do? (this post)</li>
The Filesystem Namespace</a></li>
</ul>

In particular, this post is looking at the "Everything is a file descriptor</em>"
meaning.</p>
To see how the Everything Is A File approach works in practice, let's look at
three kinds of classic resources that have been a part of Unix and the
Everything Is A File tradition for a long time.</p>
First, actual files.</p>
</a></p>
Files are resizable arrays. They support random access to their data. Streaming
access patterns are implemented on top of this, using a "current position"
cursor.</p>
Next, serial ports.</p>
</a></p>
Serial ports can transmit and receive bytes. There are many protocols that can
be used with serial ports, but here, we're talking about Everything Is A File,
so we're going to look at what you get when you talk directly to a serial-port
device file, such as /dev/ttyS0</code> on a Unix system. This is a higher-level
interface than the hardware wires, but only by a little.</p>
And third, TCP sockets.</p>
</a></p>
TCP sockets are multiple abstraction layers removed from the hardware wires.
They can also transmit and receive bytes, but they also have higher-level
functionality, such as addressing that allows connections to be routed
between many different wires.</p>
On their own, each of these three things has a set of functions that it
naturally supports, like these:</p>
</p>
These sets represent something inherent about each of these resources. Files
have a "current position" cursor that can be moved with an "lseek" operation,
but sockets and serial ports don't. TCP sockets have connections and one can
query the IP address of the remote end, but files and serial ports don't.
Files, serial ports, and TCP sockets are three different things, that support
three different sets of operations.</p>
But, the Everything Is A File approach says to treat them all as files. How
does that work?</p>
Read and Write</h2>
The first thing to notice here is that all three of these resources support
read</code> and write</code>. And, read</code> and write</code> are the only operations that show
up in all three of these resources.</p>
One of the big reasons Everything Is A File works as well as it does is that
a lot of use cases only need read</code> or write</code>. As long as you
stick to one of those two operations, the Everything Is A File abstraction works
pretty well. Serial ports and TCP sockets can be thought of as infinite-length
files.</p>
But, things get tricky. We said read</code> or</em> write</code> above, but
what if you read</code> and</em> write</code>? Serial ports and TCP sockets are both
interactive</em>. It's common to write some bytes, and then read the response.
However, files aren't interactive. If you write some bytes and then do a read,
you'll read bytes from the array starting at the byte after the one you just
wrote to.</p>
</p>
On serial ports and TCP sockets, interleaving reads and writes like this is
very common. In files, it takes a lot of creativity to even imagine why one
would ever want to do this.</p>
Unix: Everything Is A File-SerialPort-TCPSocket-etc.</h2>
What happens if you try to call a file operation, like lseek</code>, which seeks
the "current position" of a file to a different location, on a resource like
a serial port or a TCP socket, which doesn't have a concept of a "current position"?</p>
    </span>// Open the serial device "file".
</span>    </span>int</span> fd = </span>open</span>("</span>/dev/ttyUSB0</span>", O_RDWR);
</span>
</span>    </span>// Attempt to seek to byte-offset 8.
</span>    </span>lseek</span>(fd, SEEK_SET, </span>8</span>) </span>// Fails with `ESPIPE`, "Invalid seek".
</span></code></pre>
Unix lets you to attempt this, and the operation fails at runtime. This is
effectively dynamic typing; the file descriptor refers to a resource with a
dynamic type, which in this case is "serial port". The lseek</code> call does a
dynamic type check, and if dynamic type isn't "regular file", then the lseek</code>
call fails.</p>
So Everything Is A File doesn't mean that everything is really</em> a file. There's
still a distinct "serial port" type</em>. Everything Is A File just means that you
can use file APIs at compile time, and the types aren't checked until runtime.</p>
And even though Unix invented the concept, Unix doesn't fully realize
Everything Is A File. When Unix added sockets, it didn't insist on using only
operations in the file API. Instead, it added a bunch of new functions. So in
practice, it has one big combined API with all the functions that it supports
available on all resources:</p>
</p>
All these new functions fail at runtime if you pass them a file descriptor for
something that isn't actually a socket.</p>
Plan 9: Everything Is A ~~File~~Convention at Runtime</h2>
Unix didn't insist on rigidly following Everything Is A File when it came to
sockets, but Plan 9 did. As we saw in
Measuring System Interface Complexity</a>,
Plan 9's way of adding resource-specific functions while staying within the file
API is to dynamically serialize requests as data, which can be written through
the file API.</p>
For example, for sockets, Plan 9 uses ASCII strings to encode commands like
"connect" and "accept" and send them to the network driver, which then parses
the ASCII strings and then calls the appropriate code inside the driver. This
way it can nominally use only the File API, since it's just using write</code>, but
it can still make arbitrary function calls.</p>
</p>
Unix's ioctl</code> is another form of this pattern, where dynamic values determine
the callee. The same values can even have different meanings, depending on what
type of resource they're being used with.</p>
Abstractions that don't abstract</h2>
Even though Everything Is A File is about using the file API in a very abstract
way, it doesn't end up providing much actual abstraction</em>. It doesn't usually
let user code written to work with files automatically work with things that
aren't files, unless it's just using read</code> or just using write</code>.</p>
Whether it's the Unix approach of having lots of functions that fail at runtime
if the dynamic type is wrong, or the Plan 9 approach of serializing lots of
functions into data that can be smuggled through the file API, user code still
has to know what the dynamic type is in order to make meaningful use of it.</p>
In all, Everything Is A File is often given credit for abstracting over widely
diverse resources. However it's really just read</code> and write</code> doing all the
abstracting. The rest of the file API doesn't enable reuse of logic across
different types.</p>
So the real abstraction here is just Lots Of Things Are Streams</em>.</p>
</a></p>
The big picture</h2>
Other than the stream-style abstraction with read</code> and write</code>, a recurring theme in
Everything Is A File is deferring things which might otherwise be compile-time or link-time
concerns into runtime concerns. Type checks happen at runtime. And if we go all-in on
Everything Is A File, even function dispatch happens at runtime.</p>
Doing all these things at runtime has many implications. One is that it opens up the
possibility that core parts of program behavior could be influenced by malicious data.
If function names are encoded as strings like "accept" or "connect", it's harder to
ensure Control Flow Integrity (CFI) because it opens up opportunities for malicious
code to cause other strings to be used instead, redirecting control flow to other
functions.</p>
Similarly, authorization becomes more complex because every single operation an
application does is something which could dynamically represent something it shouldn't
be permitted to do, so the platorm has to be continually performing authorization.</p>
Another implication is that doing things at runtime makes it harder to understand
programs without running them. If you want to know what resources a program will
access, which functions it will use to do the access, the only way to be sure is
to be there as it happens at runtime. This makes it harder to
write interposition or virtualization layers, harder to know what resources an
application will need ahead of time, and harder to figure out what's wrong when
the application's assumptions of its environment aren't met.</p>
So, if Everything Is A File has downsides, why do it?</p>
There's more to say about the hierarchical namespace side of Everything Is A File in
future posts, but here we're just looking at the file-descriptor meaning. And the main
thing this meaning does is:</p>
It works around the problem that adding new functions in Unix is hard.</p>
Hard</h2>
Sometimes adding new functions in Unix is hard because you have to touch a lot
of things. You probably need to touch multiple places in the kernel. You probably
have touch touch the libc header files. You probably have to touch libc itself.
And you probably have to teach source languages that have C FFI layers about the
new function.</p>
And sometimes these things are made harder by organizational reasons. When someone
is writing a driver, and they often can't conveniently make changes to the core
OS kernel or libc. Adding new functions in Unix and libc is often expected to be
done with central coordination.</p>
And sometimes adding new functions in Unix is hard because interfaces are described
using C header files, and C header files don't contain enough information to
auto-generate intermediaries, such as tracing tools, filtering tools, RPC protocols,
bindings in languages other than C, and many other things. When you see a pointer,
you don't know if it's a pointer to a single element or an array, and if it's an
array, what the array length is. And either way you don't know if the memory is used
for just the duration of the call, retained after the call, or freed by the call. C is
accustomed to passing all of these concerns off to its users, but this means that it
takes a lot of manual work when adding new functions to update everything.</p>
And sometimes adding new functions in Unix is hard because new functions come with
the expectation of long-term stability. The C ABI approach means that ABI details
get baked into everything that interfaces with them. If you ever change the signature
of a function, except in very specialized ways, you cause silent corruption for all
users. And if you ever remove a function, you have to all the work you did to add the
function in inverse. So in practice, you don't add new functions, unless you're
really ready to commit to them for the long term.</p>
</a></p>
So the big advantage of this meaning of Everything Is A File is that you can add
new logical functions while avoiding the difficulties of adding new actual
functions.</p>
The 9p protocol</h2>
Plan 9 has an additional consideration: It has a protocol, called 9p, which
encodes a kind of file API. This ends up being another arena where adding new
functions is hard.</p>
A network protocol needs a specification, and 9p has one, but it's kind of an
anti-specification. It defines just enough functionality to work with a hierarchical
namespace and files, and excludes everything else. It deliberately doesn't define the
rest of the protocol, pushing everything else out to dynamic conventions, like
String OS</a>. This is very convenient for a
spec, because specs take a lot of work to maintain.</p>
In a spec, adding new functions is hard.</p>
What if...</h2>
The constraints that make adding new functions in Unix and other domains hard
have been with us for a long time, such that we don't always realize when it's
there, shaping the way the systems around us work, and even shaping the way
we think about new systems.</p>
Everything Is A File bundles up a wide range of assumptions in a way that aren't
always easy to see. If we unpack those assumptions, it helps us ask the question:</p>
What if we designed systems differently?</p>


Mustang update
2023-03-31T00:00:00+00:00
Mustang</a> gained lots of new features recently. It's now complete enough to run
lots of real-world Rust programs, with no libc:</p>

Tide's hello web server example</a> (epoll and async!)</li>
Tokio's tinyhttp web server example</a> (more epoll and async!)</li>
Most of coreutils</a> (lots of stuff!)</li>
Ripgrep</a> (threads!)</li>
</ul>
See here</a> for instructions on how to port programs to mustang. And if
you give it a try, post about your experience here</a>, whether it works or not!</p>
Looking ahead</h2>
Currently mustang requires programs be modified to include
mustang::can_run_this!()</code> in their main source file, which is very inconvenient.
The only way I currently know how to fix that is to teach the Rust toolchain about
mustang. Is that the next step?</p>
On one hand, mustang is cool and fun. But a lot of other things are cool and fun
too. An official Rust toolchain would involve a lot of work from a lot of people
who have a lot of other important things to work on. And for the most obvious use
cases, one of the Musl targets would be a better choice, being more featureful
and stable and so on.</p>
So mustang is kind of waiting for a sign.</p>


Measuring System Interface Complexity
2022-11-29T00:00:00+00:00
This post is in a series about "Everything Is A File":</p>

Is Everything A File?</a></li>
Measuring System Interface Complexity (this post)</li>
What does Everything Is A File do?</a></li>
The Filesystem Namespace</a></li>
</ul>

One of the most easily quantifiable ways to measure the "simplicity" or "elegance" or an operating system this is to count the number of functions in its system interface, or "syscalls". Linux has over 300. That's lot! For comparison, Plan 9 has around 30. It would seem that Plan 9 is, like, 10x</em> more elegant than Linux.</p>
Ok, so listen to this. That's cool. But guess what.</p>
I have a design for an OS which is 30x</strong> more elegant than even Plan 9. It's 300x</em> more elegant than Linux!</p>
Introducing "String OS".</p>
</p>
What is the secret to String OS' amazing simplicity and elegance? I'll tell you.</p>
It has only one syscall!</p>
fn </span>the_syscall</span>(</span>arg</span>: String) -> String;
</span></code></pre>
That's it! That's the entire interface.</p>
The way it works is, you describe requests you want to make in strings, and you call the_syscall</code> and pass them in. You get back a string, which describes the result. That's it.</p>
It's trivial to interface with from any programming language; all you need is a string type and the ability to call functions!</p>
And it's fully serializable, loggable, extensible, interposable, replayable, remotable, and network-transparent.</p>
🤯</big></p>
Other operating systems have "shells" to provide interactive experiences. String OS doesn't hide behind a shell. You can type commands straight into the OS.</p>
</p>
What's that you say? You want to know the list of commands?</p>
...</p>
Seriously though</h2>
String OS is a made-up example, but the real world sometimes isn't that far off. Consider Unix's ioctl</code>:</p>
int </span>ioctl</span>(</span>int </span>fd</span>, </span>unsigned long </span>request</span>, ...);
</span></code></pre>
Here, instead of a string, we're passing an integer, but it's not that different. The way ioctl</code> is implemented, it does a big switch on the request</code> argument, and dispatches to one of hundreds of different functions, depending on the values of its arguments. When OS's do this kind of thing, it fools our little "number of syscalls" metric.</p>
But perhaps there's a way we can fix the metric by looking a little closer. Perhaps instead of looking at the literal number of system calls, we count the number of underlying implementation functions you can reach. We can look through the big switch</code>. Or for String OS, we can look through the parsing and dispatch code. If we did that for Linux, we'd add over a thousand more syscalls. Yeah, there's a lot</em> of ioctl</code> codes.</p>
Is this Plan 9's cue to take the stage? Shall we sound the fanfare? After all, Plan 9 famously doesn't even have</em> an ioctl</code> function. It avoids ioctl</code> entirely</em>. So elegant!</p>
</a>
(this picture illustrates an elegant mathematical proof; have we reached this level of elegance in system interface design yet?)</p>
Or does it?</p>
Now, come to think of it, how...</p>
er, how...</p>
</a></p>
How can Plan 9 have only 30 syscalls? How does it work?</p>
It turns out, Plan 9 has a thing called a "ctl file"</a>. Plan 9 file descriptors can have a special file associated with them named "ctl". The way you use it is, you open that file, and write a specially-formatted ASCII string into it, containing a command and parameters to pass to it. From the link:</p>

Writing the string connect 2048</code> to the ctl file sets the packet type to 2048 and configures the connection to receive all IP packets sent to the machine.</p>
</blockquote>
This means that we're using the existing write</code> system call to send a serialized message, which effectively encodes independent system calls and their parameters. We've avoided an ioctl</code> system call, but we've effectively just re-introduced new logical syscalls using dynamic dispatch within the "ctl" file driver.</p>
So while Plan 9 has a very small number of direct system calls in its system interface, part of how it achieves this is by hiding logical functions behind a string parsing and dynamic dispatch layer.</p>
In short, it works like String OS!</p>
And more</h2>
Even when they aren't literally serializing commands into strings, Unix's and Plan 9's
pervasive use of filesystem namespaces means that the system calls they do have are
very string-oriented. Strings identify resources in the namespaces.</p>
And more broadly, the Everything Is A File design focuses everything on a relatively
small set of file-like operations, where features that don't fit into file-like usage
patterns get accessed through integer or string values and dynamic dispatch.</p>
So what?</h2>
So a few things.</p>
First, counting the number of direct syscalls is a very</em> unreliable way to measure OS design elegance.</p>
Second, is String OS a good way to design system interfaces? I'll write more about this in a future post. As a sneak preview, using strings to identify resources is a sign that there may be ghosts present</a>, but there's more to say 😃.</p>


What is a Capability?
2022-11-21T00:00:00+00:00
This blog post aims to provide a simple answer to the question: What is a
Capability? I answer this question from my own perspective, as someone who
didn't previously know anything about component models.</p>
This post is forward-looking; not all of the pieces described here are usable yet. It's a look at what's coming.</p>
Preliminaries</h2>
When I say "capabilities", I'm talking about capabilities in the context of the
Wasm component model</a>. These are capabilities in the sense of
capability-based security</em></a>.</p>
There is a relationship with the term "object capabilities", or "ocap", however
that terminology involves some nuance and I'm aiming for a simple intuitive
description here.</p>
I'm not talking about wasmCloud "capabilities"</a>
or Linux "capabilities"</a>.
Similarly, there are connections that could be drawn here, but I'm aiming to keep
things simple.</p>
Ok, Let's go!</h2>
One of the great things about Wasm is that it can't do anything.</p>
</p>
Wasm has no syscall instructions. It has no I/O ports. All it can do is compute.
And it can import functions from the outside and call them, and export functions
to the outside and have them be called.</p>
Link-time capabilities</h2>
So the way to let a Wasm component do</em> something outside of itself is to give
it functions it can import that do things, or conversely, to call its exports
from functions that do things when they return.</p>
</p>
And that's great, because it means the Wasm program can do what the things
we provided it can do, and nothing else.</p>
Imports and exports are supplied at link time, so we call these
link-time capabilities</em>.</p>
Capabilities with too much authority</h2>
Link-time capabilities are useful, but often they're too</em> useful. As a
simple example, the following interface describes a link-time capability:</p>
/// Execute the command described in the provided
</span>/// `command` string.
</span>do-command: func(command: string) -> string
</span></code></pre>
Imagine this is a function that passes the string argument to some command
interpreter, that can execute arbitrary commands. This can potentially be
a very</em> powerful API, capable of doing almost anything.</p>
Imagine I have a large codebase, and I want to pass around a reference
to some variable within in the command interpreter. The only way to do it
with this do-command</code> API is to pass around the string name.</p>
// Set `x` to 2 within the command interpreter.
</span>do_command</span>("</span>x = 2</span>");
</span>
</span>// Set the variable named in `some_dynamic_string` to 2.
</span>do_command</span>(format!("</span>{}</span> = 2</span>", some_dynamic_string));
</span></code></pre>
With this, it can be very difficult to answer questions like "what are all
the places in the code that use this variable?" or, looking at a particular
callsite, "what are all the variables this callsite could access?". Strings
can originate from many places at runtime, potentially even untrusted places
like attacker-controlled inputs, so it's very difficult to make any kind of
comprehensive guarantees with an API like this.</p>
In systems where command strings are computed from other strings, it's
also notoriously difficult to prevent quoting and whitespace bugs, leading
to bugs like SQL Injection</a>.</p>
And when we look at modular systems, when a string is passed from one
component to another, it often means that there's an assumption that the
two sides are implicitly sharing a namespace,
as if a ghost were connecting the two</a>.</p>
</p></a></p>
Furthermore, APIs like this do-command</code> are awkward to virtualize, for sandboxing,
logging, testing, or other purposes. They require intermediaries to always
parse the command string, even when they're only interested in specific kinds
of requests.</p>
Ultimately, APIs like this combine too many authorities into a single capability.
Capabilities like this are called coarse-grained</em>, since an individual capability
exposes access to many distinct logical resources.</p>
Traditional Filesystem and IP networking APIs tend to be coarse-grained, with
functions like open</code> or connect</code>:</p>
open: func(path: string) -> file
</span>
</span>connect: func(hostname: string) -> socket
</span></code></pre>
Filesystems and network namespaces both often contain many different logical
resources, often with differing security considerations. The code that calls
open</code> or connect</code> can open any file or host in the namespace, identified only
by some runtime string value. And sandboxing such APIs can be tricky,
because resources can often have multiple names, with links or aliases, which
can appear anywhere in the namespace.</p>
</p>
One of the great properties of link-time capabilities is that one can find
all the places that use them at link time. But this advantage is negated if the
capabilities they represent are coarse-grained and you can't tell which specific
resources are being accessed until runtime anyway.</p>
Runtime capabilities</h2>
To avoid these problems, we need runtime capabilities</em>. The Wasm component
model provides this with handles</em>.</p>
</p>
Handles are references to resources</em>, and are values which can be passed
around as arguments and return values between components. And unlike other
values, handles are unforgeable</em>. The only way to construct a handle to a
resource is to be the implementor of the resource. The only components that
can obtain a handle are components it's been explicitly passed to.</p>
Handles are similar to file descriptors in Unix. They're references to
external things, where the access</em> to the thing is represented by the
reference itself. And like file descriptors, they may be represented in
source languages as i32</code> values or similar, where those i32</code> values are
indices into a table of the actual unforgeable handle values.</p>
Revisiting the filesystem and networking APIs above, we can make them
more fine-grained by adding handles arguments, such as a directory in which
to open files, or an address pool with which to initiate a network connection:</p>
open-at: func(dir: directory, path: string) -> file
</span>
</span>connect-in: func(pool: address-pool, hostname: string) -> socket
</span></code></pre>
This way, instead of having an API which can reference any resource in an
implicitly shared namespace, conveyed as if by ghost</a>, we have an API where
the namespace to use is explicitly communicated as an explicit argument.
This gives the caller, or an intermediary, the opportunity to be selective
about what resources are passed to the callee, without needing to configure
an external sandbox.</p>
Wrap up</h2>
To do anything, Wasm components need capabilities. These can be provided at
link time, via imports and exports, which is sometimes useful, but can
easily be too coarse-grained. The Wasm component model also includes
handles</em>, which identify capabilities at runtime.</p>
Interfaces that expose access to logically distinct resources should represent
them as distinct capabilities.</p>


Is Everything A File?
2022-11-17T00:00:00+00:00
This post is the first in a series about "Everything Is A File":</p>

Is Everything A File? (this post)</li>
Measuring System Interface Complexity</a></li>
What does Everything Is A File do?</a></li>
The Filesystem Namespace</a></li>
</ul>

"Everything Is A File" is one of the most fundamental and widely influential
pillars of Unix design philosophy.</p>
It's one of the major sources of simplicity</em> that propelled Unix beyond its
predecessor Multics, and helped it achieve widespread popularity. It enabled a
small set of simple tools to operate on data from a wide variety of sources.</p>
At the same time, it's also become a major source of complexity</em> in modern
computing. And it's been so pervasive for so many decades at this point, that
we often don't recognize it as such.</p>
"Everything Is A File" itself has at least three different meanings:</p>

Everything has a name in a hierarchical namespace</li>
Everything is a file descriptor</em></li>
Everything is byte sequences</li>
</ul>
Strictly speaking, Unix itself doesn't perfectly conform to any of these, but
it does follow them enough in enough places that they're recognizable and
influential. And then there's Plan 9 which took some of these ideas further.</p>
So there's a lot to explore here. This is the first of a series of blog posts
exploring different aspects of Everything Is A File, with an overall focus on
system interface design</em>.</p>
To get things started, here's a quick exploration of one small but illustrative
quirk of Unix.</p>
A tale of two file descriptors</h2>
Unix's dup</code></a> function returns a new file descriptor referring to the same
resource as an existing file descriptor. The new file descriptor is independent
if the old one; we can close</code></a> either one and the other remains open.</p>
When file descriptors refer to files, they keep track of a "current position"
that tells read</code></a> and write</code></a> where in the file to read and write. Since the
file descriptors are independent of each other, one might think that each file
descriptor would have its own file position, like this:</p>
</p>
But what actually happens in Unix is that the file position is shared between
the two file descriptors, like this:</p>
</p>
If we do an lseek</code></a> to change the position of one, it changes the position
on the other one at the same time. Users of each have to be aware that the way
they use one file descriptor might affect the other. And if they have this level
of coordination, why did they need to call dup</code> in the first place?</p>
Everything Is A File</h2>
Pipes aren't files, but Everything Is A File wants to treat everything in a
uniform way, and it would be expensive to make pipes work like files. Files
are random-access, and pipes aren't.</p>
It's not that expensive to make files work like pipes though. All it takes
is adding this "file position" to open file descriptors. In this way, Unix's
read</code> can read from a file just as it can read from a pipe.</p>
But then, we might ask, how should dup</code> work? If we do dup</code> in the natural
way for files, each file descriptor would have its own current position. But
the equivalent of that for pipes would be expensive to implement; it'd require
storing a copy of all data sent through the pipe. So Unix instead says that
dup</code>'d file descriptors act like they do on pipes, which means they share a
current position.</p>
It makes sense from that perspective. But if we go back and look at it from
the perspective of lseek</code>, it doesn't make sense again. lseek</code> doesn't even
work on pipes, so why are pipes the thing that determine how lseek</code> has
to work?</p>
So, what if...</h2>
Underneath this is an "Is A"</a> relationship. Everything Is A</em> file, and an
open file Is A</em> stream.</p>
What if we split out a streaming view</em> as a separate entity?</p>
An open file would continue to support all the essential file operations, like
fsync</code>, pread</code>, pwrite</code>, and so on, but not read</code> or write</code>. A streaming
view would support read</code> and/or write</code>.</p>
We'd then add a new function, which takes an open file and an offset, and
returns a streaming view of the file at that offset.</p>
</p>
If we had a system like that, the lseek</code> function wouldn't be needed, and we'd
avoid this whole question of how lseek</code> and dup</code> interact in surprising ways.</p>
There's more we could say here about ergonomics and efficiency, but for now,
this gives a glimpse of a shape for a simpler and more orthogonal system, where
files just do one thing and do it well: hold arrays of data, and streams do one
thing and do it well: stream data.</p>
Wrap up</h2>
We're just getting started here.</p>
Everything Is A File conveys some underlying truths. Being able to write programs
that can automatically read from files, pipes, sockets, and more is really powerful.
But Everything Is A File also distracts us by pointing us toward the "file"
concept as the vehicle for achieving this.</p>
There's lot's more to explore. Follow along for future explorations!</p>


Fork versus Elegance
2022-11-10T00:00:00+00:00
The research paper A Fork in The Road</a> contains a good summary of the problems
with the fork</code> call in Unix.</p>
As the paper points out, posix_spawn</code></a> provides an alternative to fork</code> which
solves some of the performance problems. However, if we set aside the performance
side for the moment, posix_spawn</code>'s API kind of gives an impression that, if
that's the alternative, perhaps we should reconsider.</p>
posix_spawn</code> needs a whole flock of posix_spawn_*</code></a> helper functions, which
provide ways to configure various aspects of the child process:</p>
</p>
This is inelegant,
because POSIX already has functions for doing all these things, except that they
only work on the parent process. So posix_spawn</code> has alternate versions of all
these things, that operate on the child instead of the parent. And even with all
the features made available this way, it still doesn't turn out to cover
everything that people want to do.</p>
In all, posix_spawn</code> just doesn't feel "Unixy".</p>
In contrast, fork</code> allows the child to be configured with the same APIs as the
parent. It can do everything</em>, with no API duplication. Does this mean that
maybe fork</code> is an elegant way to design systems after all?</p>
Not necessarily.</p>
A different perspective</h2>
One reason why posix_spawn</code> is simultaneously complex and insufficient is
that Unix has a lot of resources implicitly associated with processes, that
then need to be configured.</p>
One such resource is the current directory. It acts a lot like a file descriptor
for a directory, except that the OS implicitly holds onto to it on behalf of
the process.</p>
There isn't any fundamental reason why userspace couldn't hold onto this file
descriptor itself. If a current directory handle were passed into a process
alongside stdin, stdout, and stderr, userspace could use it and manage it
explicitly, and there'd be no need for a dedicated chdir</code> system call.
And in a posix_spawn</code> situation, the parent could pass any file descriptor to
be the child's current directory, using the same mechanism as passing other
file descriptors.</p>
So, what if we had an OS that took this even further? What if we moved even
more state out of the process, and into resources that would be explicitly
managed via file descriptors?</p>
That would let parent processes easily configure resources for their children
processes without needing special APIs. All they'd have to do is use the
regular system calls to set up file descriptors they want their child processes
to have.</p>
But wait, there's more</h2>
If we managed Linux namespaces with file descriptors, we could avoid having a
bunch of flags on clone</code></a> to say what parts of the parent get copied to the
child. And we could give userspace more control over how it wants to share
things.</p>
To be sure, at this point we're talking about a fair number of file descriptors.
A design like this would likely also want better ways to manage and pass
around file descriptors. As the paper above points out, the fork</code>+exec</code> way
of having children inherit file descriptors from the parent (with O_CLOEXEC</code> as
an awkward opt-out) already isn't great. Perhaps what we'd want is a way to
pass file descriptors to the child explicitly, rather than relying on children
implicitly inheriting them. At this point though, I'll leave that for future
possible blog posts.</p>
What does it all mean?</h2>
fork</code>'s apparent simplicity is built on an underlying complex system, which
attaches a lot of implicit resources to processes.</p>
A new OS that doesn't attach as many implicit resources to processes wouldn't 
necessarily need either fork</code> or posix_spawn</code>'s flock of helpers as
primitives.</p>


What is a World?
2022-11-03T00:00:00+00:00
Worlds</em> have emerged as an important tool for WASI.</p>
Luke Wagner's The Path to Components</a> talk gives a great introduction to
Worlds, with the story to how we got here, and the path ahead.</p>
WebAssembly/wasi-kv-store#2</a> is a PR to the wasi-kv-store showing a concrete
example of Worlds, and WebAssembly/component-model#83</a> is a PR to the
component-model spec with a lot of detailed information.</p>
This blog post aims to provide a simple answer to the question: What is a World?
I answer this question from my own perspective, as someone who didn't
previously know anything about component models.</p>
This post is forward-looking; not all of the pieces described here are usable yet. It's a look at what's coming.</p>
What is a World?</h2>
A World is a virtual</em> environment</em> for a component</em>.</p>
This packs a lot of meaning into a few words, so let's unpack!</p>
For a Component</h3>
A Wasm component</em> is a deployable</em> unit</em> of software</em></a>.</p>
An Environment</h3>
An environment</em> here is the surroundings that a Wasm component can run in. It
contains imports</em> and exports</em> that a component's exports and imports can
link to. Wasm components are isolated, so this is the only way they can
interact with the outside.</p>
A World is a description of what an environment contains. A component can be
built to run in a particular World, and then any environment that implements
that World can run it.</p>
That's Virtual</h3>
Virtual</em> here means that Worlds aren't tied to concrete implementations.</p>
A database API might be implemented using a remote database in a production
environment, and using a local-file database in a testing environment. A logging
API might write log messages to a local file, a remote log service a console,
or a chat channel.</p>
The implementation of a World may be provided by native host code, by other
Wasm components, or a mix of both.</p>
The strong isolation properties of Wasm components means that components can't
simply "look around" and see what's happening on the system around them. For
example, if a logging API doesn't say where the log output is going, and the
World doesn't include filesystem access, the component can't look around and
see whether the output is going to a file. The output could go somewhere else
entirely, and the component will still work.</p>
Virtualization enables testing with mocked dependencies, adapting components
written for one environment to run in another, polyfilling older versions of
APIs for backwards compatibility, and more.</p>
Wrap up</h2>
A World is a virtual</em> environment</em> for a component</em>.</p>
The WASI Subgroup is expected to define some standard Worlds, which environments
may chose to implement, allowing developers to produce components that run in
many different environments. Hosts may also chose to define their own bespoke
Worlds, either by extending or combining standard Worlds, or by describing new
Worlds from scratch.</p>
There's a lot more that can be said about Wasm components and Worlds; check out
the links in the intro, and follow me for future explorations!</p>


What is a Wasm component?
2022-08-10T00:00:00+00:00
When I first heard people in the Wasm world talking about these things called
"components" for Wasm, I had no idea what they were talking about.</p>
Other people have now written about what a Wasm component is, such as
this blog post from Fermyon</a> which makes an analogy relating Wasm concepts to
native code and OS concepts, and this podcast with Lin Clark</a> which makes an
analogy between Wasm components and Lego blocks. And the Wasm component model
proposal repo</a> has documentation about its goals</a>, intended use cases</a>, and
design choices</a>, and an FAQ</a>. These are all great resources, and I recommend
them.</p>
This blog post aims to provide a simple answer to the question: What is a Wasm component?
I answer this question from my own perspective, as someone who didn't
previously know anything about component models.</p>
This post is forward-looking; not all of the pieces described here are usable
yet. It's a look at what's coming.</p>
What is a Wasm component?</h2>
A Wasm component is a deployable</em> unit</em> of software</em>.</p>
This packs a lot of meaning into a few words, so let's unpack!</p>
A Unit</h3>
A Wasm component is a self-contained unit. It doesn't need wrappers, glue,
header files, or build artifacts, to be usable. It may have dependencies,
but its dependencies are on declared and self-describing APIs, rather than
on specific artifacts.</p>
In comparison, Wasm modules</em> often have implicit contracts with the outside
world. When a Wasm module passes around an i32</code> value, it might be a number,
which might be signed or unsigned, or it might be a set of flags, a function
pointer, or a data pointer. If it's a data pointer, the pointer is expected
to be used with a particular linear memory address space, where it points to
a region of memory with a layout, a lifetime, and other conventions.</p>
This means that Wasm modules have the flexibility to represent many different
kinds of source languages. But it also means that many source-language
assumptions aren't represented in a Wasm module's type system. Anyone using
a module needs to share those assumptions. This means a Wasm module
conceptually acts as a slice of a larger whole.</p>
A Wasm component</em> is a whole. It may still be used as a dependency within
a larger conceptual program, but it doesn't depend on being used that way.
Components just have APIs which can be used wherever they're relevant.</p>
Deployable</h3>
Wasm's high level of deterministic behavior and isolation provides very
consistent behavior, even across diverse environments.</p>
Wasm components build on that. They add features supporting capability-based
security, which allows them to share I/O facilities with other components
without sharing namespaces or user identity. And they don't use a
globally-shared linear memory, which protects against cross-component
memory-safety hazards.</p>
Wasm components also add explicit linking features. Linking works the same
way whether it's performed at development time, or just before the program is
run. The output of linking components is a new component. This new component
can then be optimized, including with inlining, dead code elimination, and
other optimizations that can then cross the original component boundaries.</p>
This enables the ability to take a collection of Wasm components and either
run them as a collection, or link them into a single optimized easy-to-deploy
component.</p>
Software</h3>
Wasm components are "software", rather than "Python software",
"Haskell software", or any other specific source language's software. The
source language a Wasm component is implemented in is not exposed in the API
of a component.</p>
That's not to say that Wasm components have to always be implemented in
software. Components encapsulate their implementation, so component APIs
can also be implemented by host environments, including using special
hardware. But from the outside, the API behaves the same as if it were
software.</p>
Wrap up</h2>
A Wasm component is a deployable</em> unit</em> of software</em>!</p>
There's a lot more that can be said about components; check out the links
in the intro, and follow me for future explorations!</p>


No Ghosts!
2022-03-15T00:00:00+00:00
This post proposes and explores a design principle for components in complex
software systems:</p>
No Ghosts!</em></p>
The ideas in this post aren't new; they come from papers and blog posts such as
Robust Composition</a>, Capabilities: Effects for Free</a>, Parse, don't validate</a>,
the nanoprocess model</a>, and the design choices in the Wasm component model</a>, which
itself incorporates ideas from Erlang, OCaml, Rust, COM, and many others. This post
is an attempt to articulate what I see as one of the themes that runs through all of
these. It's not any kind of official position, and it certainly won't be the last
word on any of these topics.</p>
I'd also be very interested in feedback on what makes sense here, what doesn't, and
what's missing: mastodon</a>,
zulip</a>,
email</a>.</p>
This is motivated by WASI and Wasm components, however the core ideas generalize
to any complex software system, including those using libraries, daemons,
containers, VMs, microservices, or a combination.</p>

One of the main ways we can make complex systems manageable is to make them
modular</em>. This means being able to add, remove, change, or understand individual
components (or whatever a system is composed of) without needing to consider the
system around them.</p>
It's common for systems with large numbers of components to have problems with
unexpected interactions between components. A common response to these kinds of
problems is to impose a level of modularity by introducing heavyweight barriers</em>,
such as sandboxes, process boundaries, firewalls, or service meshes. These
comprehensively block many of the avenues for components to interact. Then,
since components still do need to communicate, it's common to effectively poke
small holes in the barriers, such as allowing specific HTTP connections to pass
through.</p>
However, while the barriers-and-pinholes approach fixes the immediate problems,
barriers are sometimes too</em> comprehensive. They get in the way of connecting
things that we do</em> want to connect. This then leads us to do more things
through the pinholes, or poke more pinholes, which then increases the risks of
unexpected interactions again.</p>
</a></p>
There are many reasons why we get stuck on this path, but one is that
heavyweight barriers tend to focus us on limiting the mechanisms</em> that let
components interact, such as which components a component can directly talk to,
and what kinds of messages it can send. However, the more fundamental problems
are often in the relationships</em> between components. One of the things we can
do to avoid these problems, and promote modularity in a sustainable way, is to
design component APIs that avoid ghosts</em>.</p>
👻? 😱!</p>
Ghosts?</h2>
By “ghost” here, I mean any situation where resources are referenced by
plain data</em>.</p>
And by “plain data” here, I mean strings, integers, or any other data where
independently produced copies of the data are interchangeable. For example, two
completely independent parts of a system may create a string with the value
"Purple"</code>, and the two strings will be interchangeable.</p>
Plain data can contain filenames, network addresses, usernames, or other forms
of data which effectively reference</em> resources.</p>
For example, when we say that a particular string contains a filesystem path, we
mean that it refers to</em> an entity in a filesystem namespace. Filesystem namespaces
are not explicitly passed as arguments in the APIs of many popular systems, so
from the perspective of an API, while paths are explicit string parameters, the
additional information referenced</em> by those paths is not. In this post, we'll
say this additional information is being carried by a “ghost”:</p>
    </span>// We're explicitly passing a path, but implicitly
</span>    </span>// passing the namespace to resolve it in.
</span>    </span>do_stuff</span>("</span>/tmp/data.txt</span>");
</span></code></pre>
By passing a filename, the caller here is requiring that the callee have a
specific filesystem namespace, in order to interpret that filename. This is
an example of a relationship</em> between components that's difficult to control
with heavyweight barriers focused on mechanisms</em>. The actual message is
just a string, which could be communicated through practically any pinhole.
And once the caller can send filenames through, it can depend on the callee
having a particular namespace and being able to resolve those filenames, and
we have the potential to get complex relationships between caller and callee,
despite whatever barriers we put between them.</p>
As another example, suppose one part of a system sets an environment variable,
and another part of the system reads it.</p>
⬅ In one place:</p>
export </span>TIMEOUT</span>=</span>30
</span></code></pre>
➡ In another:</p>
char </span>*timeout = </span>getenv</span>("</span>TIMEOUT</span>");
</span></code></pre>
Here, two independent parts of the system both use the string "TIMEOUT"</code> as
an identifier to send a message between them. As far as these specific parts of
the code know, it's as if the content of the message is carried by a ghost,
from one part to the other.</p>
IP addresses are another example of plain data that references other
resources. If one part of a system listens on a socket and sends the
IP address to other parts of the system for them to connect to, the address
is a plain-data list of integers, while the interpretation of those integers
depends on a particular network view.</p>
</p>
Ghosts can also occur within key-value stores, registries, brokers, buses, and
many other things where the identifiers are plain data. There are situations
where plain-data identifiers are the only option, such as when working with
external resources. But when designing component APIs, we should seek to avoid
ghosts where we can, and seek to identify and encapsulate ghosts where we can't.</p>
Granted, the way all these things work isn't literally</em> supernatural. We
can figure out how filesystem namespaces, environment-variable dictionaries,
networks, and other things make our resources available if we know some
things about the surrounding system. However, that goes against our goal of
modularity. We specifically don't want individual components knowing about
the system around them.</p>
The trouble with ghosts</h3>
Ghosts are often convenient</em>, in the way that duct tape is convenient. They
can quickly connect two things, even in a large system, without extensive
changes. And on small scales, they sometimes work well. But like duct tape,
they aren't a material one wants to build complex structures from.</p>
</a></p>
Ghosts have four distinct problems as systems scale up in complexity:</p>

Ghosts don't always go to the places we want them to 👻➡😞. When we pass
plain-data references around, they depend on the ghosts going to the same
places. If our references go somewhere that the ghosts don't go, attempting
to resolve them may fail, or may resolve to something unintended. An example
of this is CWE-706</a> “Use of Incorrectly-Resolved Name or Reference”.</li>
</ul>

Ghosts may go places we don't want them to 👻➡😲. For example, environment
variable values are propagated to all child processes, even those that
don't need them, and some programs log the contents of their environment for
diagnostic purposes. If our variables contain sensitive information, it may
get exposed. Similarly, ghosts may also persist for longer than we want them
to, because cleaning them up can lead to dangling or even aliasing
references. Examples of this include CWE-532</a> “Insertion of Sensitive
Information into Log File” and CWE-386</a> “Symbolic Name not Mapping to
Correct Object”.</li>
</ul>

Ghosts may collide with other ghosts 👻➡💥⬅👻. In a complex system,
the same name can end up getting used in multiple places. Naming
conventions can help, but aren't enough if there are multiple instances of
the same component within the larger system. In the case of filesystem
namespaces, sometimes two different parts of a system need different versions
of a resource, but they both expect it to be at the same path. An example of
this is CWE-435</a> “Improper Interaction Between Multiple Correctly-Behaving
Entities”.</li>
</ul>

And sometimes, ghosts come from</em> places they're not expected to ❓➡👻.
When plain data can reference resources, any plain data within a system
could potentially be representing a reference. Plain data may also be
influenced by attackers. Examples of this are CWE-22</a>
“Improper Limitation of a Pathname to a Restricted Directory
('Path Traversal')” and CWE-73</a> “External Control of File Name or Path”.</li>
</ul>
Systems which use ghost patterns often face several challenges:</p>
Ghosts complicate static analysis</h4>
Being unable to know where ghosts are going and where they're coming from makes
it difficult and often impossible to answer questions such as:</p>


“I have sensitive data flowing through part of the system. Where are
all the places that might be able to access it?”</p>
</li>

“If I change the behavior of something, what are all the things I
need to update?”</p>
</li>

“If there's a bug in something, what parts of the system could be affected?”</p>
</li>

“If an attacker can control certain input data, what are all the things
which they might be able to influence?”</p>
</li>
</ul>
Ghosts complicate debugging</h4>
A common way to debug complex systems is to isolate parts of the system and
study how they behave independently. Ghosts create situations where components
work differently when run independently than when they're run together, or
work differently in different environments, making this kind of debugging more
difficult.</p>
Ghosts create hidden cause-and-effect relationships, making it harder to
understand the system's behavior.</p>
Ghosts are often a sign of over-sharing</h4>
Over-sharing happens when a component is given access to resources that it
doesn't need. This often happens in namespace-oriented systems because it's
difficult to precisely configure namespaces to be fine-grained and share only
what's needed to each component. Even with features such as bind mounts on Linux,
it can be tricky to make sure that every part of a complex system has access to
all the things it needs, at the paths it expects them to be at, and nothing it
doesn't need. As a result, programs are often run with more filesystem access
than they strictly need.</p>
This makes it difficult to follow the Principle of Least Authority</a> (PoLA).</p>
Ghosts can contribute to confused deputies</h4>
A common pattern in complex systems composed of multiple privilege levels
is that some components are considered to run on behalf of specific users,
which determine their privilege level. We can call components that work this
way deputies</em> of the users that own them.</p>
When components send plain-data requests to components running as different
users, senders may be able to reference resources they shouldn't be able to
access. In such situations, receivers perform access control</em>, explicitly
checking requests to see whether the sender has the appropriate privileges.
This is often tricky, especially when an API has a complex surface area.
Receivers may get confused</em> into doing things they shouldn't allow senders
to ask them to do.</p>
This is a form of the confused deputy problem</a>.</p>
A different kind of relationship</h3>
Systems which have a concept of handles</em>—values which can be passed between
components, but which are not plain data</em>—can use them to avoid ghosts.
Handles provide a way to make specific resources accessible across a component
boundary without requiring any other relationship.</p>
</a></p>
Handles make cause-and-effect relationships clear, since they are explicitly
passed between components. And, receivers can assume that any handle they are
passed represents a resource that the sender is allowed to ask them to
operate on. That way, receivers need less authority of their own, which
reduces the risk of them accidentally misusing their authority.</p>
Ghosts can hide inside explicit sharing</h2>
One of the tricky things about ghosts is that they're about relationships</em>
rather than specific mechanisms</em>. Mechanisms tend to be easy to understand,
and to sandbox. But, relationships that permit ghosts can pass through even
the most restrictive sandboxes.</p>
This blog post talks a lot about implicitly shared resources, however that's
not the only place ghosts can hide. For example, consider our example above of
caller and callee implicitly sharing a filesystem namespace, and passing
strings representing paths:</p>
    </span>do_stuff</span>("</span>/tmp/data.txt</span>");
</span></code></pre>
This is a ghost pattern, with a string carrying a reference to an implicitly
shared namespace. A simple way we might try to eliminate such a ghost is to
replace the use of an implicit namespace with an explicit filesystem root
parameter:</p>
    </span>do_stuff_in_root</span>(root_handle, "</span>tmp/file.txt</span>");
</span></code></pre>
This might be tempting, as it means that most of our code doesn't need to
fundamentally change. It's a mostly mechanical change to just add root
parameters in places where they're needed, and everything else about our
code can stay the same.</p>
We might then be tempted to claim that we've eliminated our ghosts here, because
we now do the sharing via explicit communication rather than an implicitly
shared resource. And we might indeed find that this code does afford us some
added flexibility.</p>
The problem is that this doesn't change the relationship</em>. We're still using
strings to identify specific resources within the filesystem root we're
passing around. And that means we still have resources being referenced by
plain data.</p>
There's effectively a ghost, hiding inside</em> the resource.</p>
</a></p>
One way to think about it is in terms of granularity. While passing around
handles to “root”, “world”, “namespace” or “registry” resources is better
than implicit sharing, those kinds of resources tend to be coarse-grained.
They can end up having ghosts hiding inside them. Plain-data references to
specific items within coarse-grained resources can still have dynamic
cause-and-effect relationships, and can still dangle, collide, or be
influenced by attackers.</p>
To avoid ghosts, it's not enough to change the mechanisms. To change the
relationships, we need to switch from coarse-grained sharing to fine-grained
sharing with handles. Instead of whole filesystems, we should ideally
reference specific directories or even individual files, such as like this:</p>
    </span>// Open the file using our own privileges.
</span>    file_handle = </span>open</span>(root_handle, "</span>tmp/file.txt</span>");
</span>
</span>    </span>// Instead of passing root_handle and a path, pass
</span>    </span>// *just* the one file handle to the other component.
</span>    </span>process_open_file</span>(file_handle);
</span></code></pre>
How to smell a ghost</h2>
There are some common signs that a ghost may be present.</p>


String parameters which don't represent user data</em>. String types in
programming languages can hold many different kinds of things, such
as names or text fields. And when a program is talking to the outside
world, strings may also contain external identifiers such as filenames,
network addresses, or URLs. But when software is talking to
other software, resources should ideally be identified by handles,
rather than by string identifiers. And as a bonus, this also helps
minimize exposure to Unicode subtleties</a> and quoting subtleties</a>.</p>
Heuristic: “Strings are for humans” 🌟</p>
</li>

The word “the”</em>. Whenever we find ourselves thinking about the</em>
filesystem, the</em> network, the</em> process, the</em> host, the</em> OS,
or the</em> computer, it often means we're making assumptions about state
that might be shared between parts of a larger system. Wherever possible,
components should not be aware of “the host”, or any entities associated
with it, as nouns.</p>
Heuristic: “Components should be hostless” 🌟</p>
</li>

User identity outside the user interface</em>. While there's a place for
user-facing software to maintain an explicit knowledge of who they're
acting on behalf of, components interfacing with other components should eagerly
resolve that user authority to obtain finer-grained handles which can then
be passed to other components. That way, those other components don't need
the full access of the user, and will be less likely to make assumptions about
shared state associated with the user.</p>
Heuristic: “Handles are permissions” 🌟</p>
</li>
</ul>
Wrapping it up</h2>
An important property for complex software systems is that they be modular,
where parts can be isolated from the whole. Ghosts, or resources referenced by
plain data, create implicit relationships which must be considered when we add,
remove, change, or understand individual components. They impede modularity,
making complex systems less manageable.</p>
This leads to a design principle for components in complex software systems:</p>
No Ghosts!</em></p>
👻? 🚫!</p>


Bugs in Hello World
2022-03-08T00:00:00+00:00
Hello World might be the most frequently written computer program. For decades,
it's been the first program many people write, when getting started in a new
programming language.</p>
Surely, this humble starting-point program should be bug free, right?</p>
</a></p>
After all, hello world programs only do one thing. How could there be a bug?</p>
Hello world in C</h2>
There are a lot of different ways to write hello world in C. There's
the Wikipedia version</a>, the hello world in the K&R book</a>, and there's
even the oldest known C hello world program from 1974</a>.</p>
</a></p>
Here's another, this one in "ANSI C"</a>:</p>
/* Hello World in C, Ansi-style */
</span>
</span>#include </span><</span>stdio.h</span>>
</span>#include </span><</span>stdlib.h</span>>
</span>
</span>int </span>main</span>(</span>void</span>)
</span>{
</span>  </span>puts</span>("</span>Hello World!</span>");
</span>  </span>return</span> EXIT_SUCCESS;
</span>}
</span></code></pre>
This is the most careful version of the bunch. It uses (void)</code> to ensure that
main</code> is a new-style declaration. It uses the EXIT_SUCCESS</code> macro instead
of just assuming that the platform uses 0 to indicate success, which isn't
necessary, according to the C standard, but we're not taking any chances here.
And it uses the appropriate headers to avoid implicitly declaring puts</code>. This
version attempts to do everything</em> right.</p>
And yet, it still has a bug.</p>
All the versions linked above have a bug.</p>
A bug?</h2>
Linux has this fun device file called "/dev/full", which is like its more
famous cousin "/dev/null", but when you write to "/dev/full", instead of
throwing away the data, it fails. It acts like a file on a filesystem that
has just run out of space:</p>
$</span> echo "</span>Hello World!</span>" > /dev/full
</span>bash:</span> echo: write error: No space left on device
</span>$</span> echo $</span>?
</span>1
</span></code></pre>
This is a great little tool for testing that programs handle I/O errors
correctly. It's inconvenient to create actual filesystems with no
space left, or disks that actually fail, but it's really easy to ask a
program to write its output to "/dev/full" and see what happens.</p>
So let's test the C example above:</p>
$</span> gcc hello.c</span> -o</span> hello
</span>$</span> ./hello > /dev/full
</span>$</span> echo $</span>?
</span>0
</span></code></pre>
Unlike when we used echo</code> in the shell above, here, we got no output, and
the exit status was zero. That means the hello</code> program reported successful
execution. However, it didn't actually succeed. We can confirm that it
encounters a failure using strace</a>:</p>
$</span> strace</span> -etrace</span>=write ./hello > /dev/full
</span>write</span>(1, "</span>Hello World!\n</span>", 13)          = </span>-1</span> ENOSPC (No space left on device)
</span>+++</span> exited with 0 +++
</span></code></pre>
There's our "No space" error getting reported by the OS, but no matter,
the program silently swallows it and returns 0, the code for success. That's
a bug!</p>
How severe is this bug? Arguably, hello world isn't going to be safety-critical
anywhere. However, hello world does do something that programs in the real
world do: print to standard output, which might be redirected to a file. And
files in the real world can run out of space. If a program doesn't detect this
kind of error and report it through its return code, its parent process won't
know that the child failed, and will continue running as if nothing was wrong,
even though the output it expected to have been produced has silently lost data.</p>
For example, consider a program that prints a yaml</a> file to standard output.
If standard output runs out of space, the output may be truncated at some
arbitrary point, though it may still be valid yaml</a>. So we should expect
programs to detect and report this kind of situation.</p>
What about other languages?</h2>
We looked at bash and C above; what about Python, which tells us that
"Errors should never pass silently"</a>? Here's Python 2:</p>
$</span> python2 hello.py > /dev/full
</span>close</span> failed in file object destructor:
</span>sys.excepthook</span> is missing
</span>lost</span> sys.stderr
</span>$</span> echo $</span>?
</span>0
</span></code></pre>
It did print a message to stderr, though it's a confusing message. However, it
also returned 0, which means it's telling whoever ran it that it exited
succeesfully.</p>
Fortunately, Python 3 properly reports the error, and prints a nicer
error message too:</p>
$</span> python3 hello.py > /dev/full
</span>Exception</span> ignored in: <_io.TextIOWrapper name='</span><stdout></span>' mode='</span>w</span>' encoding='</span>utf-8</span>'>
</span>OSError: </span>[</span>Errno 28</span>]</span> No space left on device
</span>$</span> echo $</span>?
</span>120
</span></code></pre>
Using hello world programs from common tutorial sites in a few languages that
I happened to try, here are the results:</p>
Language</th> Has the bug</th> Versions tested</th></tr></thead>

C</td> Yes</td> (all)</td></tr>
C++</td> Yes</td> (all)</td></tr>
Python 2</td> Yes</td> Python 2.7.18</code></td></tr>
Ruby</td> Yes</td> ruby 2.7.2p137 (2020-10-01 revision 5445e04352) [x86_64-linux-gnu]</code></td></tr>
Java</td> Yes</td> openjdk 11.0.11 2021-04-20</code></td></tr>
Node.js</td> Yes</td> v12.21.0</code></td></tr>
Haskell</td> Yes</td> The Glorious Glasgow Haskell Compilation System, version 8.8.4</code></td></tr>
</td> </td> </td></tr>
Rust</td> No</td> rustc 1.59.0 (9d1b2106e 2022-02-23)</code></td></tr>
Python 3</td> No</td> Python 3.9.5</code></td></tr>
Perl</td> No</td> perl 5, version 32, subversion 1 (v5.32.1) built for x86_64-linux-gnu-thread-multi (with 46 registered patches...)</code></td></tr>
Perl 6</td> No</td> v2020.12</code></td></tr>
Bash</td> No</td> GNU bash, version 5.1.4(1)-release (x86_64-pc-linux-gnu)</code></td></tr>
Awk</td> No</td> GNU Awk 5.1.0, API: 3.0 (GNU MPFR 4.1.0, GNU MP 6.2.1)</code></td></tr>
OCaml</td> No</td> 4.08.1</code></td></tr>
Tcl</td> No</td> 8.6.11</code></td></tr>
C#</td> No</td> Mono JIT compiler version 6.8.0.105</code></td></tr>
</tbody></table>
A more complete and current list is maintained here</a>.</p>


Porting Rust's std to rustix
2022-01-04T00:00:00+00:00
Rustix</a> is a system-call wrapper library with multiple backends. It has a raw
Linux syscalls backend, as well as a libc backend, and other backends</a> are in
development. Rustix is designed for memory safety, I/O safety</a>, and
performance.</p>
And this</em></a> is a branch of Rust's std partially ported to use rustix in
place of direct libc calls. Read on for why this is cool, and stay for the
benchmarks!</p>
Factoring out unsafe</code>, error handling, and raw pointers</h2>
The first reason that porting std to rustix is cool is that rustix factors out
a lot of unsafe</code> blocks from std. Talking to the OS still requires unsafe</code>, but
with rustix the unsafe</code> blocks are focused on individual syscalls. That way, the
unsafe</code> blocks that remain in std are the interesting ones, where std itself is
doing something interesting that needs unsafe</code>.</p>
Rustix also provides idiomatic Result</code> error handling for system calls. And
it uses Rust references and slices instead of raw pointers. These make it
easier to read std's code and focus on the important semantics of the system
calls, without the distractions of libc API mechanics.</p>
And, rustix simplifies some minor infelicities in syscall APIs related to
C integer type sizes.</p>
Putting these all together in an example, this code</a>:</p>
   </span>let</span> len = cmp::min(buf.</span>len</span>(), <wrlen_t>::</span>MAX </span>as </span>usize</span>) as </span>wrlen_t</span>;
</span>   </span>let</span> ret = </span>cvt</span>(</span>unsafe </span>{
</span>       c::send(</span>self</span>.inner.</span>as_raw</span>(), buf.</span>as_ptr</span>() as </span>*const c_void</span>, len, </span>MSG_NOSIGNAL</span>)
</span>   })?;
</span></code></pre>
becomes this</a>:</p>
   </span>let</span> ret = rustix::net::send(&</span>self</span>.inner, buf, SendFlags::</span>NOSIGNAL</span>)?;
</span></code></pre>
This puts the focus on the send</code> operation, without the distractions of
unsafe</code>, raw pointers, wrlen_t</code> types, and cvt</code> error handling.</p>
A path to a Rust on Linux without libc</h2>
A second reason this is cool is that it's a step towards a Rust toolchain
on Linux that doesn't depend on libc.</p>
Rustix is able to make direct Linux syscalls from Rust code. And origin</a> is a
Rust library which is able to startup and shutdown processes and threads
(comparable to crt1.o and libpthread).</p>
With these, we have all the things needed to run Rust programs on Linux. And
it turns out there are two different ways to do this. The first way is mustang</a>.</p>
Mustang is uses a library called c-scape</a>, which wraps rustix with libc-compatible
APIs, allowing std to use rustix without modifications, including
threading support</a>. This has gotten a lot of functionality up and running;
mustang can run a lot of real-world code now. And mustang helps test rustix and
origin. And beyond that, the c-scape libc compatibility layer has several additional
uses. But, mustang in its currrent form looks like it would be awkward to upstream
into Rust.</p>
Fortunately, mustang's architecture of keeping c-scape as a separate layer on top
of rustix and origin, with rustix and origin providing idiomatic Rust APIs, means
that another way is possible as well. This blog post is about starting to port std
to rustix directly. In addition to not using libc code, this path doesn't use libc
APIs either.</p>
Performance</h2>
And another reason this is cool is that rustix-enabled std also brings several
modest speedups, compared to std in upstream Rust. On machines I've tested it on:</p>

std::fs::metadata</code> is about 3% faster</li>
std::fs::File::open</code> is about 5% faster</li>
std::fs::read_to_string</code> is about 3% faster </li>
std::time::Instant::now</code> ranges from 1% to 10% faster on Linux</li>
</ul>
See the benchmark source code</a> to see what's being measured.</p>
Rustix makes metadata</code>, open</code>, and other filesystem path operations faster by
avoiding a dynamic allocation when converting from Rust strings into
NUL-terminated C strings. Rustix uses stack-allocated memory for strings up
to a reasonable size, which is much faster.</p>
Rustix makes Instant::new</code> faster on Linux by simplifying error handling,
since we know that reading the system monotonic clock never fails (and as in
current std, it still does panic if a failure ever does occur).</p>
And rustix contains a number of other optimizations, such as using inline
system calls and avoiding the TLS errno</code> variable on Linux, but so far
benchmarks confirm the common wisdom that these aren't usually very
significant compared to the cost of the actual syscalls.</p>
Looking forward</h2>
A goal for this port of std to rustix is to eventually propose it to be merged
into upstream Rust. I'm hoping this blog post will start some conversations
about what this should eventually look like.</p>
With rustix's libc backend, std can continue to support all the libc-using
platforms that Rust currently supports. And with either backend, rustix brings
the advantages of factoring out unsafe</code>, error handling, and raw pointers,
and its optimizations for converting to C-style strings.</p>
This project promotes several other goals as well, such as promoting
I/O safety</a> concepts and APIs, helping test some of the infrastructure used
by cap-std</a>, and helping set the stage for future projects related to
sandboxing, WASI, nameless</a>, and other areas.</p>
Thanks!</h2>
Thanks to @nivkner for implementing support for child processes, to
@Urgau for adding arm support and implementing several features in rustix,
@cole-miller for implementing getcwd</code> and chdir</code> in mustang, and @jplatte
and @ratmice and others for contributing useful patches, and @tshepang in
particular for bringing up ripgrep</code> as a testcase (and it almost works</a>
now!).</p>


Context Brainstorming
2021-12-23T00:00:00+00:00
This is a blog post brainstorming about contexts</a>.</p>
I'll us the term contexts</em> here, as tmandry is leaning to</a>, since it seems to
make sense to keep capabilities distinct concepts. Idiomatic capability-based
code and the Principle of Least Authority prefer fine-grained access to resources</a>,
which contexts don't seem like a good fit for. So let's keep these concepts distinct
for now.</p>
yoshuawuyts showed me there is way we might use something like contexts to
retrofit an awareness of ambient authority into Rust. Here's an attempt to sketch
up more of what that might look like.</p>
Automatic contexts</h2>
Let's extend the contexts proposal with a concept of automatic</em> contexts, that
functions would implement by default. Just like how Rust has automatic trait impls.
Like automatic trait impls, you can opt out, with negative with-declarations,
using !</code> syntax.</p>
And let's introduce the concept of supercontexts, which are contexts that imply
other contexts. Much like supertraits in Rust. This isn't strictly necessary,
but it helps with granularity.</p>
With those, and the observation that contexts are a way of coloring functions</a>,
let's introduce some hypothetical automatic contexts:</p>


global_allocator</code>, the ability to use the Rust global allocator</a>.</p>
</li>

ambient_authority</code>. Similar to this AmbientAuthority</code></a>, but
as a context, so it can be more. This would be a supercontext which includes:</p>


fs</code> - the current process' filesystem namespace</p>
</li>

net</code> - the current process' network namespace</p>
</li>

time</code> - the current process' time namespace. Preventing code from
observing time entirely is hard, especially if there can be multiple
threads, so maybe this time</code> would just be about the explicit time
APIs rather than blocking all potential time sources.</p>
</li>

stdio</code> - access to the ambient stdin, stdout, and stderr</p>
</li>

process</code> - the ability to spawn arbitrary child processes</p>
</li>

mutable_static</code> - write to or</em> read from statically-allocated mutable
and</em> interior-mutable state in the process. There are use cases where
statically-allocated state is useful, but since we have contexts here, for
maximal modularity, these cases should ideally use contexts instead of
implicitly associating state with the whole process.</p>
</li>

and others. OS's attach a lot of miscellaneous authorities to processes.
Ideally we'd make sure we have everything covered.</p>
</li>
</ul>
</li>
</ul>
These being automatic is kind of a way to retroactively reinterpret existing
Rust code. All code now defaults to having these contexts, and we can then
opt out of them, like this:</p>
fn </span>useless</span>()
</span>with
</span>    !ambient_authority 
</span>{
</span>}
</span></code></pre>
Here, we can know immediately that this useless</code> is a useless function just by looking
at its signature. It has no return value, no arguments, no ambient authority. All
it could do is return, panic, or infloop.</p>
Panic could unwind, and it'd be nice to add a context for that too:</p>

unwind</code> - the ability to unwind the stack</li>
</ul>
then !unwind</code> could be used for functions that can't unwind. Maybe this
could even be connected to LLVM's nounwind</code></a>. Anyway, with !unwind</code>, we could
write code like this:</p>
fn </span>totally_pure</span>(</span>a</span>: &A) -> B
</span>with
</span>    !ambient_authority +
</span>    !unwind
</span>{
</span>    </span>// lots of interesting stuff
</span>}
</span></code></pre>
I think someone told me once that the Rust compiler can know whether
types have interior mutability. Let's assume it can, and that this includes
types that hold I/O handles</a>. In theory, if A</code> here has no interior mutability,
this should allow Rust to annotate functions like this with optimizer attributes
like LLVM's readonly</code></a>, meaning calls to it could be redundant-code-eliminated.</p>
Beyond just LLVM though, this could enable MIR-level redundant-code elimination
of calls, even pre-monomorphization. No need to do complex alias analysis or
escape analysis, because the type system just tells you what you need to know
up front!</p>
But it wouldn't get dead-code elimination, because of the possibility of
inflooping. More on that later.</p>
Pure, except where indicated otherwise</h3>
By the way, if one of the arguments has a type that does have an I/O handle,
including a filesystem handle, then the function can always do I/O. The fs</code>
context is about the process' filesystem namespace. So with
!fs</code>, you can't do File::open</code>, but you can use a Dir</code></a> you've been given
as an argument to do Dir::open</code></a>, because it's resolved relative to a directory
you have an explicit handle to, rather than the process' filesystem namespace.</p>
Similarly, passing a &mut</code> reference into a function marked this way requires
no special ceremony. Unlike "pure" keywords in languages where purity is all
or nothing, the rule here is, if the signature has a &mut</code>, the callee can
access it as a &mut</code>, including mutating it:</p>
fn </span>pure_except_as_obvious</span>(</span>a</span>: &A, </span>m</span>: &</span>mut</span> M, </span>f</span>: &File) -> B
</span>with
</span>    !ambient_authority +
</span>    !unwind
</span>{
</span>    </span>// lots of interesting stuff, including mutating `*m` and writing to `*f`.
</span>}
</span></code></pre>
See First-class I/O</a> for more discussion of this.</p>
Security</h2>
It may be surprising that that I haven't talked about security in this post yet. It
turns out that capability-based security really is just a special case of a
deeper capability-based design philosophy. It's similar to how Rust's borrow
checker is, on its face, a memory-management strategy, but also much deeper,
with things to say about such seemingly unrelated areas as thread safety, pointer
aliasing, iterator invalidation, and refactoring. There's a lot going on here.</p>
It also turns out that security for untrusted or compromised-supply-chain
code is complex. For example, if we want to completely sandbox a piece of Rust
code with language mechanisms, we need to make sure it can't use unsafe</code> blocks,
since unsafe Rust could trivially escape any sandbox. Security exploits are ok
relying on UB if it works with enough probability in practice.</p>
Getting closer: unsafe</h3>
This post is all about contexts though, so let's see if we can use them to fix
that problem too:</p>

new_unsafe</code> - an automatic context representing the ability to introduce
new unsafe contexts. This corresponds to the ability to write unsafe { ... }</code>.</li>
unsafe</code> - a retroactive reinterpretation of what unsafe fn</code> desugars to.
Includes new_unsafe</code> as a subcontext, or doesn't, depending on how
unsafe blocks in unsafe functions</a> goes.</li>
</ul>
As an aside, contexts would also be a path for libraries to define unsafe-like
concepts for their own invariants, which is something I occasionally see people
asking for in Rust.</p>
With new_unsafe</code>, we could write:</p>
fn </span>untrusted_code</span>(</span>x</span>: &X, </span>y</span>: &</span>mut</span> Y) -> Z
</span>with
</span>    !ambient_authority +
</span>    !new_unsafe
</span>{
</span>    </span>// untrusted code here?
</span>}
</span></code></pre>
Would this be a secure sandbox? Not yet; one problem is that even if we
know X</code> has no interior mutability or I/O handles, this code still
exposes the address</em> of x</code> or y</code> to untrusted code, because converting a
reference to a raw pointer doesn't require unsafe</code> in Rust. The address
might tell an attacker something about the ASLR</a> in the process, which
might make other attacks more powerful.</p>
Still getting closer: raw pointers</h3>
When all you're doing is writing a blog post about contexts, everything
looks like a problem to be solved by adding a new automatic context.</p>

raw_pointers</code> - the ability to convert references into raw pointers.</li>
</ul>
In addition to solving this ALSR problem, this attribute has some
interesting possibilities. It's awkward how Rust allows APIs with reference
arguments to observe whether two references have the same address, when
this usually isn't part of the conceptual API. !raw_pointers</code> would be
a way to declare that a function doesn't do that.</p>
Further, with !raw_pointers</code>, it'd be possible to have Rust code that
doesn't depend on a byte-addressed address space. There'd be no alignment or
endianness visible. Objects could be moved at any time, just like in a
moving GC. Threads could be migrated to different stacks. This might even
open up a path to Rust being able to use Wasm reference types, which Rust
can't otherwise hold directly since they're opaque and can't have their
representation exposed.</p>
Are we secure yet?</h3>
No. But, to keep this blog post scoped, let's ignore side-channel attacks like
Spectre, hardware attacks like Rowhammer, crypto miners, and denial-of-service
attacks. And let's ignore attacks which change the behavior of the code without
breaking the sandbox, such as changing an encryption implementation to emit
syntactically valid but insecure data. That's a lot to ignore in reality, but
the solutions to those would require radically different mechanisms, so let's
put those aside for now.</p>
Ok, now</em> are we done yet?</h3>
What about global variables? We included mutate_static</code> in ambient_authority</code>
above, so they won't be mutated, but is it a problem if the untrusted code
reads any of the program's global immutable state? Could it find authentication
secrets? To answer this, we'd need to start getting more specific about the
threat model. But to keep things simple, let's say the program doesn't have
anything sensitive in immutable global state. It's best to keep sensitive
things like authorization credentials as scoped as possible in general anyway.</p>
Along those lines, what about std::env::vars</code>, std::env::args</code>, std::env::home_dir</code>
and others? They might contain sensitive information, or even just your username.
Let's say these are disallowed by !mutable_static</code> by virtue of being mutable
through libc APIs. Or, if needed, we could also add a new context to cover these.</p>
Will it ever stop</h3>
This is just a brainstorming post, and it's possible things are missing, but
it's likely any such things can be covered by adding more contexts. For the sake
of making a finite blog post, let's assume we can cover everything.</p>
So can we say then, that we now, assuming all of our assumptions, finally</em> have
a secure hypothetical sandbox here?</p>
fn </span>untrusted_code</span>(</span>x</span>: &X) -> Y
</span>with
</span>    !ambient_authority +
</span>    !new_unsafe +
</span>    !raw_pointers
</span>{
</span>    </span>// untrusted code here!
</span>}
</span></code></pre>
Yes.</p>
beat</em></p>
In theory.</p>
In practice, the Rust compiler isn't currently designed or intended to be used
as a security surface in this way. And it's not necessarily worth it for it to
try to be one. There'd be work involved, and for this to actually make sense,
we'd need to look at real-world use cases and attack vectors, and we wouldn't
be able to ignore any of the things we ignored above.</p>
Capability-based programming</h2>
However, even if we don't look to !ambient_authority</code> to be the basis of an
actual sandbox, and even if the performance impacts of the aliasing, escaping,
and side effect knowledge isn't compelling, this overall technique might still
be useful.</p>
For people reviewing code, !ambient_authority</code> could reduce the
reasoning footprint</a>, because they'd be able to make more local assumptions
about the side effects of calling functions.</p>
And for people building large complex applications, it could give them more
tools to help ensure that two parts of the application don't have unintended
interactions, as explored here</a>.</p>
And for people building wasm components</a>, it could give them more tools to
ensure that they're only using APIs which compose cleanly with other
components.</p>
Potential downsides</h2>
With all these colors, and with users having the ability to define their
own colors, we could end up with a lot of colors.</p>
Will having an ecosystem where everyone can use all these colors to enforce
their requirements with extraordinary precision increase or decrease overall
usability of Rust? Will it lead programmers to waste time pursuing every
possible dimension of theoretical purity, regardless of what really matters
in practice?</p>
Will these new colors and automatic contexts prompt new rounds of users
going through all their dependencies and insisting that they support new
colors? If so, will it cause ecosystem churn and/or awkward workarounds,
or even ecosystem fragmentation, like #![no_std]</code> sometimes does, and is
that worth it?</p>
I don't know.</p>
What I do know is, in a vacuum, it sure is fun to think up new colors.</p>
Tangent: Pretty colors</h2>
Let's think about one more possible context, for fun:</p>

turing_complete</code> - the ability to have loops (or tail
recursion, if Rust adds that), that can't be proved to terminate.</li>
</ul>
The halting problem gets talked about a lot. However, how often does one
actually write loop</code>, as opposed to just using for</code>? If we also had a way
to assert that iterator implementations don't repeat themselves, a lot of
real-world code might be able to be compatible with !turing_complete</code>.</p>
One of the tricky issues for iterators would be linked lists, which would need
to be guaranteed to be acyclic. But it's interesting to note that in Rust,
creating a circularly linked list actually requires unsafe</code> anyway. So maybe
there's something we could do here.</p>
This would also address the "or infloop" case mentioned above, so we could
also get dead-code-elimination of calls.</p>


The Spectrum from Namespaces to Values
2021-12-22T00:00:00+00:00
In order to make large applications modular, we need to think about
resources and sharing. One axis for thinking about this is the spectrum
of granularity. It's a spectrum, but we can identify several notable
levels:</p>

Level 0: File::open</code></li>
Level 1: with root: &Dir</code></li>
Level 2: dir.open</code></li>
Level 3: thing: &File</code></li>
Level 4: thing: &StreamReader</code></li>
</ul>
This post describes each of these levels and considers the impact on
application modularity of moving up through each of the levels.</p>
As a caveat, in this post, I'll be discussing trusted</em> code. It's about
modularity and flexibility. Specifically, I'm ignoring the concerns of
untrusted</em> code here. If there's code in the system which is actively
attempting to break out, it requires additional mechanisms to securely
contain it. The concepts discussed here can be used as part of a sandbox
for untrusted code, but a proper sandbox involves many additional concerns
that I'm not covering here.</p>
Usually when someone writes a post about capabilities and the Principle
of Least Authority, they're talking about security. However, this post
is about how these concepts are relevant to programming in the large
in general.</p>
With that in mind, let's start at the bottom and work our way up :-).</p>
Level 0: File::open</code></h2>
Level 0 is how most application code today uses filesystems:</p>
fn </span>foo</span>() {
</span>    </span>let</span> thing = File::open("</span>/path/to/some/dir/thing</span>");
</span>}
</span></code></pre>
There's a filesystem namespace implicitly associated with the program, and
filesystem paths are resolved in it. Lots of code works this way to day, and
it works. But, traditional OS isolation mechanisms are not sufficient for many
popular use cases today, so many use cases end up using containers</em>. A
container provides a little world for an application to run in, where it can
pretend it's running on a "normal computer" built specifically for it.</p>
However, containers are coarse-grained. One typically needs to run an entire
application within the same container. Or if applications are split into
parts that run in different containers, the parts usually need to communicate
over invasive RPC mechanisms. By climbing up to the next level of granularity,
we can gain more options.</p>
Level 1: with root: &Dir</code></h2>
Using a mechanism such as cap-std's Dir</code></a> type, we could arrange to pass
in a "root filesystem" as a parameter rather than using a namespace implicitly
attached to the program:</p>
fn </span>foo</span>(</span>root</span>: &Dir) {
</span>    </span>let</span> thing = dir.</span>open</span>("</span>/path/to/some/dir/thing</span>");
</span>}
</span></code></pre>
And if we write a lot of code like that, we might find a recent blog post
about context capabilities</a> useful. That would allow us to write the above
code like this:</p>
fn </span>foo</span>()
</span>with
</span>    root: &Dir
</span>{
</span>    </span>let</span> thing = File::open("</span>/path/to/some/dir/thing</span>");
</span>}
</span></code></pre>
Unlike the standard-library File::open</code>, this would use a File::open</code> that
also uses context capabilities, so it would also have a with root: &Dir</code>, and
it'd automatically get passed the root in which to resolve the path.</p>
This approach has the appealing property that the body of the code is the exact
same as it is in level 0. It would be easy to convert existing code to use this
style, because it just requires changes at the high-level scopes.</p>
Going from 0 to 1 means that an application could have the ability to create
little dedicated filesystem worlds, and run parts of itself in these little
worlds, with each part having its own root filesystem. It wouldn't have to
completely split into multiple containers.</p>
That's a step up. However, creating dedicated filesystems is still complex and
potentially inefficient. How do we know what those little worlds need to
contain?  It requires us to know the set of strings that the code might
dynamically pass to an open</code> function. Sometimes we can get a pretty good idea
of code code needs by studying it. And sometimes we resort to running code and
just dynamically recording what things it needs.</p>
And, if two parts of an application want to communicate, we still have to figure
out if they need to run in the same little world, or if they can run in different
worlds. And if they are in the same world, to share data through it, we then need
to somehow make sure they don't collide in unintended ways in other parts of
their shared filesystem, such as needing different versions of a dependency
installed at the same path.</p>
Level 2: dir.open</code></h2>
The next step up is to break up the monolithic "world" into more fine-grained
handles. cap-std's Dir</code> type makes it easy to have first-class directory
handles, so we can just pass those around.</p>
fn </span>foo</span>(</span>dir</span>: &Dir) {
</span>    </span>let</span> thing = dir.</span>open</span>("</span>thing</span>");
</span>}
</span></code></pre>
We could use the above-mentioned context capabilities for these. However that
syntax is less valuable once we start getting more fine-grained, as we do here
at level 2. When we're talking about a whole filesystem, it's common for code
to want to treat it like an ambiently present resource, and context capabilities
approximate that. But once we can talk about specific directories, it's more
natural for code to be aware of the specific directories it needs, and take those
directories as explicit parameters. It's no longer about "here's a world to
operate in", it's about "here's the directory you asked for".</p>
Passing individual directories is a step up from level 1. With this, when
an application is split into parts, we don't need to create whole filesystem
views for each part. We just need to give each part its own dir</code> directory.
And if they do need to share, they just share those directories, and we don't
have to worry about them colliding in other areas of their filesystems.</p>
But if the only thing foo</code> needs is one file in that directory, passing
it a whole directory is still more complex than we need. How can we know when
two parts of an application need to share a directory? As above, the only way
to know how they interact is to look at the set of strings they dynamically
pass into open</code> functions.</p>
Level 3: thing: &File</code></h2>
If foo</code> just needs a single file, it can be passed just that file:</p>
fn </span>foo</span>(</span>thing</span>: &File) {
</span>    </span>// ...
</span>}
</span></code></pre>
This way, an application can be split into parts, and the parts can freely
operate on different files. The application doesn't need to worry about
whether it needs to separate those files into separate directories. Even if
the files all live in the same directory, the parts of an application
accessing individual files won't collide, because they only have their
&File</code>, not the whole directory.</p>
Level 4: thing: &StreamReader</code></h2>
The next level up is to observe that most applications don't need an actual
File</code> as such. Even though Unix told us that "everything is a file", pipes
and sockets aren't really files, in that they don't always have filenames, and
don't support all of the File</code> operations. However, in a lot of use cases,
applications would ultimately be ok reading from a pipe or a socket just as
well as an actual file. In some cases, we may want them to simply read data
from a buffer in memory. The thing that most application code needs is
just a source of data to read from.</p>
This could be impl Read</code>, Box<Read></code>, or the io-stream crate's StreamReader</code></a>.
Or the async versions of any of those. But that's getting into the mechanics
of how the I/O is done. It's an interesting topic, but not the focus of this post.</p>
The important part for this blog post is that level 4 is where we reduce a piece
of code's requirements down to just an input stream, so it can be connected to
other pieces of code without having to think about what other resources
that might cause them to implicitly share. Input streams can be streamed
across networks. They can be buffered, or stored and replayed. If we can write
code that works in terms of streams instead of files, we have a lot of flexibility,
because we don't need to worry about the code implicitly depending on file-specific
semantics.</p>
And beyond</h2>
The next step up from byte streams would be typed streams, iterators, or
generators. A big sequence of u8</code>s requires the producers and consumers of
the data to have some implicit agreement about how the bytes are interpreted.
An API providing a sequence of values of some type T</code> would allow producers
and consumers to be self-describing. The mechanics of these APIs are another
interesting topic, and also not the focus of this post :-).</p>
The focus is that once we have an abstraction level where bodies of code can
communicate typed values, these bodies of code can communicate precisely what
they need, without incompatibilities among things they implicitly share.</p>
At each step up, we got closer to "what does the code actually need?", and
found ways to give it what it needed, while reducing the amount it implicitly
shares with other parts. This reduced the potential for conflicts with other
code, and simplified the task of breaking up applications into parts.</p>
An analogous concept in computing is false sharing</a>, except here, instead of
performance and sharing parts of cache lines that we don't need, it's about
modularity and sharing parts of monolithic resources that we don't need.</p>
By applying the Principle of Least Authority, and passing around fine-grained
resources to code that advertises specific needs, we improve modularity.</p>


Thread support in Mustang
2021-10-14T00:00:00+00:00
Mustang</a>, a system for running Rust programs entirely written in Rust, has
make a lot of progress since the last blog post:</p>

New targets: riscv64 and arm, joining x86_64, aarch64, and x86;
thanks to @Urgau for arm support in rustix!</li>
Threading support, including TLS, TLS destructors, and detaching</li>
Panic and unwind support, thanks to the unwinding</a> crate!</li>
Math library support, thanks to the libm</a> crate!</li>
A proper allocator, thanks to the dlmalloc</a> crate!</li>
No more debugging messages on stderr by default</li>
DNS (ToSocketAddrs</code>) support</li>
Much smaller code size</li>
Lots more test coverage</li>
</ul>
Mustang's thread library is a chance to explore the role of a thread library,
the special syscalls and registers that only a thread library uses, and the
interaction between a thread library and other system calls. The rest of
this blog post takes a closer look.</p>
I'd like to thank Steed's pthread implementation</a> for the initial
inspiration here, and demonstrating how to use clone</code>, futex</code>, and the
platform thread register.</p>
So, what does a thread library do?</p>
Create threads</h2>
First, what is a thread?</p>
A thread consists of an OS thread, which on Linux is just a special kind of
process, and some data: metadata, user TLS data, stack memory, and a stack
guard. Creating a thread involves allocating memory for the data, and then
creating an OS thread configured to use them.</p>
Mustang's origin</a> crate allocates all the memory for a thread in a single
contiguous anonymous mmap</code> allocation. Most platforms have a special
"Thread Pointer" register which is used to point to thread-specific data,
%fs</code> on x86_64, %gs</code> on x86, tpidr_el0</code> on aarch64, tp</code> on RISC-V, and
so on, and we use this to point to this allocation. Compiled code uses this
register to locate TLS data, and origin uses this register to locate its
metadata.</p>
</p>
The specific layout differs between architectures</a>, for example with the
metadata being located before or after the user TLS data. And things would be
much more involved if we were talking about dynamic linking. But this covers
the basics.</p>
Once the data is allocated and initialized, it's time to create the OS thread.</p>
The clone</code> system call</h3>
Origin creates an OS thread using the clone</code></a> system call. Linux considers
this to be similar to fork</code></a>, which creates new processes, but with more
options. A new thread is a new execution context, with its own CPU register
state, similar to a new process. The main thing that makes it a thread instead
of a process is that it happens to use the same virtual address space and other
parts as its parent.</p>
</p>
Like fork</code>, clone</code> is invoked by one execution context, and when it returns
there are two execution contexts, one for the parent and one for the child.</p>
Unlike fork</code> though, when the child shares virtual memory with the parent, it
can't continue on the same stack, so we pass clone</code> the pointer to the stack
we allocated above for the child to run on.</p>
The child starts execution by returning from the clone</code> call, and the return
value of the clone</code> call is 0 in the child, and the child's thread id in the
parent, so the code after the clone</code> call can test whether it's running in
the child or the parent.</p>
The clone</code> system call doesn't have a built-in way of passing any arguments to
the child to tell is what to do. To do that, we place additional arguments in
registers that we know the system call doesn't clobber, effectively "passing"
them to the child.</p>
To illustrate, here's an illustrated version of the RISC-V code for this. Other
architectures do equivalent things, though eg. x86 has more ABI details in
play.</p>
// Place syscall arguments in a0, a1, ...,
</span>// and `__NR_clone` in a7.
</span>//
</span>// And place the function pointer for the child to call, and
</span>// the argument to pass to it, in a8 and a9, which are not
</span>// used by the syscall, but not clobbered either.
</span>...
</span>
</span>ecall                </span>// Do the `clone` system call.
</span>bnez a0, </span>0</span>f          </span>// If we're in the parent, branch to 0:
</span>
</span>// We're in the child! Move the argument and function
</span>// pointer into calling-convention registers for the call to
</span>// `entry`.
</span>mv a0, a8            </span>// `arg`
</span>mv a1, a9            </span>// `fn_`
</span>
</span>// Zero out the frame address and return address. The call
</span>// below will never return.
</span>mv fp, zero
</span>mv ra, zero
</span>
</span>// Call into our Rust code which will call the callee,
</span>// passing it the argument. The Rust code will terminate
</span>// the thread and not return here.
</span>tail </span>entry
</span>
</span>// We're in the parent! Return from the system call in the
</span>// normal way.
</span>0</span>:
</span>...
</span></code></pre>
origin's philosophy is to do as little as possible in assembly, so this code
just sets the frame pointer and return address to null, to ensure that we never
try to return back to our little assembly fragment, moves the arguments into
place for the platform calling-convention argument registers, and calls into
Rust code, specifically the threads::entry</code> function</a>, to do the rest.</p>
As an aside, this is very similar to the code origin uses for the start of
the process, _start</code></a>, which calls program::entry</code> function</a>. The OS
doesn't start a process in a manner compatible with the platform
calling-convention, so we use a minimal amount of assembly to set up a call
into Rust code, also including setting the return address and frame pointer
to null.</p>
In both threads and processes, we always exit by calling the exit</code>
(for threads) or exit_group</code> (for the process) system calls, and never by
returning to assembly code.</p>
Join threads</h2>
When one thread wants to wait for another to exit, it performs a "join".</p>
Once the other thread exits, the joining thread is woken up, and it frees
any resources associated with the other thread, and collects the return
values (though Rust's std::thread</code> implementation doesn't use thread return
values as such, so the return value part isn't implemented in Mustang yet).</p>
In the clone</code></a> system call that we called in create_thread</code> above, one of the
flags we pass is CLONE_CHILD_CLEARTID</code>. This tells Linux to clear the child thread
id and wake up any futexes that are waiting on that memory location when the
child exits. It's important that it does both, so that the parent's futex call
can avoid waiting if the child has exited first.</p>
If the child hasn't exited yet, join_thread</code>'s futex</code> call waits until the
child exits and Linux wakes it up. If the thread wasn't detached (more on that
below), joining then frees the thread's memory—its stack, TLS data, and
metadata.</p>
In the code, this is join_thread</code></a>, which calls wait_for_thread_exit</code>, and
then free_thread_memory</code>.</p>
Run thread destructors (registered with __cxa_thread_atexit_impl</code>)</h2>
Variables declared with the thread_local</code></a> macro can have Drop</code>
implementations, and those drop</code> functions are called on a thread's copy
of the data when the thread exits.</p>
__cxa_thread_atexit_impl</code> is the C ABI function to register a cleanup
function to call when a thread exits. It simply pushes the function onto
a Vec</code>, and then the thread exits, it calls call_thread_dtors</code></a> to
call the functions in the Vec</code>, in reverse order of their registration,
to ensure that object initializations and finalizations are nested.</p>
Detach threads</h2>
"Detaching" a thread declares that it will never be "joined" by any other
thread, so it needs to release its resources on its own when it exits.</p>
This is a fairly straightforward matter of just doing bookkeeping to
keep track of whether a thread is in the detached state, and freeing resources
when it exits if it is, but there are a few catches.</p>
One is that we create threads with clone</code> with the CLONE_CHILD_CLEARTID</code>
flag, which tells Linux to zero out our tid field when the child exits.
That helps join_thread</code>, but for detached threads, it means that if we free
our memory before we exit, the kernel will still think it needs to clear the
tid field. Something else in the program could reuse our free'd memory, and
it could get corrupted if Linux clears out what is no longer our tid field.
So when a thread is marked detached, we call the set_tid_address</code> system
call, passing it a null pointer, which effectively disables the
CLONE_CHILD_CLEARTID</code> for the thread.</p>
The other is that a thread's memory is freed by an munmap</code> system call, but
if we call the usual munmap</code> function that wraps the system call, it will
free the current thread, including the stack we're running on out from
underneath us. The munmap</code> wrapper will then access freed memory trying to
read its return address for the return jump.</p>
To fix this, we need more assembly code. We need a special code
sequence to make a munmap</code> system call, and then make an exit</code> system
call to exit the thread, without touching the stack after the munmap</code>.
Fortunately, this can be done in just a handful of instructions</a>.</p>
Note that on 32-bit x86, we normally don't make syscalls directly, because
the usual int 0x80</code> mechanism is very slow, and Linux provides a much
faster way to make syscalls through the vDSO</a>. However, we can't use that
here because the vdso is effectively another syscall wrapper that
expects to store a return address on the stack. So we use int 0x80</code>
so that we can completely prevent touching the stack. Fortunately,
threads don't exit all that often, so the performance loss here isn't
that important.</p>
Synchronize threads</h2>
It's common for threading libraries to provide synchronization primitives
such as mutexes and reader-writer locks.</p>
Mustang's main strategy is to use parking_lot</code></a> to implement these, since
parking_lot</code> is a widely-used library, and there's not a lot to add here.</p>
Except there's one place Mustang can't yet use parking_lot</code>: the Rust global
allocator uses a Mutex</code>, because it's allocating memory for multiple threads
from a shared heap. But the twist for Mustang is, parking_lot</code>'s Mutex</code>
does global allocation.</p>
Uh oh.</p>
Allocator tries to acquire a lock, which tries to perform an allocation,
which tries to acquire a lock, which tries to perform an allocation...</p>
💥</p>
So what do we do? For now, Mustang has its own Mutex implementation</a>, built
on top of atomics and Linux's futex</code>. It's nothing fancy, it's unfair,
it's inefficient in a bunch of ways, and it's certainly not proven correct.
But it's pretty simple and it passes all the tests! This is your regular
reminder that simply implementing everything in Rust does not make anything
automatically safer. Mustang is still experimental at this point, with lots
of unsafe</code>.</p>
But not cancellation</h2>
The other really big piece of functionality that a libpthread would provide is
cancellation</a>. Cancellation is very complex, and requires hooks in libc and
very special handling of many system calls. However, Rust doesn't support
cancellation, and it's relatively rare even in C code, so origin and mustang
don't implement it.</p>
Mustang organization</h2>
The origin crate provides low-level but still somewhat Rust-idiomatic
interfaces to process startup and shutdown, and now also threads. This
accompanies the rustix</a> crate which provides Rust-idiomatic interfaces to
system calls.</p>
The c-scape crate provides libc and libpthread ABIs as wrappers around
rustix and origin. Right now, this allows existing code, such as Rust's
std</code> to run on origin and rustix without any extra porting work.</p>
But also, code that wants to can bypass the c-scape compatibility layer,
and call into rustix and origin directly. This eliminates some overhead,
but more importantly, it offers greater safety and simplicity, because there
are fewer raw pointers, raw file descriptors, and raw error return values.</p>
And beyond!</h2>
Of course, even rustix and origin are still very low-level. Most users should of
course continue to use std</code>, and high-level libraries such as
cap-std</a>. But perhaps someday
there could be a way of using rustix in std</a>, simplifying the code by factoring
out raw pointers, raw file descriptors, and raw error handling.</p>
And perhaps someday, one could even imagine, an official Rust target</a> for
Rust programs built entirely in Rust, with I/O safety down to the syscalls</a>.</p>


Rust programs written entirely in Rust
2021-09-07T00:00:00+00:00
mustang</code></a> is a system for writing Rust programs entirely in Rust, meaning
they don't use libc</code>, crt1.o</code>, or any C code. It's experimental, but it's
complete enough to run a std</code>-using Hello World and other simple programs
on Linux on x86-64, x86, and aarch64.</p>
See mustang</code>'s README</a> for information on how to use it. This post walks
through the major steps leading to this point.</p>
System calls</h2>
The first building block for mustang</code> was rustix</code></a>, a crate providing a
syscall-like API, but which can be configured to use different backends,
currently either libc or raw Linux syscalls. rustix</code> has been a place to
help prototype the io-lifetimes</code></a> crate and I/O safety feature, support
Wasmtime</a>'s WASI implementation, support cap-std</a>'s filesystem sandbox
implementation, and to push the boundaries</a> of low-level performance
while maintaining safe and relatively idiomatic Rust APIs.</p>
These continue to be the main motivators for rustix</code>, and that's an
important point: while mustang</code> isn't particularly practical, it isn't all a
one-off effort. Some of the major pieces have real-world practical use cases.</p>
The first part of the inspiration for mustang</code> was the realization that rustix</code>
could be used in this way.</p>
Process startup</h2>
crt1.o and process startup has always had an aura of mystery about it, at least
to me. There are parts implemented in assembly, parts that use obscure C
compiler features, parts that are heavily conditionalized with ifdefs, and
support for dynamic linking requires several particularly tricky features.</p>
The other part of the inspiration for mustang</code> was the realization that simple
Rust programs don't need most of what goes into crt1.o and libc startup code
The minimum needed is just a little bit of assembly</a> to translate from the
initial process state that the OS sets up into the Rust calling convention.
Once Rust code is running, it just needs to compute argc, argv, and envp</a>,
call any .init_array</code> functions</a>, and then call main</code></a>.</p>
This code is in mustang's origin</code></a> crate; it's very small.</p>
To tell the linker to avoid linking in crt1.o so that it uses origin</code>'s startup
code instead, mustang</code> defines custom targets</a> named *-mustang</code>. These primarily
add -nostdlib</code> to the link command so that crt1.o</code> is not linked in.</p>
Cargo's build-std</a> mode allows mustang</code> to easily compile the standard
library with its custom targets, so it doesn't need to make any upstream
standard library changes (for now...).</p>
C-scape</h2>
If we can start a process with Rust code, and make system calls with Rust code,
what's left that's not Rust?</p>
As a practical matter, Rust's standard library depends on libc</code> and uses many
libc APIs. One option would be to build a runtime library that runs on
no_std</code>. However, in the limit, that would essentially require a
reimplementation of std</code>, which would be a lot of work, and a huge burden to
maintain.</p>
So mustang</code> instead defines C-compatible ABIs</a> providing functions like write</code>,
strlen</code>, memcpy</code>, mmap</code>, and other things used by Rust's standard library,
using rustix</code> internally to do actual system calls. These implementations are
currently very minimal, but they are enough to support Hello World, running
with std</code>.</p>
And going forward, it should be relatively straightforward to add additional
libc</code> APIs as needed by std</code>.</p>
That said, the goal here is to just implement enough of libc to get std</code>
working on top of it. Mustang</code> doesn't need to implement a whole new libc,
which would be a lot more work, and much of it wouldn't be needed in Rust
programs.</p>
This code is in mustang's c-scape</code></a> crate.</p>
The global_allocator</code></h2>
Rust allows users to configure their global allocator with the
global_allocator</code></a> attribute. The default global allocator ultimately
uses libc malloc</code>. Writing a new malloc</code> is a lot more work that I was
looking to do here, so the question here was, is there a nice Rust allocator
crate that can be used with #[global_allocator]</code>?</p>
It turns out, there are a few options, but one which turned out to be
particularly easy to integrate was wee_alloc</code></a>. Built for reducing code size
on WebAssembly, wee_alloc</code> doesn't do fancy optimizations, but it does plug
directly into #[global_alloctor]</code> with no additional hassle, and it is
portable, so it was very easy to set up. And it only uses a few things from
libc</code> which c-scape</code> is easily able to support.</p>
Putting it all together</h2>
The mustang</code> crate pulls all these pieces together to make it easy to use
them all at once. But note that since mustang</code> is aiming at supporting
regular std</code>-using Rust code, it isn't something you explicitly call
yourself. As such, it currently needs an extern crate mustang</code> to link it
into the program.</p>
A note about safety (and a lack thereof)</h2>
There is often a temptation whenever one is working with C code to
Rewrite It In Rust, however it's not always a good idea. In mustang</code>'s case,
existing libc implementations are mature, robust, and well optimized. Libc
interfaces are relatively portable, and on some operating systems, libc
interfaces are the only stable interfaces to operating system functionality.
There's no urgency to stop using libc or to rewrite it.</p>
mustang</code> is inspired by the realization that some of the major pieces were
already available, and the remaining pieces to get basic examples working could
be written with a relatively quick effort. It's new code, it's experimental, it
has a lot of unsafe</code> code, it lacks important optimizations, it's overall
far less safe to use than just using libc, and it would take a lot of work to
change this. But, it's fun, and educational.</p>
It's also possible that in the future, mustang</code> could be part of a path to
designing new low-level system features like command-line arguments, environment
variables, or OS error codes are handled with more safety or other ideas such as
first-class I/O</a>. This is also part of why the goal isn't to just build a Rust
implementation of libc: many libc APIs consist of a small amount of code behind
a function signature that uses raw pointers; just putting Rust behind those
kinds of APIs isn't going to make them significantly safer. But that's another
post; for now...</p>
Hello, World!</h2>
See mustang</code>'s README</a> for step-by-step instructions for how to build an
run mustang</code>.</p>
If you're interested in following the story, helping out, or even just asking
questions, come say hi in the chat channel</a>, or file an issue</a>!</p>


Broken Encapsulation
2021-09-02T00:00:00+00:00
What kinds of bugs should safety</em> in Rust protect against?</p>
Rust clearly wants to say that safety is about protecting programs against some
kinds of bugs, but not all possible</em> bugs. Where should the boundary be?</p>
Safety should at the very least mean protection against memory corruption through
dangling and out-of-bounds pointers. To do that, it's necessary to protect
against all Undefined Behavior, because if behavior of a program is undefined,
anything could happen, including arbitrary memory corruption.</p>
There are also categories of bugs that we don't expect Rust's safety to protect
against. For example, Eq</code></a> implementations must be reflexive, symmetric, and
transitive, however it would likely be impractical to enforce all the invariants
of all such APIs. Fortunately, violating these invariants doesn't cause
Undefined Behavior; it just causes some algorithms to do the wrong thing.</p>
So, Undefined Behavior is a very practical place to put the boundary.</p>
However, there are several potential situations which do not necessarily involve
Undefined Behavior, but which are still arguably within the spirit of
Rust's safety:</p>


I/O safety</a>: Raw file descriptors have the same fundamental properties that
make raw pointers unsafe: They can dangle and they can be be forged. One
crate operating on a dangling or forged file descriptor can end up doing I/O
on file descriptors held in other unrelated crates. Without I/O safety,
it's impossible to characterize the I/O of a crate without considering the
behavior of all other crates it might be linked with.</p>
(I/O safety does intersect with memory safety through mmap</code>, but using mmap</code>
safely is non-trivial in any case, and this is not the only motivation for
I/O safety.)</p>
</li>

POSIX's munlock</code></a> function: This function takes a raw pointer and length, but it's
defined to fail gracefully if given invalid pointers. It doesn't mutate any memory,
or cause any subsequent memory access to behave differently with respect to Rust
language semantics, so it arguably never causes Undefined Behavior. However, if a
crate is internally using locked memory to protect sensitive data, exposing munlock</code>
as a safe function would mean that a wayward munlock</code> call in another crate could
bypass the first crate's encapsulation and munlock</code> the memory, compromising the
sensitive data.</p>
</li>

POSIX's write</code></a> function: this function also takes a raw pointer and a length, and
also guarantees to not segfault or mutate any memory. POSIX isn't clear on whether
write</code> has Undefined Behavior in the presence of data races or provenance violations
or other infelicities with its buffer, but for the sake of this post, let's assume
it doesn't. The memory is read by the OS, which one could argue isn't bound by the
same rules as userspace. In that case, one can argue that write</code> never has Undefined
Behavior. But making it safe would mean safe code in any crate could read encapsulated
memory in any other crate, which seems outside the spirit.</p>
</li>
</ul>
So, instead of Undefined Behavior</em>, a slightly more expanded concept that covers
these cases might be described as Broken Encapsulation</em>. This is a superset of
Undefined Behavior, because anything that causes Undefined Behavior can break any
language-level encapsulation boundary. And like Undefined Behavior, it still excludes
things like Eq</code>'s invariants.</p>
Language-level encapsulation boundaries help in maintaining
Reasoning Footprints</a>, especially in programs that contain many crates. It's what
lets us look at an individual crate and understand its behavior in isolation,
without having to think about whether any other crate in the program could
accidentally observe the crate's internal data, do I/O on its internal file
descriptors, or cause its internal secrets to be swapped out of memory and
potentially compromised.</p>
In practice, thinking about Broken Encapsulation is only slightly different from
thinking about Undefined Behavior, but it also reflects a broader observation:
Guarding against Undefined Behavior is about ensuring that certain kinds of
bugs don't happen, while guarding against Broken Encapsulation is also about
helping users build large programs out of smaller parts.</p>


(renamed)
2021-07-12T00:00:00+00:00
Posish has been renamed to Rustix. Here's the new URL for this page</a>!</p>


(renamed)
2021-07-12T00:00:00+00:00
Rsix has been renamed to Rustix. Here's the new URL for this page</a>!</p>


I/O safety and speed: Why not both?
2021-07-12T00:00:00+00:00
Rustix: I/O Safety in practice</h2>
The I/O Safety RFC</a> is now merged! But it's all fairly abstract, so what will
this look like in practice with real APIs?</p>
One of the ways is rustix, the fastest POSIX-ish, Unix-ish, Linux-ish, and
libc-ish API for Rust!</p>
And it's also I/O-safe! And memory-safe! These are actually the bigger
motivators for this crate, and it helped guide the development of the I/O
safety APIs. But it's also fast, which is fun, and shows that Rust's
"abstraction without overhead"</a> is preserved. More on that later though.
First, what's all this about safety, you say? I'm glad you asked 😉!</p>
Safety</h2>
Memory safety is of course an established concept for Rust. Memory-safe
abstractions for POSIX/libc functions is a well-established idea, with Rust
references, slices and return values in place of raw pointers, and rustix
provides all this as well.</p>
I/O safety is a newly-introduced concept</a> and is about treating file
descriptors similar to pointers, with concepts of ownership</a> and borrowing</a>.
See the I/O Safety RFC</a> and the io-lifetimes crate</a> for details. Rustix uses
io-lifetimes types for all its APIs that work with file descriptors.</p>
A tour of a function in rustix</h3>
Let's take a quick look at the openat</code> function, which demonstrates several
of the things rustix does.</p>
The libc version of openat</code> looks like this</a>:</p>
    </span>pub unsafe extern </span>"</span>C</span>" </span>fn </span>openat</span>(
</span>        </span>dirfd</span>: c_int,
</span>        </span>pathname</span>: </span>*const</span> c_char,
</span>        </span>flags</span>: c_int,
</span>         ...
</span>    ) -> c_int
</span></code></pre>
Rustix's looks like this</a>:</p>
    </span>pub fn </span>openat</span><P: Arg, Fd: AsFd>(
</span>        </span>dirfd</span>: &Fd,
</span>        </span>path</span>: P,
</span>        </span>oflags</span>: OFlags,
</span>        </span>create_mode</span>: Mode
</span>    ) -> Result<OwnedFd>
</span></code></pre>
Here's a breakdown of the differences:</p>

    pub fn openat<P: Arg, Fd: AsFd</span>>(
        dirfd: &Fd</span>,
        path: P,
        oflags: OFlags,
        create_mode: Mode
    ) -> Result<OwnedFd>
</pre></code>
Rustix's dirfd</code> accepts any type which implements the AsFd</code></a> trait, so you
can pass in a &File</code> or &TcpStream</code> or other things conveniently and safely
instead of having to call .as_raw_fd()</code> and pass in the result. AsFd</code>
works like a borrow, and similar to Rust references, it's prevented from
escaping and dangling.</p>
AsFd</code></a> is similar to AsRawFd</code></a>, except that AsFd::as_fd</code></a> returns a
BorrowedFd</code></a> instead of AsRawFd::as_raw_fd</code></a>'s RawFd</code></a>, which is an
integer, it returns a BorrowedFd</code>, which is a special type scoped to the
lifetime of the borrow.</p>

    pub fn openat<P: Arg</span>, Fd: AsFd>(
        dirfd: &Fd,
        path: P</span>,
        oflags: OFlags,
        create_mode: Mode
    ) -> Result<OwnedFd>
</pre></code>
Similar to nix</a>, rustix's path</code> is also generic and accepts any type of
string that implements the Arg</code></a> trait, which includes &str</code>, &Path</code>,
&CStr</code>, &OsStr</code>, and other string-like types in Rust's standard library.
Actual POSIX-ish system calls expect NUL-terminated strings, so there are
optimized paths that avoid dynamic allocation in most cases where you pass
in something other than a &CStr</code>. And rustix avoids making any assumptions
about PATH_MAX</code></a>, so the lengths of strings you can use is between you and
the individual syscalls.</p>

    pub fn openat<P: Arg, Fd: AsFd>(
        dirfd: &Fd,
        path: P,
        oflags: OFlags</span>,
        create_mode: Mode</span>
    ) -> Result</span><OwnedFd>
</pre></code>
Similar to other libc-wrapping creates, rustix's oflags</code> argument takes an
OFlags</code>, which uses the bitflags</code> crate</a> to create a type-checked flags
type instead of a raw integer type. Rustix uses an explicit create_mode</code>
argument instead of ...</code> so it avoids the unsafety of varargs. And rustix
reports errors via a Result</code> instead of a plain c_int</code>, and returns a
Error</code></a> inside the Result</code> instead of setting errno</code>.</p>

    pub fn openat<P: Arg, Fd: AsFd>(
        dirfd: &Fd,
        path: P,
        oflags: OFlags,
        create_mode: Mode
    ) -> Result<OwnedFd</span>>
</pre></code>
On success, rustix returns an OwnedFd</code></a> instead of a raw file descriptor,
modeling the file descriptor's lifetime in the type system, so similar to
a Rust Box<T></code> or other owning type, it's prevented from dangling.</p>
OwnedFd</code> is very similar to std::fs::File</code></a>. From a user perspective, the
main difference between OwnedFd</code> and File</code> is that OwnedFd</code> doesn't suggest
any file-like purpose or behavior, it's just a generic owned file descriptor
for any kind of resource. Most use cases will likely want to wrap OwnedFd</code> in
higher-level types. When OwnedFd</code> is added to the standard library, I expect
File</code> itself will be implemented in terms of OwnedFd</code> in this way as well.</p>

    pub fn</span> openat<P: Arg, Fd: AsFd>(
        dirfd: &Fd,
        path: P,
        oflags: OFlags,
        create_mode: Mode
    ) -> Result<OwnedFd>
</pre></code>
And finally, the openat</code> system call itself is documented</a> to not have any
other side effects, so it has first-class I/O</a>, though at a fairly coarse
granularity since the path</code> can contain ..</code> components and can thereby
reference any path in the filesystem (if you're worried about that kind of
thing, see cap-std</a>).</p>
So with all these together, rustix's openat</code> is a safe function, both in
memory safety and I/O safety</a>.</p>
Cool cool what about speed though?</h3>
On Linux on x86-64, x86, and aarch64, rustix uses avoids libc</code> entirely,
avoiding errno</code> and pthread cancellation checking. On stable Rust it uses
out-of-line asm to perform syscalls; on Rust nightly, it uses inline asm and,
since it adds very little code itself, it can fully inline the syscall
instructions into the user callsites. On all other platforms, it currently
uses libc</code> and errno</code>, but it's factored to facilitate additional backend
implementations in the future.</p>
It also uses an optimized error type and the string argument optimization
mentioned above. In microbenchmarks, at this moment in time, filesystem
syscalls with string path arguments are about 15% faster than nix, and about
3% faster than libc with allocating temporary CString</code>s. And it uses the
vDSO</code> to make clock_gettime</code> really fast on Linux (and to avoid using the
super-slow int 0x80</code> mechanism on x86), just like Linux libc implementations
do, but without depending on libc.</p>
That said, the bigger picture is that most of the time for syscalls is spent in
the OS itself, and rustix makes most syscalls only a few percentage points
faster at most. But these days, with system calls often getting slower, it's
still fun to have some gains in performance in this space, even if they are
relatively small. And it demonstrates that the new abstraction of I/O safety
doesn't entail new overhead.</p>
Is that all?</h3>
Rustix provides a few additional niceties, including:</p>


It always uses 64-bit file offsets, so users don't need to juggle off64_t</code>
vs. off_t</code>, or remember to pass O_LARGEFILE</code> when needed.</p>
</li>

On 32-bit Linux platforms that use direct Linux syscalls, it uses a 64-bit
time_t</code> type when the underlying kernel supports it, so clocks won't wrap
around in the year 2038</a>.</p>
</li>

And as a fun bonus, some names have been made more human-friendly and less
historic-accidental, with names such as accept_with</code> and dup2_with</code> to
take extra flags arguments instead of accept4</code> and dup3</code>, and seek</code>
instead of lseek</code>. The code also uses doc aliases</a> so that you can still
find things via the traditional C names.</p>
</li>
</ul>
What about portability?</h3>
Rustix isn't a fully-fledged portability layer; it doesn't support Windows and
some of its APIs are OS-specific, and even OS-version-specific. Rustix doesn't
do non-trivial emulation of features, and prefers to let higher-level layers
provide that kind of portability when needed.</p>
Many POSIX-ish APIs are very low-level, and not a suitable abstraction layer
for efficiently implementing on different kinds of platforms. See Rust's
standard library for an illustration of this: For example,
std::fs::Metadata</code></a> is more abstract than struct stat</code></a>, and this gives
platforms more options when implementing functions like
std::fs::File::metadata</code></a>.</p>
Other interesting cases are epoll</code></a> and io_uring</code></a>. Rustix doesn't yet have
APIs for these, and it's an open question whether it's worth it. These APIs are
very low-level, they have complex file descriptor ownership, and they are most
often wrapped in higher-level APIs. It's not yet clear whether I/O safety makes
sense at the abstraction level of these APIs, or whether they should just
continue to use RawFd</code> and introduce safety at higher levels of abstraction.</p>
What about close</code>?</h3>
It's called drop</code></a> 😃.</p>
Rustix doesn't currently have a close</code> function in its public API. It would be
straight-forward to add it, and the signature would either have an OwnedFd</code>
argument or a T: IntoFd</code> generic argument, to express that the file
descriptor ownership is being consumed. However it'd also be redundant for most
users. OwnedFd</code> calls close</code> in its Drop</code> implementation, so in most cases,
the way to close a file descriptor is to simply drop it.</p>
(Drop</code> doesn't have a way to report errors, so this may not be the last word
on this subject, but this is a separate and much more complex topic involving
things like NFS in async mode.)</p>
How complete is rustix?</h3>
It has everything that cap-std</a> and wasmtime</a>'s WASI implementation need,
as well as a few other projects, though it doesn't have everything. If you find
it's missing something you need, please file an issue!</p>
Where does I/O safety go from here?</h3>
Now that the I/O Safety RFC</a> is merged, my next step in this wing of the story
is to prepare a PR adding OwnedFd</code></a>, BorrowedFd</code></a>, and friends to the Rust
standard library. Once that's done, I'll update the io-lifetimes crate</a> to use
the new standard library types and traits instead of defining them itself.</p>
io-lifetimes defines more functionality than will be in the initial standard
library PR, such as views</a>, from_into_fd</code></a>, and the
AsFilelike</code></a>/AsSocketlike</code></a> Windows/Unix portability layer, so it'll remain
useful for some time, but it'll get thinner as the standard library takes over
more functionality.</p>
And for the Rust ecosystem as a whole, the I/O safety RFC outlines
the path forward</a>.</p>


Introducing cap-std, a capability-based version of the Rust standard library
2021-06-14T00:00:00+00:00
Introducing cap-std</code></h2>
cap-std</code> is a project to create capability-based versions of Rust standard
library and related APIs.</p>
Capability-based here means that the APIs don't access files, directories,
network addresses, clocks, or other external resources implicitly, but instead
operate on handles that are explicitly passed in. This helps programs that work
with potentially malicious content avoid accidentally accessing resources other
than they intend, and does so without the need of a traditional process-wide
sandbox, so it can be easily embedded in larger applications.</p>
Background</h2>
Some of the most devious software bugs are those where the code looks like it
does one thing, and usually does that thing in practice, but sometimes, under
special circumstances, does something else. Here's a simple example using
Rust's filesystem APIs:</p>
    </span>fn </span>hello</span>(</span>name</span>: &Path) -> Result<()> {
</span>        </span>let</span> tmp = </span>tempdir</span>()?;
</span>        fs::write(tmp.</span>path</span>().</span>join</span>(name), "</span>hello world</span>")?;
</span>    }
</span></code></pre>
The expected behavior of this function is to write "hello world" to a file within
a temporary directory. The code looks like it will do this. And indeed, it will
usually</em> do this. But if the path passed in is ../../home/me/.ssh/id_dsa.pub</code>,
then the behavior of this function could be to corrupt the user's ssh public key 😲.
That's... not remotely within what we said the expected behavior is. It usually
doesn't do that, but under the right circumstances, it could.</p>
And since name</code> is just a string, if the string is computed in a way that could
be influenced by an attacker, the right circumstances could easily be made to
occur in practice.</p>
The cap-std</code> project</a> provides Rust crates with lightweight ways to avoid such
problems. In particular, the cap-std</code> crate</a>'s Dir</code> type represents a
directory, with methods</a> corresponding to Rust's std::fs</code> functions, for
opening and working with files within the directory, that ensure that all paths
stay within that directory. For networking, the Pool</code></a> type represents a set of
network addresses, ensuring that all network accesses made through the API
are to addresses in the pool.</p>
In contrast to conventional sandboxing, cap-std</code> doesn't have any global state,
so using it in one part of an application doesn't require using it in the rest of
the application. Library crates can use cap-std</code> internally without imposing any
sandboxing constraints on their users.</p>
What can Dir</code> do?</h2>
cap_std</code> is just a library, so by itself, it isn't a sandbox for arbitrary
Rust code—it can't prevent arbitrary Rust code from using std::fs</code>'s
path-oriented APIs. Instead, it protects against malicious content</em>, when
filesystem paths can be influenced by untrusted inputs, and malicious
concurrent modifications</em>, when another program running at the same time
has the ability to remove, rename, or create files, directories, symlinks,
or hard links in ways that could cause a program to inadvertently access
unintended resources.</p>
Revisiting our example above, with cap-std</code> we might write:</p>
    </span>fn </span>hello</span>(</span>name</span>: &Path, </span>tmp</span>: &Dir) -> Result<()> {
</span>        tmp.</span>write</span>(name, "</span>hello world</span>")?;
</span>    }
</span></code></pre>
In this code, if the passed-in path uses ..</code> to access directories outside of
the one passed in, tmp.write</code> returns an error.</p>
One key difference from before is that instead of creating the temporary
directory itself, this function requests a directory be passed in. The Dir</code>
type here serves as a "vocabulary" type, allowing the function to declare that
it wants a directory to be passed in, and that it intends to access resources
within that directory, rather than accessing arbitrary locations in the
filesystem. These kinds of declarations can help reduce the
reasoning footprint</a> of a function call.</p>
The Dir</code> crate also makes it much easier to write this code robustly. There's
no need to think about ..</code> or absolute paths at the application level, and no
need to handle symlinks specially, which with the Rust standard library today
isn't even possible to do robustly without platform-specific code</a>.</p>
It also gives callers increased control. The caller gets to choose how and
where to create the directory, and when to remove it. Callers could choose to use
something like cap-tempfile</code></a>'s tempdir</code> function to easily create a
temporary directory in a conventional location and automatically remove it
afterwords, however they could also opt to create the directory somewhere
else and manage it manually.</p>
Note that Dir</code> is passed by immutable reference, even though it's being used
to mutate external filesystem state. This follows Rust's conventions, for
example in std::fs::File::set_len</code></a>, and it reflects an underlying truth
about filesystems. &mut</code> in Rust is sometimes called an "exclusive" reference,
because when someone has a &mut</code>, they're the only one which can access the
underlying object. However, this is generally not a safe assumption when
working with filesystem objects, because other programs could concurrently
access or even mutate files or directories without Rust's type system having
any say in the matter. Consequently, it makes sense to think of filesystem
state as being external to the program, with File</code> and Dir</code> objects being
just handles that are themselves typically immutable.</p>
Dir</code> can also be combined with other security techniques. In a project which is
written to carefully avoid using untrusted paths, it can add an extra layer of
defense in depth.</p>
And in the Wasmtime</a> project, the next step we described earlier</a> is now
finished, and we're now using cap-std</a> in combination with with our WebAssembly
sandbox to implement WASI, providing sandboxed access to system resources.</p>
A simple example</h2>
The main pattern for filesystem operations using the cap-std</code> crate is to
obtain a Dir</code> and use methods on it, which closely resemble the functions in
std::fs</code>.</p>
One of the ways to obtain a Dir</code> is to use the cap-directories</code></a>
crate to request a Dir</code> for a standard directory (similar to the
directories-next</code></a> crate, but returns a Dir</code> instead of a Path</code>).
For example, to obtain the data directory for an example program:</p>
    </span>let</span> project_dirs =
</span>        cap_directories::ProjectDirs::from(
</span>            "</span>com.example</span>",
</span>            "</span>Example Organization</span>",
</span>            "</span>`cap-std` Key-Value CLI Example</span>",
</span>            cap_directories::ambient_authority()
</span>        )
</span>    };
</span>
</span>    </span>let</span> data_dir = project_dirs.</span>data_dir</span>().</span>unwrap</span>();
</span></code></pre>
Then in place of fs::read</code> and fs::write</code> to read and write files, one
can use data_dir</code> here to do data_dir.read(key)</code> and
data_dir.write(file_name, value)</code>.</p>
Note the use of the ambient_authority()</code> function here, which is a no-op that
returns an instance of the opaque AmbientAuthority</code></a> type, and serves to mark a
place in the code where ambient authority is being invoked. cap-directories</code>,
and related crates have an overall invariant that functions don't create their
own absolute filesystem paths, and always rely on resources being passed in as
handles. Functions which don't uphold this invariant, such as
cap_directories::ProjectDirs::from</code></a>, take an AmbientAuthority</code> argument to
advertise their ability to open resources given only a string.</p>
This makes it easy to search a codebase to find all the places where a
non-sandboxed cap-std API is being used. It can also be scanned for with
Clippy using a clippy configuration file</a>.</p>
To see all this put together in a complete example, see
the kv-cli example</a>
in the cap-std</code> repository. This program implements a simple key-value store,
using filesystem paths as keys, and using cap-std</code> ensures that it only
accesses paths within its own data directory. Attempts to escape the directory
with ..</code> fail gracefully. This is true even if a concurrently running program
renames directories on the path or changes symlinks—something that's very hard
to get right using std::fs</code> APIs.</p>
$</span> cargo run</span> --quiet --example</span> kv-cli color green
</span>$</span> cargo run</span> --quiet --example</span> kv-cli color
</span>green
</span>$</span> cargo run</span> --quiet --example</span> kv-cli temperature cold
</span>$</span> cargo run</span> --quiet --example</span> kv-cli temperature
</span>cold
</span>$</span> cargo run</span> --quiet --example</span> kv-cli color
</span>green
</span>$</span> cargo run</span> --quiet --example</span> kv-cli /etc/passwd
</span>Error:</span> a path led outside of the filesystem
</span>$</span> cargo run</span> --quiet --example</span> kv-cli ../../../secret_cookie_recipe.txt
</span>Error:</span> a path led outside of the filesystem
</span></code></pre>
Another useful crate is cap-tempfile</code></a>, which creates temporary directories
and provides a Dir</code> to access them.</p>
It's also possible to create a Dir</code> by opening a raw path, using
Dir::open_ambient_dir</code>. Note that this function takes an AmbientAuthority</code>
since it does not uphold the sandboxing invariant that the rest of the API
does.</p>
A real-world example</h2>
Web servers often need to serve files from a given directory, and it'd be nice
to have a guarantee that they don't accidentally stray outside that directory.</p>
tide-native-static-files</a> is a fork of a real-world Web server project built
on the Tide framework, ported to use cap-std instead of directory paths.</p>
The port is very straightforward, mostly consisting of passing around a Dir</code>
instead of a string holding a base directory name. And in many cases, working
with a Dir</code> is actually simpler than working with a string. The complete set
of changes needed for this port can be seen here</a>.</p>
Implementation Landscape</h2>
One of the reasons that Rust doesn't already have a Dir</code> type, when it does have a
File</code> type, is that popular OS filesystem APIs don't make this as efficient
or idiomatic as just using paths to name directories. However, this is changing.</p>
One of the inspirations for cap-std</code> is the CloudABI project</a>, which among
other things developed a technique of using a sequence of openat</code> calls to
emulate path lookup in userspace in a way that's robust in the face of concurrent
renames. cap-std</code> uses a variant of this technique, optimized to use fewer
intermediate system calls, to implement a portable sandboxed path lookup algorithm.</p>
And, Linux recently</a> added a system call, openat2</code>, which has the ability to
restrict path lookup</a> so that it stays within a given directory, which is exactly
the behavior we want here. It doesn't require a process-wide mode, and it avoids
the overhead of doing multiple system calls. cap-std</code> uses this in place of its
portable algorithm whenever it can. On systems which support openat2</code>, most
functions in the API perform only one or two system calls.</p>
Linux and other operating systems are also exploring adding more such features,
and as these features become available, it will become increasingly practical to
not just implement a Dir</code> type, but to implement it with WASI-style sandboxing
protections built in.</p>
Philosophy</h2>
cap-std</code> came about because we were looking to generalize the filesystem
sandboxing techniques we were using in Wasmtime's WASI implementation to make
them more broadly applicable, and we were particularly inspired by
async-std</code>'s philosophy</a>:</p>

the best API is the one you already know.</p>
</blockquote>
Rust already has a standard library API. It's very good overall, and a lot of
care has gone into ensuring that it's implementable on many platforms. It's
used by a lot of code, and well known to a lot of developers.</p>
cap-std</code> is an approach that takes advantage of this. Developers who know std</code>
can easily learn cap-std</code>. Applications using std</code> can be ported to cap-std</code>, with
the main concern being about how to ensure that directory handles are available
to all the places that need them, rather than with dealing with differences
in the API or in filesystem behavior.</p>
The close alignment between cap-std</code> and std</code>, combined with the close
alignment between async-std</code></a> and std</code>, also make it straightforward to do
both at the same time, producing cap-async-std</code></a>.</p>
If you're familiar with using std::fs</code>, you should be familiar with cap-std</code>'s
APIs without any surprises. Similarly, if you're familiar with async-std</code>,
cap-async-std</code>'s APIs should work as expected.</p>
Current status</h2>
Cap-std works on Linux, macOS, FreeBSD, Windows, and more, with stable Rust. On
Linux, cap_std::fs</code></a> is optimized to use new system calls including openat2</code>,
when available, which significantly reduces the sandboxing overhead.</p>
Support for compiling to WASI is under active development.</p>
Speaking of WASI...</h2>
The sandboxing performed by cap-std</code> is the same as what's provided by WASI APIs.</p>
While cap-std</code> is designed so that it can be used as a library within otherwise
unsandboxed native applications, WASI applies the same kind of sandboxing to all
filesystem accesses, so that it serves as an extension to the core WebAssembly
sandbox.</p>
This means that when the cap-std</code> library is compiled for the WASI platform, it
will be able to bypass its own sandboxing techniques and simply call into the WASI
system calls directly, achieving smaller code size and tighter integration with
the underlying WASI platform.</p>
The Future!</h2>
We're continuing to add more testing, fuzzing, and optimization. A port to WASI
is underway.</p>
We're also starting to think about extending the capability-based model to more
parts of Rust's API. The most obvious next step is std::net</code>, which is in a very
early state right now, but this is a space we're thinking about for the future! Other
areas that may be interesting include std::env</code> for information passed in by the
host environment, std::process</code> for launching sandboxed processes, and anything else
that allows programs to interact with the outside world.</p>
And as we're doing with cap-directories</code></a> and cap-tempfile</code></a>, we're also interested
in ways that we can do more than just translate the standard library API into a
capability-based model, but also make the capability-based model easy to use.</p>


First-Class I/O
2021-04-06T00:00:00+00:00
@withoutBoats makes this observation about Rust in "Notes on a smaller Rust"</a>:</p>

Pure functional programming is an ingenious trick to show you can code
without mutation, but Rust is an even cleverer trick to show you can just
have mutation.</p>
</blockquote>
A particular aspect I'd like to explore here is: Can we apply this
observation to I/O?</p>
Haskell also has an ingenious trick to do first-class I/O without mutation. So,
can we have first-class I/O in Rust that just uses mutation?</p>
What is First-Class I/O?</h2>
As in Haskell, first-class I/O in Rust would mean functions that do I/O would
do so through values which can be passed around the program as arguments or
return values:</p>
fn </span>do_some_io</span>(</span>f</span>: &File) -> io::Result<()> {
</span>    ...
</span>}
</span></code></pre>
File</code> here is an example of a first-class value which represents a resource
that supports I/O. But unlike Haskell, instead of using monads, File</code> here just
has side-effecting operations like read</code> and write</code>.</p>
Of course, Rust already has lots of these kinds of types, including in the
standard library with types like File</code> and TcpStream</code>. And a lot of Rust
code already follows this pattern. Strictly speaking, many of the operations
don't take &mut</code> references, but these types conceptually use
interior mutability</a>, which isn't unique to I/O.</p>
The remaining piece that Haskell has that Rust doesn't here is that in Haskell,
all</em> I/O is done through values which are passed around through the program.
Any function which does I/O says so in its signature. In Rust, when a function
has a File</code> argument in its signature, you know it's going to do I/O using that
File</code>, but a function which doesn't have File</code> or any other I/O type might
still access files.</p>
Of course, monads in Haskell have many other uses too. This blog post is just
looking at I/O operations in function signatures, which has implications for how
two otherwise unrelated pieces of code interact, as well as for one piece of
code can redirect the I/O of another.</p>
Example: Stdout</h2>
An example in Rust of code that uses global I/O is std::io::stdout</code>. Any
code can call this and obtain a Stdout</code> value that can do I/O without having
any mention of it in the surrounding function's signature. Stdout</code> uses a
builtin mutex, even though many use cases don't need that, and is a common
performance pitfall. Diligent users can of course use StdoutLock</code> to reduce
the performance impact in some cases, but what if we could eliminate the
builtin mutex altogether?</p>
What if stdout was a value that a program would acquire once, and then pass
around to all functions that want to print to it? Using Rust's usual ownership
and borrowing rules, it wouldn't need a mutex for many use cases. And users
could of course still explicitly wrap it in a Mutex</code> or similar in cases where
they really want shared access to it, just like anything else in Rust.</p>
The io-streams</code> crate</a> has an implementation of this. The
StreamWriter::stdout</code></a> function returns an output stream which writes to
the process' standard output. It implements the standard Write</code></a> trait
so it's easy to use, and it conceptually owns its resource, so it doesn't
need a builtin mutex.</p>
Behind the scenes, this function actually acquires a StdoutLock</code> to prevent
accidental mixing of std::io::stdout</code> usage with StreamWriter::stdout</code>
usage, to uphold its exclusive ownership assumption.</p>
Example: Files</h2>
As another example, any code can do File::open(...)</code> and pass it a string,
to open any file in the process' filesystem namespace, without declaring it
in a function signature.</p>
This means that if you want to run piece of code that does this in a different
directory, the only way to do so is to create a new process, with a new
filesystem namespace. This is a very heavy-weight operation, both in terms of
performance and memory usage, but also in terms of portability and complexity.</p>
Some codebases have a convention of having a "root" path that is passed in
that everything is relative to, which helps, but doesn't enforce that all
paths are relative to the root, or that paths don't lead outside the root
using ..</code>.</p>
The cap-std</code> crate</a> has a Dir</code></a> type, which represents a directory,
which can make filesystem access a first-class part of a function's signature,
and allow callers to specify a different directory for the I/O to happen in.
And, it also performs sandboxing, ensuring that paths relative to the Dir</code>
stay within the Dir</code>.</p>
Filesystems are effectively pools of aliased mutable state, and with relatively
weak synchronization primitives. While cap-std</code> doesn't address all the
problems that can arise from this, it can be one tool for helping manage
this state with standard Rust idioms.</p>
Tangent: Rethinking "pure" functions</h2>
Rust doesn't have a way to declare functions as "pure", having no side effects.</p>
fn </span>this_is_pure</span>(</span>x</span>: </span>i32</span>, </span>y</span>: </span>i32</span>) -> </span>i32 </span>{
</span>   x + y
</span>}
</span></code></pre>
(As an aside, this function could panic on overflow; in what follows, I assume
"pure" permits panics.)</p>
Features to enable this have been proposed</a> a few times, but they haven't been
added to the language, in part because the need for such features tends to be
lower in Rust than other languages. From an optimizer perspective, this
property can often be inferred, at least in simple cases.</p>
And from a programmer perspective, programmers don't need "pure" to know whether
arguments are mutated or not, because in Rust, these things are already
declared, reliably, in the signature. The presence or absence of &mut</code> or
types with interior mutability tells you everything you need to know about
which arguments could be mutated. As such, much of what "pure" would mean would
be redundant with information which is already there:</p>
/// Can you guess which state this function mutates?
</span>fn </span>increment</span>(</span>x</span>: &</span>mut i32</span>) {
</span>   </span>// You're right!
</span>   *x += </span>1</span>;
</span>}
</span></code></pre>
The things that aren't covered by the signature are global mutable state and
I/O. Global mutable state tends to be less important in Rust than other
languages because Rust pretty strongly discourages global mutable state.</p>
However, Rust doesn't outright prohibit global mutable state, and doesn't
really discourage global I/O, which is present even in the standard library.</p>
So instead of a "pure", that prohibits all mutations, the more interesting
property would be what I'll call "explicit", which would mean "no global
mutable state or I/O". When used on a function with no &mut</code> or
interior-mutable types in the signature, "explicit" would be the same as
"pure", and could be declared as "pure" to optimizers. However, "explicit"
could be used in functions that do have mutable arguments too, where it
would indicate that those are the only things that are mutated.</p>
/// All mutations and I/O are accounted for!
</span>fn </span>this_is_explicit</span>(</span>f</span>: &File, </span>x</span>: &</span>mut i32</span>) -> io::Result<()> {
</span>    writeln!(f, "</span>hello world</span>")?;
</span>    *x += </span>1</span>;
</span>    Ok(())
</span>}
</span></code></pre>
The "explicit" property would also serve to indicate a lack of
ambient authority</em>, from a capability-oriented security</a> perspective.</p>
Would it make sense to add an "explicit" keyword to Rust then? Probably not
as such; it's likely that the majority of functions in Rust would qualify as
"explicit", so adding it as a function attribute would add a lot of clutter.
A more ergonomic approach might be to give crates a way to declare that all
their functions are explicit, with a way to opt out for individual functions.</p>
A possible direction for future exploration would be to use the
rustc driver API</a> to create a custom static analysis tool that could recognize
some form of syntax for this and then check that "explicit" functions don't
accidentally call "non-explicit" functions without explicit overrides, much
like unsafe</code>.</p>
Conclusion</h2>
First-class I/O points to a way of thinking about software where "the machine"
it's running on or "the process" it's running in aren't the focus. Instead of
ad-hoc conventions for coordinating access to a shared filesystem namespace or
other process-associated resources, code using first-class I/O can pass values
around to manage its I/O resources similar to how it already manages other
program resources.</p>
First-class I/O can be a useful concept to apply broadly, such as how
capability-oriented security</a> helps enable shared-nothing linking</a>. It can
also be useful to apply incrementally, such as how the io-streams</code> crate</a>
or the cap-std</code> crate</a> can help parts of a program cooperate with each other
efficiently and idiomatically.</p>
Thanks to Pat Hickey</a> and
Luke Wagner</a> for feedback on this post!</em></p>


Canonicalization
2018-10-22T00:00:00+00:00
Canonicalization and canonical forms are one dimension of organizing the
work of an optimizing compiler.</p>
Intro</h2>
A lot of code constructs can be written in multiple ways. For example:</p>
   x + 4
</span>   4 + x
</span>   (x + 2) + 2
</span></code></pre>
Canonicalization</em> means picking one of these forms to
be the canonical form</em>, and then going through the program and
rewriting all constructs which are equivalent to the canonical
form into the canonical form.</p>
In the case of add, it's common to pick the form that has a constant
on the right side, so we'd rewrite all
these constructs to x + 4</code>.</p>
Why is canonicalization useful?</h3>
The goal of canonicalization is to make subsequent optimizations more
effective</em>. This is a key point, and we'll get into some of the subtleties
below. But there are a lot of cases where it's just obviously a good thing
to do. It means that subsequent optimizations that look for specific patterns of code
only have to look for the canonical forms, rather than all forms.</p>
Another way of saying this is, having a canonicalization pass is a
way of factoring out the parts in a compiler that know all the different
forms x + 4</code> could take, so that most optimization passes don't have
to worry about this. They don't have to look for 4 + x</code>, because they
can assume that looking for x + 4</code> covers that. Handy!</p>
How do we choose a canonical form?</h3>
Sometimes it's easy. The canonical form for 2 + 3</code> is 5</code>, because
that's clearly simpler in every possible way.</p>
Sometimes it's an arbitrary choice. It often doesn't matter than much
whether one picks 4 + x</code> over x + 4</code>, but it is helpful to pick one
or the other, so sometimes it's just human aesthetics.</p>
It's tempting to pick whatever form would be fastest, or optimal</em>, on
the target machine. And indeed, sometimes what's fastest aligns with what's
simplest. 2 + 3</code> canonicalizing 5</code> is typically such a case. But
sometimes it doesn't.</p>
Yeah so what about x * 2</code>...</h3>
x * 2</code> is equivalent to x + x</code>; which of these should be
the canonical form? It might seem like we might want to say: pick whatever's
optimal for the target architecture. Addition is generally
faster than multiplication, so that would suggest we pick x + x</code> as
the canonical form.</p>
But, x + x</code> can actually make things harder for subsequent
optimizations, because it means that now x</code> has multiple uses. Having multiple uses
makes some optimizations more complex -- in terms of the dependence graph,
this is a DAG rather than a tree, and trees are generally simpler to
work with. So maybe x * 2</code> is actually a better canonical form, even if it's a worse optimal form.</p>
That said, we might consider canonicalizing this to x << 1</code>, which has only one use of x</code>,
and has the nice property of making it as obvious as
possible that the least significant bit of the result is zero. That way,
we don't have to have as much random knowledge of multiplication by
powers of two strewn throughout the compiler.</p>
Efficiency on the target machine still matters, but we can defer thinking
about that until codegen, where it's no longer as important to enable
subsequent optimizations. At that point, we're going to start caring about picking between +</code>,
*</code>, and <<</code> based on which one executes fastest.</p>
The basic philosophy
of canonicalization says that canonical forms should be translated into
optimal forms toward the back of the compiler, after all mid-level
optimizations which benefit from canonical form are done. It's also
worth noting that codegen itself benefits from having the
code coming into it be in canonical form, so that it doesn't have to
recognize all the ways to write x << 1</code>, and can just recognize one
pattern for that and emit the optimal code for it.</p>
This is often a source of confusion: Is canonicalization
the same as optimization? It's often done as part of the "optimizer", and
many of the things it does produce more optimal code directly. But
ultimately, in its purest form, canonicalization just focuses on
removing unnecessary variation so that subsequent optimizations can
be simpler.</p>

Canonical form, canonical form

Canonical form hates optimal form

They have a fight, canonical wins

Canonical form…</p>
</blockquote>
(sung to the tune of "Particle Man" by They Might Be Giants)</p>
Sometimes it's ambiguous: redundancy elimination</h3>
Is redundancy elimination a canonicalization or an optimization?</p>
It's certainly simpler to compute a given value once and reuse the value, rather than compute it twice.
But does that aid subsequent optimizations? It depends.</p>
A case where it does aid subsequent optimizations is when it eliminates redundant memory accesses.
That way, it's essentially saying that no later passes have to even ask what the dependencies are
for a given memory access, because the memory access has been eliminated.</p>
A case where it doesn't is where it can take expression trees where everything has a single use and
give some values multiple uses. Some kinds of optimization passes are harder to do on DAGs than on
trees, so this can result in pessimizations in some cases, depending on what kinds of things the
rest of the compiler is doing.</p>
However, we typically do think of redundancy elimination as being a canonicalization.
It's trivial to convert multiple-use values into single-use values by duplicating code,
while going the other direction requires some analysis.</p>
Even more ambiguous: inlining</h3>
Inlining can act like canonicalization, especially in cases where the inlined function body can
be optimized away. However, thinking of inlining in terms of canonicalization doesn't lead to a natural threshold. Should we maximally
inline everything as far as possible? Or should we do the reverse and maximally outline and deduplicate
outlined functions? Neither extreme is particularly practical, so most compilers use heuristics in
practice rather than having a rigid definition of canonical form relative to calls.</p>
That said, even though there's no clear boundary, we can imagine a rough guideline.
Think of old-school C code, written at a time when "C is a portable assembly language"
was more true than it is today, where calls that were important to inline were
written as macros. In many ways, one of the jobs of compilers for higher-level languages
is to compile them down to roughly this level, so that they can be optimized using
optimization techniques which work well at this level. In theory, we could define the
task of a canonicalizing inliner to just be to inline code down to what it would have
looked like if that same code had been written in old-school C (at least with
respect to inlining). That's not very pure, and it's difficult to precisely describe,
but it's an intuitive and relatively practical compromise between extremes.</p>
Canonical form isn't just for arithmetic expressions!</h3>
Everything in an IR may be subjected to canonicalization.</p>
For example, a control flow canonicalization might involve sorting the
basic blocks of a program into Reverse Post-Order (RPO). Doing this is
a simple way to ensure that the code is optimized the same regardless
of how the user organizes the code inside their functions.</p>
Dead code elimination is also a kind of canonicalization. The canonical
form for dead code is no code.</p>
In aggressive loop-transforming compilers, another form of canonicalization
is to maximally fission loops into as many parts as possible, and then
assume subsequent passes will fuse them back together into optimal
loops that effectively utilize available registers and cache space.</p>
Canonicalization as compression</h3>
It's often the case that smaller forms are preferred over longer
forms, so canonicalization tends to make code smaller, making it a
form of compression. Also, since it effectively reduces non-essential
entropy, it can make subsequent general-purpose compression more
effective as well.</p>
If we could conceptually perform all theoretically possible
canonicalizations on a program, we'd end up with something related to
its Kolmogorov Complexity</a>
(it may not be identical, since canonicalization puts the needs of
subsequent optimizations first, rather than absolute compression).
Maximal canonicalization is frequently impossible in practice, because
of the halting problem, but also because even in cases where it's
theoretically possible, it can require impractical amounts of computation.</p>
That said, it is pretty fun to think that for any given program, there
is a theoretical "maximally canonical form" for that program, that
all equivalent ways of writing that program could be reduced to. It
is tempting to think of this as a kind of pure essence of the program.</p>
Excessive canonicalization</h3>
Canonicalization discards inessential information.
However, sometimes that information can be useful to preserve.</p>
An example arises in instruction scheduling: Say a user writes code like</p>
  x = a + b;
</span>  y = c * d;
</span></code></pre>
Assuming there's no aliasing going on here, there's no reason why
one of these statements has to be ordered before the other. Their
order in the user's source code is inessential information.
"Sea of nodes" style compilers may canonicalize to the point where
there is no inherent ordering between these two statements.</p>
The compiler backend ultimately has to produce machine code, which
on conventional architectures requires the compiler to pick some</em>
ordering. Compilers can be pretty smart, and can take into
consideration many things, such as available execution resources
before and after these statements to know what order a 
CPU would prefer to see them in. However, as smart as they can be,
compilers can't always find the optimal answers. Optimal instruction
scheduling is NP complete, but also, it may come down to runtime
factors that ahead-of-time compilers don't have. And 
CPU hardware performance characteristics aren't always fully
documented.</p>
"Do No Harm"?</h4>
Most software has no idea how it'll be mapped on to CPU pipelines.
But some does. And it's these cases where getting the mapping right is
most important.</p>
On such software, optimizations that make actual improvements are fine, however it can be
important that compilers not make anything worse</em> than if they
had translated the code naively. Aggressively canonicalizing compilers
have a risk that they will throw away information and at the end
reconstruct a form which is worse than if they had just simply translated the code as it was written.</p>
This, along with the ambiguous cases above, suggests that canonicalization
be used in practical rather than rigid ways.</p>

The theoretical shape of optimization.</h2>
A compiler typically starts with human-written source code.</p>
Typical human-written code
follows various human-oriented sensibilities. Different humans may have different aesthetic sensibilities,
and this can lead to writing the same code in different ways. But of
course this isn't interesting to optimizers.</p>
So the first then we typically do when the code hits the optimizers is
to start canonicalizing. Throw away useless fluff that humans
imbue their code with.</p>
Canonicalizing optimizations are cascading; often doing one canonicalization
will enable more. Folding an expression to a constant may allow other
expressions to be folded to constants. Replacing a load by forwarding a value
through a prior store may create a direct edge between two expression trees
and introduce opportunities to simplify further.</p>
So ignoring the practical concerns we mentioned above, we can imagine a
theoretical compiler that does all the canonicalization it knows how to do
up front. And while we don't realistically approach Kolmogorov Complexity
levels, we do lift the program closer to what we might think of as its
essence, the simplest form that does what it needs to do.</p>
And then, the compiler can begin to optimize, rewriting canonical forms into
optimal forms, as it lowers the code all the way down to assembly code.</p>
For practical reasons, it isn't always possible to build compilers in terms
of a purely canonicalizing phase and a purely optimizing phase, however this
can be a useful reference point for understanding compilers in practice.</p>

sunfishcode's blog

Writing into uninitialized buffers in Rust

Errors and Pipelines, a diversion

Thinking about streams in WASI

Errors from `close`

what</h2> There's something of a longstanding debate.</p>

seriously?</h2> So back in the day, when the Network File System (NFS) was first released, it was a triumph of Unix design. The mighty filesystem could abstract over everything</em>, even the network.</p>

FUSE</h2> And besides, it's not just NFS. Or other network filesystems that do similar things.</p>

Capabilities and Filesystems

Bridging between source languages, in Wasm

What are the options?</h2> Looking outside of Wasm at how other systems provide for cross-language interfaces, there are roughly three different categories of approaches:</p>

WASI 0.2 Launched

Wrapping it up</h2> WASI 0.2 is a major milestone that many people have contributed to, and it's now launched! It's an API, but it also represents a new way to define APIs for Wasm.</p>

(renamed)

Eyra does the impossible

Embrace the Kinda

The Filesystem Namespace

What does Everything Is A File do?

Mustang update

Measuring System Interface Complexity

So what?</h2> So a few things.</p> First, counting the number of direct syscalls is a very</em> unreliable way to measure OS design elegance.</p>

What is a Capability?

Is Everything A File?

Fork versus Elegance

What is a World?

What is a Wasm component?

Wrap up</h2> A Wasm component is a deployable</em> unit</em> of software</em>!</p> There's a lot more that can be said about components; check out the links in the intro, and follow me for future explorations!</p>

No Ghosts!

Ghosts complicate static analysis</h4> Being unable to know where ghosts are going and where they're coming from makes it difficult and often impossible to answer questions such as:</p>

How to smell a ghost</h2> There are some common signs that a ghost may be present.</p>

Bugs in Hello World

Porting Rust's std to rustix

Context Brainstorming

Getting closer: unsafe</h3> This post is all about contexts though, so let's see if we can use them to fix that problem too:</p> new_unsafe</code> - an automatic context representing the ability to introduce new unsafe contexts. This corresponds to the ability to write unsafe { ... }</code>.</li>

The Spectrum from Namespaces to Values

what</h2>
There's something of a longstanding debate.</p>

`Wrapping it up</h2> WASI 0.2 is a major milestone that many people have contributed to, and it's now launched! It's an API, but it also represents a new way to define APIs for Wasm.</p>`

Wrap up</h2>
A Wasm component is a deployable</em> unit</em> of software</em>!</p>
There's a lot more that can be said about components; check out the links in the intro, and follow me for future explorations!</p>