Rust at scale: packages, crates, and modules

Published:


Good decisions come from experience. Experience comes from making bad decisions.

I was lucky enough to see how the Internet Computer (IC) Rust codebase has grown from a few files to almost 350,000 lines of code within just about two years. The team learned that code organization that works just fine for relatively small projects might start dragging you down over time. In this article, we shall evaluate code organization options that Rust gives us and look at how to use them well.

Dramatis Personae

Rust terminology might be confusing for newcomers. I believe one of the reasons for that is that the term crate is somewhat overloaded in the community. Even the first edition of the venerable The Rust Programming Language book contained the following misleading passage

Rust has two distinct terms that relate to the module system: ‘crate’ and ‘module’. A crate is synonymous with a ‘library’ or ‘package’ in other languages. Hence “Cargo” as the name of Rust’s package management tool: you ship your crates to others with Cargo. Crates can produce an executable or a library, depending on the project.

Wait a minute, `library` and `package` are different things, aren't they? Mixing up these two concepts can lead to a lot of frustration, even if you already have a few months of experience with Rust under your belt. Tooling conventions contribute to the confusion as well: If a Rust package defines a library crate, cargo derives the library name from the package name by default (you can override the library name to be completely different, but please don't).

Let us become familiar with the code organization concepts we will be dealing with.

Module
A Module is a language construct that acts as the basic building block of code organization within a crate. A module is a container for functions, types, and nested modules. Modules also specify the visibility for all the items that they contain or re-export.
Crate
A Crate is the basic unit of compilation and linking. Crates are also part of the language (crate is a keyword), but you usually don't mention them much in your source code. There are two main types of crates: libraries and executable files.
Package
A Package is the basic unit of software distribution. Packages are not part of the language, so you will not find them in the language reference. Packages are artifacts of the Rust package manager, Cargo. Packages can contain one or more crates: at most one library and any number of executables.

Modules vs Crates

In this section, I will use the terms "crate" and "package" almost interchangeably, assuming that most of your crates are libraries. As you remember, we can have at most one library crate per package.

When you factor a large codebase into components, there are two extremes you can go to: (1) to have a few big packages with lots of modules in each package, or (2) to have lots of tiny packages with just a bit of code in each package.

Having few packages with lots of modules definitely has some advantages:

Sounds like modules are a clear winner. In the ideal world where arbitrary-sized Rust crates compile instantly, turning the whole repository into one huge package with lots of modules would be the most convenient setup. The bitter reality though is that Rust takes quite some time to compile, and modules do not help you shorten the compilation time:

It all boils down to the tradeoff between convenience and compilation speed. Modules are very convenient, but they don't help the compiler do less work. Packages are less convenient, but they deliver a better overall development experience as the code base grows.

Advice on code organization

As usual, do not follow this advice blindly. Check if it makes your code structure clearer and your compilation times shorter.

Split dependency hubs.

There are two types of dependency hubs:

A small subgraph of the Internet Computer project package dependency graph. types and interfaces are type-one dependency hubs, replica is a type-two dependency hub, test-utils is both a type-one and a type-two hub.

types
types
interfaces
interfaces
consensus
consensus
p2p
p2p
messaging
messaging
execution
execution
replica
replica
test-utils
test-utils
dev-dependency
dev-dependency
dependency
dependency
Viewer does not support full SVG 1.1

The main reason why dependency hubs are undesirable is their devastating effect on incremental compilation. If you modify a package with lots of reverse dependencies (e.g., types), cargo has to re-compile all those dependencies to check that your change makes sense.

The only way to get rid of dependency hubs is to split them into smaller packages.

Sometimes it is possible to eliminate a dependency hub. For example, test-utils is a conglomeration of independent utilities. We can group these utilities by component they help to test and factor them into multiple <component>-test-utils packages.

More often, however, there is no way to get rid of a dependency hub entirely. Some types from types are pervasive. The package that contains those types is doomed to be a type-one dependency hub. The replica package contains the main function that ties everything together so it has to depend on all components. The replica is doomed to be a type-two dependency hub. The best you can do is to localize such hubs and make the code inside relatively stable.

Consider using generics and associated types to eliminate dependencies.

Among the first few packages that appeared in the IC codebase were: types, interfaces, and replicated_state (that package defines data structures that represent the state of a single subnet). But why do we even need the types package? Types are an integral part of the interface, why not define them in the interfaces package as well?

The problem is that some interfaces operate on instances of ReplicatedState. And the replicated_state package depends on type definitions from the types package. So if all the types lived in the interfaces package, there would be a circular dependency between interfaces and replicated_state. Generally, there are two ways to break a circular dependency: (1) to move common definitions into another package, or (2) to merge two packages into a single one. Merging interfaces with the replicated state was not an option. So we conceived types to contain types that both interfaces and replicated_state depend on.

An interesting property of trait definitions in interfaces is that they only depend on the ReplicatedState type by name. These definitions do not need to know the definition of that type.

An example of a trait definition from the interfaces package that depends on the ReplicatedState type.
trait StateManager {
  fn get_latest_state(&self) -> ReplicatedState;

  fn commit_state(&self, state: ReplicatedState, version: Version);
}

This property of the trait definitions allows us to break the direct dependency between interfaces and replicated_state. We just need to replace the exact type with a generic type argument.

A generic version of the StateManager trait that does not depend on ReplicatedState.
trait StateManager {
  type State; //< We turned a specific type into an associated type.

  fn get_latest_state(&self) -> State;

  fn commit_state(&self, state: State, version: Version);
}

This little trick saves us a lot of compilation time: Now we do not need to recompile the interfaces package and its numerous dependencies every time we add a new field to the replicated state.

Prefer runtime polymorphism.

One of the big questions that the team had when we were designing the component architecture is how to connect components. Should we pass instances of components around as Arc<dyn StateManager> (runtime polymorphism) or rather as generic arguments (compile-time polymorphism)?

Composing components using runtime polymorphism.
pub struct Consensus {
  Arc<dyn ArtifactPool> artifact_pool;
  Arc<dyn StateManager> state_manager;
}
Composing components using compile-time polymorphism.
pub struct Consensus<AP: ArtifactPool, SM: StateManager> {
  AP artifact_pool;
  SM state_manager;
}

Compile-time polymorphism is an indispensable tool, but a heavy-weight one. Most team members also found that the code becomes easier to write, read, and understand when we use runtime polymorphism for composition. Delaying work until runtime also helps with compile times.

Prefer explicit dependencies.

One of the most common questions that new developers ask on the dev channel is something like "Why do we explicitly pass around loggers? Global loggers seem to work pretty well.". That's a very good question. I would ask the same thing two years ago! Sure, global variables are bad, but my previous experience suggested that loggers and metrics are somehow special. Oh well, they aren't after all.

The usual problems with implicit state dependencies are especially prominent in Rust.

The official documentation of the venerable slog package also recommends passing loggers explicitly:

The reason is: manually passing Logger gives maximum flexibility. Using slog_scope ties the logging data structure to the stacktrace, which is not the same a logical structure of your software. Especially libraries should expose full flexibility to their users, and not use implicit logging behaviour.

Usually Logger instances fit pretty neatly into data structures in your code representing resources, so it's not that hard to pass them in constructors, and use info!(self.log, ...) everywhere.

By passing state implicitly, you gain temporary convenience, but make your code less clear, less testable, and more error prone. Every type of resource that we used to pass implicitly caused hard to diagnose issues (scoped loggers, Prometheus metrics registries, Rayon thread pools, Tokio runtimes, to name a few) and wasted a lot of engineering time.

Some people in other programming communities also realized that global loggers are evil. You might enjoy reading Logging Without a Static Logger, for example.

Deduplicate dependencies.

Cargo makes it easy to add dependencies to your code, but it provides few tools to consolidate and maintain those dependencies in a large workspace. At least until cargo developers implement RFC 2906. Until then, every time you bump a dependency version, try to do it consistently in all the packages in your workspace. Cargo update can help you with that.

The same applies to package features: if you use the same dependency with different feature sets in different packages, that dependency needs to be compiled twice.

Unfortunately, using multiple versions of the same package might also result in correctness issues, especially with packages that have zero as their major version (0.y.z). If you depend on versions 0.1 and 0.2 of the same package in a single binary, cargo will link both versions into the executable. If you ever pulled your hair off trying to figure out why you get that "there is no reactor running" error, you know how painful these issues can be to debug.

Put unit tests into separate files.

Rust allows you to write unit tests right next to your production code.

A module that has unit tests and production code in the same file, foo.rs.
pub fn frobnicate(x: &Foo) -> u32 {
    todo!("implement frobnication")
}

#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn test_frobnication() {
        assert!(frobnicate(&Foo::new()), 5);
    }
}

That's very convenient, but we found that it can slow down test compilation time considerably. Cargo build cache can get confused when you modify the file, tricking cargo into re-compiling the crate under both dev and test profiles, even if you touched only the test part. By trial and error, we discovered that the issue does not occur if the tests live in a separate file.

Moving unit tests into foo/tests.rs.
pub fn frobnicate(x: &Foo) -> u32 {
    todo!("implement frobnication")
}

// The contents of the module moved to foo/tests.rs.
#[cfg(test)]
mod tests;

This technique tightened our edit-check-test loop and made the code easier to navigate.

Common pitfalls

In this section, we will take a look at some tricky issues that Rust newcomers might run into. I experienced these issues myself, and I saw several collegues running into them as well.

Confusing crates and packages

Imagine you have package image-magic that defines a library for working with images and also provides a command-line utility for image transformation called transmogrify. Naturally, you want to use the library to implement transmogrify. Your Cargo.toml file will look like the following snippet of code.

Contents of image-magic/Cargo.toml.
[package]
name = "image-magic"
version = "1.0.0"
edition = 2018

[lib]

[[bin]]
name = "transmogrify"
path = "src/transmogrify.rs"

# dependencies...

Now you open transmogrify.rs and write something like:

use crate::{Image, transform_image}; //< Compile error.

The compiler will become upset and will tell you something like

error[E0432]: unresolved imports `crate::Image`, `crate::transform_image`
 --> src/transmogrify.rs:1:13
  |
1 | use crate::{Image, transform_image};
  |             ^^^^^  ^^^^^^^^^^^^^^^ no `transform_image` in the root
  |             |
  |             no `Image` in the root

Oh, how is that? Aren't lib.rs and transmogrify.rs in the same crate? No, they are not. The image-magic package defines two crates: a library crate named image_magic (note that cargo replaced the dash in the package name with an underscore) and a binary crate named transmogrify. So when you write use crate::Image in transmogrify.rs, you tell the compiler to look for the type defined in the same binary. The image_magic crate is just as external to transmogrify as any other library would be, so we have to specify the library name in the use declaration:

use image_magic::{Image, transform_image}; //< OK.

Quasi-circular dependencies

To understand this issue, we'll first have to learn about Cargo build profiles. Build profiles are named compiler configurations that cargo uses when compiling a crate. For example:

release
This is the profile that you want to use for the binaries you deploy to production. Highest optimization level, disabled debug assertions, long compile times. Cargo uses this profile when you run
cargo build --release.
dev
This is the profile that you use for the normal development cycle, the profile that you get when you run cargo build. Debug asserts and overflow checks are enabled, optimizations are disabled for much faster compile times.
test
Mostly the same as the dev profile. This profile is enabled when you run cargo test. When you test a library crate, cargo builds this library with a test profile and injects the main function that executes the test harness. Cargo builds dependencies of the crate being tested using the dev profile.

Imagine now that you have a package with a fancy library foo. You want to have good test coverage for that library, and you want the tests to be easy to write. So you introduce another package, foo-test-utils, that make testing code that works with foo significantly easier.

It also feels natural to use foo-test-utils for testing the foo itself. Let's add foo-test-utils as a dev dependency of foo. Wait doesn't this create a dependency cycle? foo depends on foo-test-utils that depends on foo, right?

Contents of foo/Cargo.toml.
[package]
name = "foo"
version = "1.0.0"
edition = "2018"

[lib]

[dev-dependencies]
foo-test-utils = { path = "../foo-test-utils" }
Contents of foo-test-utils/Cargo.toml.
[package]
name = "foo-test-utils"
version = "1.0.0"
edition = "2018"

[lib]

[dependencies]
foo = { path = "../foo" }

There is no circular dependency because cargo compiles foo that foo-test-utils depends on using the dev profile. Then cargo compiles the test version of foo with the test harness using the test profile and links it with foo-test-utils.

Dependency diagram for foo library test.

foo-test-utils (dev)
foo-test-utils (dev)
foo (test)
foo (test)
incompatible
incompatible
foo (dev)
foo (dev)
Viewer does not support full SVG 1.1

Contents of foo-test-utils/src/lib.rs.
use foo::Foo;

pub fn make_test_foo() -> Foo {
    Foo {
        name: "John Doe".to_string(),
        age: 32,
    }
}
Contents of foo/src/lib.rs.
#[derive(Debug)]
pub struct Foo {
    pub name: String,
    pub age: u32,
}

fn private_fun(x: &Foo) -> u32 {
    x.age / 2
}

pub fn frobnicate(x: &Foo) -> u32 {
    todo!("complete frobnication")
}

#[test]
fn test_private_fun() {
    let x = foo_test_utils::make_test_foo();
    private_fun(&x);
}

However, when we try to run cargo test -p foo, we get a cryptic compile error:

error[E0308]: mismatched types
  --> src/lib.rs:14:17
   |
14 |     private_fun(&x);
   |                 ^^ expected struct `Foo`, found struct `foo::Foo`
   |
   = note: expected reference `&Foo`
              found reference `&foo::Foo`

What could that mean? The reason we get an error is that type definitions in the test version of foo aren't compatible with type definitions in the dev version of foo. These are different, incompatible crates even though these crates have the same name.

The way out of this trouble is to define a separate integration test crate in the foo package and move the tests there. You'll be limited to testing only the public interface of the foo library.

Contents of foo/tests/foo_test.rs.
#[test]
fn test_foo_frobnication() {
    let foo = foo_test_utils::make_test_foo();
    assert_eq!(foo::frobnicate(&foo), 2);
}

The test above compiles fine because both this test and foo_test_utils are linked against the version of the foo library build with the dev profile.

Dependency diagram for foo_test integration test.

foo-test-utils (dev)
foo-test-utils (dev)
foo_test (test)
foo_test (test)
foo (dev)
foo (dev)
Viewer does not support full SVG 1.1

Quasi-circular dependencies are tricky and confusing. They also tend to have negative effect on compilation time. My advice is to avoid them when possible.

Conclusion

In this article, we looked at the tools that Rust gives us to organize our code. Rust's module system is very convenient, but packing many modules into a single crate tend to have negative effect on the build speeds. Our experience suggests that factoring the system into many cohesive packages instead is a better approach in most cases.