Back

This page is still work in progress.

Recently, I got the chance to intern at Microsoft Research in Cambridge, UK. It was a great opportunity since I was in the CCF team. CCF is very close to my research in TEE-based distributed systems and I got to work on developing a new BFT consensus protocol: PirateShip. (It is a wordplay on "Raft", the famous consensus protocol, get it?) I will talk about it some other day. Back to the topic at hand. I thought, since CCF was built in C++ with OpenEnclave, and since my research code also used C++ and OpenEnclave, that I will be able to code in it. To my surprise my manager told me to code the prototype for PirateShip in Rust. This was a blessing in disguise: I have been putting off learning Rust since I didn't have a project that I could use Rust's concepts on. But, building this gave me the chance to learn by doing, which I feel is the best way to learn something.

Before you go any further, this blog post is not meant to be some sort of guide. It is just some useful patterns that I found works in practice. It may describe anti-patterns as well. I am no Rust expert. If you feel there is a flaw in my understanding or coding style, feel free to send me an email. Lastly, here is a link to the repo if you want to check it out.

Everything is a type/trait

Rust has created this incredible network of types and traits that make up its type system. Every behavior is encoded in some trait or type templates.

While this is all good for expressiveness and compile-time checking, it can quickly become quite cumbersome to specify the type of something. How about an atomic ref counted hashmap of String to String that's pinned in the heap? Arc<Pin<Box<HashMap<String, String>>>>.

Anytime my types would get too complicated, I got into the habit of creating a new type out of it. For example,

pub struct PinnedHashMap (pub Arc<Pin<Box<HashMap<String, String>>>>)

I would also have to define the Deref and DerefMut traits on these types, so as to not type the .0 everytime I had to use them. Note that, defining type PinnedHashMap = ... also works, but you won't be able to define new methods on it later on if you wanted to.

Pin your singletons

Networked programs like consensus protocols are mainly developed in the event driven pattern. There are clear benefits to it, e.g., load balancing, easy backpressure creation, separation of concerns etc. To implement event driven pattern in Rust, it is easy to use the async programming features. More on that later. One aspect of the event-driven architecture is that state is stored in a centralised singleton struct that every thread/task/coroutine accesses. However, singletons shared like this must have certain characteristics:

  1. Must be available for the entire lifetime of the task.
  2. Must not move around in memory.
  3. Must be accessible without data races.

To satisfy condition 1, the struct must be 'static, but I think a better solution is to have a ref count. So, I create my singleton struct as Arc<T> and initiate each task as

spawn(async move || {
    task_fn(ctx.clone())
});

If ctx is of type Arc<T>, clone() will just increment the ref count and not create a deepcopy. Furthermore, a clone of the struct is now owned by the task itself, which much simpler to reason about while destructing.

For condition 2, I prefer to Pin<Box<>> the struct. Only Pin<> should have been fine, but big mutable structs are better suited for heap memory. Hence the final type of the struct will be Arc<Pin<Box<T>>>. (I'd also wrap this in a new type, see example above.)

Note that, had this been C++, all of this complexity would be hidden away in the runtime. You'd pass around a T* which you'd allocate during program startup and delete when it ends. But everything is Rust has to be expressed via types.

For condition 3, you need either Atomics or Mutexes. For all simple types, eg, ints and bools, I like to use their atomic versions. For all sub-structs, we must wrap them in a Mutex or RwMutex. Access can then only occur through a lock acquire. If the object is read and written to fairly uniformly, this pattern works. However, not all structs are like this, some only need occassional updates but is read heavily. Let's deal with that next.

Atomic updates of complicated data structures

As I said earlier, some structs are read-heavy (but not read-only). They need to be updated but very rarely. As such having a mutex or RwMutex is extra overhead that I don't want to bear.

Atomic Pointers come to rescue. The logic is as follows:

Handling raw AtomicPtr is a little dangerous. Thankfully, the crossbeam crate implements some of the functionalities I need. So I created a helpful utility struct template:

use std::sync::Arc;

use crossbeam::atomic::AtomicCell;


/// AtomicStruct is the atomically settable version of any struct T.
pub struct AtomicStruct<T>(pub Arc<AtomicCell<Arc<Box<T>>>>);

impl<T> AtomicStruct<T> {
    pub fn new(init: T) -> Self {
        Self(Arc::new(AtomicCell::new(Arc::new(Box::new(init)))))
    }

    pub fn get(&self) -> Arc<Box<T>> {
        let ptr = self.0.as_ptr();
        unsafe{ ptr.as_ref().unwrap() }.clone()
    }

    pub fn set(&self, val: Box<T>) {
        self.0.store(Arc::new(val));
    }

    pub fn is_lock_free() -> bool {
        AtomicCell::<Arc<Box<T>>>::is_lock_free()
    }
}

impl<T> Clone for AtomicStruct<T> {
    fn clone(&self) -> Self {
        AtomicStruct(self.0.clone())
    }
}

For the copy-on-write part, I use Arc::make_mut() on the result I get from the result of get() and then use set() to finalize the update.

Async programming

Build system