RumbleML, a declarative machine learning framework

2020-06-17T00:00:00+00:00

Introduction

Machine learning is an essential cornerstone of contemporary computer science. Its influence spans across various industries while tackling a widening range of complex problems which require increasingly sophisticated solutions. This field’s immense popularity reflects an increasing number of machine learning engineers and data scientists joining the ranks.

Unfortunately, throwing more resources at a problem is rarely the optimal solution. Specifically, we cannot simply rely on having bigger teams of engineers to solve complex problems. Instead, better tooling and improved methodologies can drive machine learning forward in a much more reliable and feasible manner. To this end, I will introduce the RumbleML machine learning framework in this article.

RumbleML is designed to address the two main problems with present-day machine learning tools that hinder productivity:

Poor data independence due to the imperative nature
Limited out of the box scalability and distributed processing support.

Data Independence

Wikipedia, the holy grail of internet knowledge, contains one of the best definitions I came across on data independence:

“[Data independence] refers to the immunity of user applications to changes made in the definition and organization of data.”

In other words, users can focus solely on their application semantics while they can rely on the system to handle underlying data operation complexity. The definition continues as follows:

“Application programs should not, ideally, be exposed to details of data representation and storage.”

This layering is best achieved by resorting to declarative paradigms. A declarative language is a language in which the users specify what they want rather than specifying how they want it. Properly achieved declarative nature brings good abstractions which render the system much easier to use. This, in turn, enables the users to be more productive and write comparably more bug-free code.

The trend of resorting the declarative paradigms instead of imperative ones has been continually gaining traction with modern languages and frameworks. For example, multi-paradigm languages such as Scala and JavaScript are becoming increasingly prominent. Additionally, the web technologies that enable you to display this article on your browser are also fueled by the underlying declarative nature of HTML and potentially by the declarative web frameworks such as “React.js”. Yet, when we look at the domain of machine learning, most frameworks expose APIs in imperative languages. This leads to limited productivity as users are encumbered to handle algorithm semantics, underlying data representations, and runtime execution all at once. Therefore, switching to declarative paradigms for machine learning frameworks could go a long way in enhancing user productivity.

Scalability

Another shortcoming of popular ML frameworks is the reliance on additional tools to provide scalability. The need for additional tools increases the burden on the users as they are forced to handle a larger tech stack during development and maintenance. Said additional tools also commonly expose iterative APIs that further aggravate the data independence limitations. For example, Apache Spark is a fantastic framework that provides scalability via distributed processing. It provides out of the box distributed scalability for arbitrary user programs. In other words, custom machine learning algorithms can be made scalable by using Spark. However, this can easily turn into a daunting task as Spark offers APIs only in imperative languages which require substantial effort to implement algorithm semantics.

The SparkML library, which is built into the Spark framework, offers a large variety of already implemented machine learning algorithms. If the desired use case fits into one of these available algorithms, the implementation efforts can be immensely minimized. Furthermore, If the data is already structured, applying machine learning would be almost trivial.

Unfortunately, in real-world applications, data is hardly ever in a state that is ripe for applying machine learning. Data cleaning and pre-processing operations are vital to the majority of machine learning pipelines. This is where the lack of data independence due to imperative APIs becomes a real pain point for the users. In the presence of unstructured data, which can be heterogeneous and nested, the best solution that Spark can offer is casting everything down to strings. This is extremely far from ideal as the entire type information gets lost in the process and the user is offered little to no help by the system in picking up the pieces.

Enter Rumble

The Rumble engine is introduced to address shortcomings of Spark with regards to heterogeneous data and the lack of data independence. Rumble interfaces Spark with functional and declarative querying language “JSONiq”. JSONiq inherits %95 of its features from XQuery while leaving out the peculiar and hard to understand bits. JSONiq language is fully composable and Turing-complete. The Rumble engine automatically maps the user queries written in JSONiq to Spark runtime execution plans. This enables users to have the scalability of Spark at no additional implementation cost while having full support for heterogeneous data processing.

Fig1. Spark + JSONiq = Rumble

The most prominent feature of JSONiq is the FLWOR expression as it provides the expressiveness of SQL’s SELECT-FROM-WHERE statements in the context of heterogeneous data. The querying and data manipulation capabilities of FLWOR expression will be briefly demonstrated with a simplified example. (The original example is taken from an earlier blogpost on Rumble which can be found in references). Imagine we have the following “person” data set that contains name and year data where year is both heterogeneous and nested.

{ "Name" : "Peter", "Year" : [ 2015, 2014 ] }
{ "Name" : "John", "Year" : [ 2013, "2018" ] }	// heterogeneous
{ "Name" : "Helen", "Year" : [ 2012, [ 2017, 2019 ] ] }	// nested

The following query aggregates the year data as JSONiq’s FLWOR expression immensely simplifies the processing of this heterogeneous dataset. Performing the same task with a non-declarative and non-heterogeneity supporting language immediately turns into low-level programming as the user is forced to handle the iteration and type information manually.

Query:

for $person in json-file("people.json")
let $years := flatten($person.Year)
return { "Name" : $person.Name, "NumberOfYears" : count($years) }

Result:

{ "Name" : "Peter", "NumberOfYears" : 1 }
{ "Name" : "John", "NumberOfYears" : 1 }
{ "Name" : "Helen", "NumberOfYears" : 3 }

Standing on the Shoulders of Giants: RumbleML

The Rumble engine recently took its capabilities yet another big step further. SparkML, that was mentioned earlier, is quite extensive as it is backed by a strong open source community of academics and professionals. Since Rumble was already capable of interfacing Spark, the engine was extended further to interface the capabilities of the SparkML library as well. Through these efforts, the RumbleML framework was conceived. This enabled the existing user productivity and automatic optimization benefits of Rumble to be complemented with ready to use algorithms of RumbleML. With its initial release, RumbleML has achieved over 80% coverage of the functionality offered by SparkML. The development efforts for increasing coverage and optimizing performance further are currently ongoing.

Fig2. Rumble + ML = RumbleML

At a high level, a machine learning query on Rumble can be imagined to consist of three main steps which are common across many similar frameworks: data loading, model training, model usage (omitting the evaluation steps). An example pipeline of this nature is demonstrated in pseudo-code format below:

let $train_data := load-data(...)       // structured data ready to be used in ML
let $test_data := load-data(...)
let $model := train-model($train_data)  // model training is a higher order function that returns a model function 
return $model($test_data)               // apply the model function to get predictions

The concrete implementation and usage of RumbleML revolve around mapping the SparkML concepts to Rumble. The core SparkML concepts of “estimator” and “transformer” are seamlessly mapped into function items of the JSONiq data model. Training sets, test sets, and validation sets, which contain features and labels, are exposed through JSONiq sequences of object items: the keys of these objects are the features and labels. As such, RumbleML simply delegates computation to SparkML without re-inventing the wheel.

Transformers

A transformer is a function item that maps a sequence of objects to a sequence of objects. It is an abstraction that either performs a feature transformation or generates predictions based on trained models. For example:

Tokenizer is a feature transformer that receives textual input data and splits it into individual terms (usually words), which are called tokens.
KMeansModel is a trained model and a transformer that can read a dataset containing features and generate predictions as its output.

Tokenizer operation:

Fig3. Tokenizer operation

KMeans operation:

Fig4. KMeans operation

Estimators

An estimator is a function item that maps a sequence of objects to a transformer. Since it is a function that returns another function, an estimator is a “higher-order function”. Estimators abstract the concept of a machine learning algorithm that fits or trains on data. For example, a learning algorithm such as KMeans is implemented as an Estimator. Calling this estimator on data essentially trains a KMeansModel, which is a Model and hence a Transformer.

Now having these concrete concepts in place, the previous query can be written in the format that can be directly executed in Rumble.

let $train_data := structured-json-file("/path/to/training/data")
let $test_data := structured-json-file("/path/to/test/data")
let $estimator := get-estimator("KMeans")
let $model := $estimator($train_data, { "k": 2 })   // Train KMeans with k=2
return $model($test_data)                           // apply model

Now putting it all together, RumbleML provides out of the box machine learning algorithm support. Rumble on the other hand offers fully composable querying capabilities of JSONiq to handle pre-processing of semi-structured data.

let $raw_data := json-file("people.json")
let $preprocessed_train_data :=
  for $person in $raw_data
  let $years := flatten($person.Year)
  return { "Name" : $person.Name, "NumberOfYears" : count($years) }
let $preprocessed_train_data := annotate(
  $preprocessed_train_data,
  { "Name" : "string", "NumberOfYears" : "integer" }
)
let $estimator := get-estimator("KMeans")
let $model := $estimator($preprocessed_train_data, { "k": 2 })   // Train KMeans with k=2
let $test_data := structured-json-file("people_test.json")
return $model($test_data) 

Conclusion

To wrap up, RumbleML is a new machine learning framework that aims to provide superior usability and scalability benefits compared to alternative ML frameworks thanks to its declarative nature and utilization of Apache Spark.

Why should you try Rumble & RumbleML?

I would strongly recommend trying Rumble for yourself by following the simple steps in our get-started page if any of the following apply to you:

Working with semi-structured, heterogeneous (mixed types) & nested data (JSON, CSV, Text, Parquet, Avro, and more…)
Working with machine learning whether you are a beginner or an expert (More than 70 featurization operations and ML algorithms are available out of the box)
Require scaling capabilities that range into terabytes (whether standalone Spark or on a cluster)
Curious about big data
Love open-source technologies

If you have any recommendations or run into any trouble, please feel free to create an issue at our open-source repository on GitHub.

References

https://github.com/RumbleDB/rumble/
https://rumble.readthedocs.io/en/latest/Getting%20started/
Rumble, an engine to run JSONiq on top of Spark, G. Fourny – Blogpost, https://blog.systems.ethz.ch/blog/2019/rumble.html
SystemML: Declarative Machine Learning on Spark, M. Boehm et. al., Proceedings of the VLDB Endowment, Volume 9, 2015-2016, http://www.vldb.org/pvldb/vol9/p1425-boehm.pdf

Building an inverted index on a large text collection with JSONiq

2019-08-21T00:00:00+00:00

One of the textbook examples for parallel processing is building a standard inverted index in the context of information retrieval. In this post, we will show how Rumble, a JSONiq engine running on Spark, can be used to build such an index in only a few lines of code. This demonstrates that, even if the input is not JSON but a collection of strings (lines of text), JSONiq can handle it and easily create an index in the JSON format.

The Corpus

For the purpose of this post, we will use the complete text of Sherlock Holmes books, which can be found and downloaded here. The code, however, can be used with any input collection.

There is nothing particular about this file; it is just text organized in a large number of small lines:

                          THE COMPLETE SHERLOCK HOLMES

                               Arthur Conan Doyle



                                Table of contents

               A Study In Scarlet

               The Sign of the Four

                  The Adventures of Sherlock Holmes
               A Scandal in Bohemia
               The Red-Headed League
               A Case of Identity
               The Boscombe Valley Mystery
               The Five Orange Pips
               The Man with the Twisted Lip
               The Adventure of the Blue Carbuncle
               The Adventure of the Speckled Band
               The Adventure of the Engineer's Thumb
               The Adventure of the Noble Bachelor
               The Adventure of the Beryl Coronet
               The Adventure of the Copper Beeches
...               

In general, the approach described here will also work with a large number of files. Spark can partition the lines across but also within one file.

Reading a large collection of strings

Rumble introduces the json-file() function to read any number of JSON Lines files as a sequence of objects.

However, the data model of JSONiq supports sequences of any items – the function text-file() reads any number of text files and returns a sequence of strings: one for each line.

Internally, Rumble actually manipulates a Spark RDD of string items.

An idempotent query that returns the entire collection would be

text-file("sherlock.txt", 100)

Where we use 100 partitions. Or more generally to read an entire directory:

text-file("/home/ghislain/corpus/*", 100)

The following query is equivalent but explicitly binds each line to a variable:

for $line in text-file("sherlock.txt", 100)
return $lin

The above queries (assuming they lie in the file query.jq) can be run on the CLI locally with:

spark-submit spark-rumble-1.1.jar \
    --query-path ./query.jq \
    --output-path ./output-directory

as well as on a cluster by submitting the appropriate parameters to spark-submit.

Omitting –output-path on a local execution will simply print on the standard input – it can be the best for experimenting. Only the top 1000 items will be shown to avoid a crash, but this can be changed with –result-size.

The output of all queries above will be in the typical format after a MapReduce of Spark job: a directory filled with part-iiiii files.

This is what we will start with.

Tokenizing

The next step in building a standard inverted index is to tokenize the lines. Tokenization in information retrieval is in itself a core topic: it may involve lemmatization, stemming, etc. Here, we will keep it simple. Let us try as a start to simply tokenize based on spaces.

In order to do so, we just need to add a second for loop on an invocation of the tokenize() function, which returns a sequence of strings:

for $line in text-file("sherlock.txt", 100)
for $token in tokenize($line, " ")
where $token ne ""
return $token

We added the where clause to filter out any empty strings (when a line starts with a space, for example):

THE
COMPLETE
SHERLOCK
HOLMES
Arthur
Conan
Doyle
Table
of
contents
A
Study
In
Scarlet
The
Sign
of
the
Four
The
Adventures
of
Sherlock
Holmes
A
Scandal
in
Bohemia
The
Red-Headed
League
A
Case
of
Identity
...

Getting the distinct terms

There are of course duplicates; we could eliminate them with a distinct-values call (this will be executed in parallel as well):

distinct-values(
  for $line in text-file("sherlock.txt", 100)
  for $token in tokenize($line, " ")
  where $token ne ""
  return $token
)

However, since we want to associate each unique term with a postings list, a group-by is a better idea. The order clause will also sort them alphabetically, like so:

for $line in text-file("sherlock.txt", 100)
for $token in tokenize($line, " ")
where $token ne ""
group by $token
order by $token ascending  
return $token

Since we didn’t do any cleanup, we get a lot of special characters.

"
"'"A
"'"Ah,
"'"And
"'"But
"'"Done!"
"'"Gone!
"'"Ha,
"'"He
"'"Hullo,
"'"I
"'"I'd
"'"I'll
"'"I'm
"'"I've
"'"It's
"'"Just
"'"Mr.
"'"No."
"'"Oh,
...

But we can easily adapt the tokenize() function call to get rid of the most common special characters:

for $line in text-file("sherlock.txt", 100)
for $token in tokenize($line, "[ \"'()&!\\-*,:\\.;/]")
where $token ne ""
group by $token
order by $token ascending
return $token

Which now looks better: numbers first.

Postings lists

The next step is to build the postings lists (we will here assume we have a posting for each term appearing in a line). We need to assign each line to a doc-id.

This can be done with the count clause. Also, in the return clause, we now need to return a more structured construct with a nested array of doc-ids. The syntax is that of JSON, but with the possibility to dymamically compute values with nested JSONiq expressions.

for $line in text-file("sherlock.txt", 100)
count $docid
for $token in tokenize($line, "[ \"'()&!\\-*,:\\.;/]")
where $token ne ""
group by $token
order by $token ascending
return {
  "Term" : $token,
  "Postings" : [ distinct-values( $docid ) ]
}

Et voilà :

{ "Term" : "000", "Postings" : [ 11552, 11631, 11634, 13921, 19725 ] }
{ "Term" : "1", "Postings" : [ 615, 3497, 3582, 9657, 22687, 33863, 40408, 62344, 76758 ] }
{ "Term" : "10", "Postings" : [ 629, 13928, 27179, 46790, 69917, 76367 ] }
{ "Term" : "100", "Postings" : [ 66036 ] }
{ "Term" : "1000", "Postings" : [ 20669, 22683 ] }
{ "Term" : "104", "Postings" : [ 67353 ] }
{ "Term" : "109", "Postings" : [ 51994 ] }
{ "Term" : "10s", "Postings" : [ 14806, 17882 ] }
{ "Term" : "10th", "Postings" : [ 14104 ] }
{ "Term" : "11", "Postings" : [ 630, 2129, 12617, 18052, 17974, 21106, 27179, 62403, 64203 ] }
{ "Term" : "117", "Postings" : [ 16194, 16199 ] }
{ "Term" : "117th", "Postings" : [ 27249, 27691 ] }
{ "Term" : "11th", "Postings" : [ 73290 ] }
{ "Term" : "12", "Postings" : [ 631, 35064 ] }
{ "Term" : "126b", "Postings" : [ 24033, 24088, 24075 ] }
{ "Term" : "127", "Postings" : [ 51993 ] }
{ "Term" : "129", "Postings" : [ 2088 ] }
{ "Term" : "12s", "Postings" : [ 16207 ] }
{ "Term" : "12th", "Postings" : [ 14105 ] }
{ "Term" : "13", "Postings" : [ 1828, 1887, 1898, 35202, 51993, 62841, 62843, 62921, 63570, 64750, 69239 ] }
...

This is it. If we output this result to a specific path, say, an “index” folder, we have built our index in the JSON Lines format.

Querying the index

Now we can use JSONiq to query the index. For example, if we want to query for “Crow”, we look up the postings list and scan the document for the lines given by these postings.

let $postings := json-file("/tmp/index")[$$.Term eq "Crow"].Postings[]
for $document in text-file("sherlock.txt")
count $line
where $line = $postings
return $document

And here we go:

     to the Crow Hill, a huge business which was in strong hands which had
     not only over the killing of the manager and engineer of the Crow

It should absolutely clear to the reader that, of course, best performance would be achieved with additional data structures such as a B+-tree or hash table to organize terms as well as a direct access to any line of text by its number: scanning at query time defeats the purpose of having an index in the first place.

However, building a standard inverted index almost gives a textbook example of how to use JSONiq, because it involves many of the different FLWOR clauses as well as a mix between a textual model and a semi-structured model.

JSONiq and data independence

We hope this post showed how JSONiq makes it simpler to do this kind of task than using PySpark or DataFrames. The key feature of the Rumble project is to maintain data independence and have the developer only thing in terms of one thing: sequences of items, even if the data is heterogeneous and/or nested.

Rumble, an engine to run JSONiq on top of Spark

2019-06-06T00:00:00+00:00

The increasing amount of data available to process, as well as the ever-growing discrepancy between storage capacity, throughput and latency, has forced the database community to come up with new querying paradigms in the last two decades.

MapReduce has become very popular due to its very simple yet general abstraction, which consists in processing key-value pairs in two phases, mapping and reducing, separated by a shuffling phase. Apache Spark has more recently generalized the MapReduce paradigm from two phases to a full DAGs of phases called RDDs.

Yet, this is not the end of the road: MapReduce and Apache Spark are tools, not goals. Writing a program to query data with Spark in Java, Python or Scala feels more like writing assembly instructions (the Spark transformations) than high-level code.

Data independence

A truly data-independent system, in the sense envisioned by Edgar Codd in 1970, must expose the data to the technical user via a functional, declarative query language accompanied by a clean data model. Such languages can then be executed on top of MapReduce or Spark in a way transparent to the user, that is, hiding underlying key-value pairs or RDDs.

A declarative language is a language in which the user says what they want, not how they want. Rather than giving a set of instructions, like a cooking recipe (break the eggs, put the sugar, heat up the oven, etc), the user states what they want (“Mom, can I have a lemon cake?”).

A functional language is a language in which one manipulates mathematical objects. A program is made of an arrangement of expressions that take and return these objects, most often deterministically.

A functional and declarative language leaves room for interpretation and optimization to the engine: it is then open whether the query runs locally, whether it is spread across cores, or across machines in a cluster, or whether (who knows) it runs on DNA. A query optimizer can also rewrite inefficient code, cache intermediate values when it makes sense, etc.

Is the Spark API functional?

Some argue the Spark API, whether in Java, Scala or Python, is already declarative and functional. It is true to some extent, in the sense that transformations are functions that manipulate RDDs. However, there are a few differences with functional languages in the classical sense:

From a productivity perspective, the functions are explicitly called, in other words the program is full of parentheses. A functional language (think Haskell, SQL, …) typically hides the underlying functions in a natural, easy-to-use, English-like syntax.
From a performance perspective, the transformations used are the query plan. Spark will not further optimize and execute the plan as given by the user.
The manipulated instances (RDD) cohabit with the host language (Java objects, etc). This adds extra complexity to the programming experience.

Enter data frames

DataFrames were a major improvement in Spark. When RDDs are highly structured (think rows with the same attributes and types), the underlying memory layout can be optimized, and Spark SQL can be used insted of explicit transformations. This is making Apache Spark more data independent, with a direct benefit in performance: SQL is declarative and (to some extent) functional, so that the underlying engine is free to find an optimal query plan, relying on decades of research done in the area of relational databases.

For example, if we have an input dataset that looks like so:

{ "Name" : "Peter", "Year" : 2015 }
{ "Name" : "John", "Year" : 2018 }
{ "Name" : "Helen", "Year" : 2016 }

It can be imported into dataframes like so:

Name (string)	Year (integer)
Peter	2015
John	2018
Helen	2016

And queried like so:

SELECT year, COUNT(Name) as count
FROM input
GROUP BY year

Heterogeneous and nested data

But in real life, data is not always highly structured.

First, data can be nested. Here is an example with arrays, but this can also be with objects, and with an arbitrary depth.

{ "Name" : "Peter", "Year" : [ 2015, 2014 ] }
{ "Name" : "John", "Year" : [ 2013, 2018 ] }
{ "Name" : "Helen", "Year" : [ 2012, 2017, 2019 ] }

The above input can be read into a dataframe like so:

Name (string)	Years (array of integers)
Peter	[ 2015, 2014 ]
John	[ 2013, 2018 ]
Helen	[ 2012, 2017, 2019 ]

Nestedness can be handled by Spark SQL. Object lookup is done with dots, and Array lookup with the EXPLODE function like so:

SELECT Name, EXPLODE(Years) as Year
FROM input

yielding the following dataframe

Name (string)	Year (integer)
Peter	2015
Peter	2014
John	2013
John	2018
Helen	2012
Helen	2017
Helen	2019

Even though having to explicitly do this conversion adds complexity, this may not feel yet like trying to feet a square peg (trees) in a round-shaped hole (a table).

But this becomes more apparent when the data is heterogeneous, like so:

{ "Name" : "Peter", "Year" : 2015 }
{ "Name" : "John", "Year" : 2013 }
{ "Name" : "Helen", "Year" : [ 2012, 2017, 2019 ] }

where the Year field is fit into a attribute with the least common denominator type: string (quotes are used to make explicit that we do not have integers and an array, but really three strings):

Name (string)	Years (string)
Peter	“2015”
John	“2013”
Helen	”[ 2012, 2017, 2019 ]”

This puts a lot of burden on the user, who has to manually parse these strings back to whichever types make sense to them.

In the real world, data can be heterogeneous: the rows may not have the same fields, some may be missing some may be extra, and the types may be different. This happens, for example, when datasets are created over decades of data gathering. The schema evolves, fields are added or removed, types are changed, but past data does not, or cannot, get converted to the new schema.

JSONiq: a language that natively processes nested, heterogeneous data

JSONiq, which inherits 95% of its features from XQuery, a W3C standard, addresses these problems. Its most useful expression, FLWOR, is the counterpart of SQL’s SELECT FROM WHERE, but more flexible.

With a data model based on sequences of items, which can be homogeneous or heterogeneous or not, it deals natively with nested and heterogeneous data.

For example, we can query the above heterogeneous dataset even though the Year field may consist of integers or arrays of integers, like so:

for $person in json-file("people")
let $years := flatten($person.Year)
return { "Name" : $person.Name, "NumberOfYears" : count($years) }

Which gives the following result:

{ "Name" : "Peter", "NumberOfYears" : 1 }
{ "Name" : "John", "NumberOfYears" : 1 }
{ "Name" : "Helen", "NumberOfYears" : 3 }

FLWOR expressions support projection, selection, grouping, ordering, joining, couting (aka zipping), etc.

Rumble: pushing down the computations to Spark

Rumble is a JSONiq engine that processes heterogeneous, nested datasets with billions of JSON objects by dynamically pushing down computations to Spark. The query above, for example, will be executed with map() and flatMap() transformations, but this is all transparent to the user.

We are now actively working on pushing down more features.

For example, the following queries, even without using FLWOR expressions, are automatically pushed down to Spark, but are significantly simpler and easier to write than using Spark RDDs or Spark SQL for this kind of dataset:

Counting the number of objects in the dataset:

count(json-file("people"))

Discovering all the top-level keys in the dataset:

keys(json-file("people"))

Discovering all the distinct values of the field Name:

distinct-values(json-file("people").Name)

Discovering all the distinct values of the years, whether nested in an array or not:

distinct-values(flatten(json-file("people").Year))

Open-source download

Rumble is available for download under an Apache 2.0 license under http://rumbledb.org . If you find a bug, miss a feature, a builtin function or a pushdown, let us know on the GitHub issues page and we will gladly help.

The design and implementation of a lock-free ring-buffer with contiguous reservations

2019-06-03T00:00:00+00:00

This is the story of how James Munns (from Ferrous Systems) and Andrea Lattuada (PhD student in the Systems Group) designed and implemented (two versions!) of an high-perf lock-free ring-buffer for cross-thread communication. If any of those words look scary to you, don’t fret, we’ll explain everything from the basics.

This post is cross-posted on Ferrous Systems’ blog.

This post is for you if you’re interested in (safe!) concurrency, systems programming, and cool ways to write efficient systems software. If you’ve never written a thread-safe data structure, this post may be a great way to get started!

Circular buffers

A BipBuffer is a bi-partite circular buffer that always supports writing a contiguous chunk of data, instead of potentially splitting a write in two chunks when it straddles the buffer’s boundaries.

Circular buffers are a common primitive for asynchronous (inter- or intra- thread) communication. Let’s start with a very abstract, idealised view of the circular buffer interface, and then consider real-world constraints one by one, till we get to the BipBuffer design.

An idealised infinite buffer

A writer (producer) and a reader (consumer) want to communicate, and have access to the same, contiguous, and infinite array. They both keep a bookmark of which part of the array they’ve (respectively) written and read. They start with these write and read pointers aligned.

When the writer wants to send data, it appends it after the write pointer and then moves the pointer to the end of the newly written chunk. The reader inspects the write pointer at its leisure (asynchronously). When the write pointer has advanced further than the read pointer, the reader can consume and act on the available data. Once that’s done, it moves the read pointer forwards to keep track of which part of the buffer it has already processed.

The reader will never attempt to read past the write pointer, because there’s no guarantee there’s valid data there (i.e. that the writer has put anything there). This also means that the read pointer can never overtake write. For now, we’re assuming an ideal memory system that’s always coherent and where writes are visible immediately and sequentially.

A bounded circular buffer

Computers don’t have magic infinite buffers. We have to allocate a finite amount of memory to use for potentially infinite communication between the writer and reader. In a circular buffer, the write pointer can wrap around the boundaries of the buffer when it reaches the end.

When new data arrives and the write pointer is close to the end, it splits the write in two chunks: one for the remaining buffer space at the end, and one for the remaining data at the beginning. Note that, if the read pointer is still close to the beginning, this has the potential of clobbering data that hasn’t yet been processed by the reader. For this reason, the write pointer is not allowed to overtake read after it has wrapped around.

We end up with two possible memory configurations:

write leads and read follows (write ≥ read), the valid data (written, but not yet processed by the reader) is in the section of the buffer after read and before write;
read leads and write follows (read > write), the valid data is after read, till the end, and from the start of the buffer till write.

Note that we disallow read == write in the second case, as this would be ambiguous: while read can catch up to write, after a wraparound write has to stay one step behind read to indicate that we’re in case 2 instead of case 1.

We repeatedly move from configuration 1 to 2, then back to 1: when read reaches the end of the buffer, it can also wrap around to continue reading at the start.

Contiguous writes/reads

This is all great, but what if we have chunks of data that should remain contiguous in memory when written to the buffer? Look here, there’s a new message to be written, but it doesn’t fit in the remaining buffer space after write.

If, for whatever reason, we aren’t allowed to split this write in two, we’re stuck. Maybe we can just wait for read to move forwards, and place our new data in a single chunk at the start of the buffer? Well, in fact, yea. But there’s a caveat.

We’ve broken the property in configuration 2 earlier: there’s a section of the buffer that’s between read and the end of the buffer, but doesn’t contain any valid data. If we didn’t do anything about it, the reader would keep consuming data, moving read forwards, and it would be oblivious to the fact that at some point it would be reading a section of the buffer that doesn’t contain any valid information.

A Hardware Interlude

Previously we asked:

What if we have chunks of data that should remain contiguous in memory when written to the buffer?

But when would we actually require that data be read or written to in a contiguous manner?

DMA - Direct Memory Access

In embedded microcontroller systems, it is common to have a single core CPU. Instead of having multiple cores, they have a set of features referred to as Memory Mapped Peripherals. These Peripherals act as hardware accelerators for specific behaviors, such as sending or receiving data from a serial port.

In order to minimize the amount of time necessary for the CPU to manually copy data from one place to another, these Peripherals can be configured to perform an action completely autonomously, streaming data to or from a section of memory on the CPU. This action of the hardware directly reading from or writing to the memory is called DMA, or Direct Memory Access.

Instead of reading or writing one byte at a time to the Serial Port, the CPU can instead start the transfer, and when it is complete, process a chunk of bytes at a time. This allows for less time waiting, and is generally a more efficient method of processing data.

A typical usage of DMA (called a DMA transaction) looks like this:

The CPU allocates N bytes of memory to be used for DMA
The CPU instructs the peripheral, such as a serial port, to receive N bytes of data, and to place those bytes in the memory allocated in step 1
Once the peripheral is configured, the CPU resumes performing other actions, and the Serial Port begins filling data into the memory buffer as it is received
When the Serial Port has received all N bytes requested, it notifies the CPU, and stops receiving data
The CPU may now process all N bytes requested, and if necessary, repeat the process at step one

Although we often only have one CPU core in most microcontrollers, we can think of these DMA actors as their own thread. They are able to operate independently of the main CPU’s actions, and read and write memory based on their own needs. In these microcontroller systems, there can be tens or hundreds of these hardware actors, all operating in parallel!

Stackless Operation

In step one of DMA procedure above, we talked about allocating N bytes of memory. On a non-embedded system, this would generally be done by allocating space on the heap - a Box in Rust, or using malloc() in C. In lightweight or timing critical embedded systems, it is uncommon to have a heap. Instead, all memory must be statically allocated, or allocated through the use of the stack.

In these systems, data structures such as Circular Buffers are used to work around these limitations. A fixed amount of space is reserved for use, and a dynamic amount of data within a fixed maximum region is used to simulate a dynamic memory region.

Unfortunately, these DMA transactions do not understand the concept of a circular buffer. They are only aware of a pointer to where the memory region starts, and how many bytes to use from the starting pointer. This means that a normal circular buffer where the data region could wrap around would not work for DMA transfers.

But why is DMA so important?

For operations used with DMA, the speed at which bytes are transferred is often many orders of magnitude slower than the operation of the CPU itself. For a 32 bit ARM CPU, copying 4 bytes from RAM takes a single cycle. In a 64MHz CPU, this means it will take 15.6 nanoseconds to copy these four bytes.

A typical serial port configuration is “115200 8N1”, which means 115,200 baud (or raw bits on the wire per second), with no parity, and 1 stop bit. This means that for every data byte sent, there will be 8 data bits, 1 unused parity bit, and 1 stop bit, to signal the end of the byte, sent over the wire.

This means that we will need 40 bits on the wire to receive a 4 data bytes. At 115,200 bits on the wire per second, this means it will take 347,220 nanoseconds to receive the same four bytes, taking 22,222 times as long as it takes our CPU to copy the same amount of data!

Instead of making our CPU waste all of this time waiting around, we allow the hardware to manage the simple sending and receiving process, allowing our CPU to either process other important tasks, or go into sleep mode, saving power or battery life.

From embedded to datacenters

People writing high-performance application for datacenter grade servers have long realised this is also true for the high-grade, power-hungry CPUs they use.

Modern, efficient network stacks for servers use similar DMA techniques to offload all of this work to the network card, so that valuable CPU time can be spent running data-crunching applications.

A fork in the road

Here’s where the original BipBuffer design decides to maintain two “regions” of valid data, one at the start and one at the end of the buffer: this way it can keep track of which sections of the buffers contain valid data. Have a look at the BipBuffer blog post on CodeProject for details on how this works.

The design based on two regions works great in a single threaded environment, but requires swapping the references to two regions when the rightmost one is depleted. This is tricky to do without explicit locking (mutexes) for cases in which the writer and reader reside on different threads.

Our use case is communication between two concurrent threads of control: either two actual OS threads, or a main thread of control and an interrupt handler in embedded or a device driver. This is where our design takes inspiration from the BipBuffer, but goes in a different direction.

Concurrency design

A common strategy to reduce the amount of coordination that needs to happen between the two threads (writer, reader) is to associate each coordination variable (pointer) with a single thread that has exclusive write access to it. This also happens to simplify reasoning about the design, because it’s always clear who’s in charge of changing which variable.

So, let’s start with a simple circular buffer that has the write and read pointers from before. The writer is the only one who ever changes write, and the reader is the only one who increments read.

So far so good. Each thread is only concerned with writing to one variable, and reading from the other.

High watermark for data

Now let’s re-introduce the requirement that the data written may need to be contiguous. If there’s no space available at the end of the buffer, the writer wraps around and writes the whole contiguous chunk at the start.

As we’ve seen, we need a way to tell the reader which part of the buffer is valid, and which was skipped to be able to write a single contiguous chunk. We’re tracking the high watermark of valid data in the buffer, so what about a watermark pointer that gets written when the writer wraps around and leaves empty space at the end?

Going back to our idealised infinite buffer from before, here’s what things would look like. Whenever the valid region isn’t split in two parts (at the beginning and end of the actual buffer) we simply need to track the write and read pointers, as before. On the other hand, when valid data wraps around the buffer, we leave an artificial “hole” in the “infinite buffer” representation. The watermark lets us keep track of where the “hole” starts, and the end of the physical buffers marks the end.

Implementation

We have all the necessary elements for our non-blocking implementation. We start with the write and read pointers aligned at the start of the buffer and the watermark aligned with the end.

struct ContiguousAsyncBuffer {
  buf: *mut u8,
  len: usize,
  read: AtomicUsize,
  write: AtomicUsize,
  watermark: AtomicUsize,
}

We use AtomicUsize to let the two threads read and update the pointers concurrently and safely. The writer/sender thread is in charge of write and watermark, the reader/receiver is in charge of read. This is important! Contended writes from multiple threads on the same memory location are a lot harder for the CPU’s cache coherence protocol to handle, and will cost latency and throughput.¹ What’s more, it’s a lot easier to reason about correctness of these concurrent protocols if each of the shared pointers are always written by a certain thread (their “owner”).

Writing

As long as there’s enough contiguous buffer space before the end of the physical buffer, as new data arrives (of length write_len) the sender thread moves the write pointer forwards to signal that a new chunk of the buffer is now valid and can be read.

// [writer thread]
buffer.write.store(buffer.write.load() + write_len)

When new data arrives and the write pointer is close to the end, it moves the watermark to its current location, then wraps around. Again, if the read pointer is still close to the beginning, this has the potential of clobbering data that hasn’t yet been processed by the reader. For this reason, the write pointer is not allowed to overtake read after it has wrapped around.

// [writer thread]
if buffer.len.saturating_sub(buffer.write.load()) >= write_len {
  // not shown: check `read` to make sure there's enough free room
  buffer.watermark.store(buffer.write.load() + write_len);
  buffer.write.store(buffer.write.load() + write_len);
} else { // not enough space, wrap around
  // not shown: check `read` to make sure there's enough free room at the beginning of the buffer
  buffer.watermark.store(buffer.write.load());
  buffer.write.store(0 + write_len);
}

You may have noticed that the writer also pushes the watermark forward when there’s room at the end of the buffer. We need to do this because we may have moved it back on a previous wrap-around and we want to avoid the reader now misinterpreting it as a sign that there’s a “hole” at the end.

Reading

We end up again with two possible memory configurations:

write leads and read follows (write ≥ read), the valid data (written, but not yet processed by the reader) is in the section of the buffer after read and before write;
read leads and write follows (read > write), the valid data is after read, till the watermark, and from the start of the buffer till write.

This makes the reader thread’s logic simple: read till you hit the write pointer, or the watermark, and update the read pointer accordingly.

A note on memory ordering

Some of you may have noticed that all of our calls to load don’t take arguments and our calls to store take a single argument, the new value for the AtomicBool. This isn’t valid code, of course. The real signatures take another argument: ordering: Ordering. This instructs llvm on how to emit the proper memory fences and sync instructions to drive the cache coherence and synchronization mechanisms built into the CPUs.

The safe thing to do here is to always choose Ordering::SeqCst, “sequential consistency”, which provides the strongest guarantees. At the hardware level, on x86, due to the hardware design, anything other than Ordering::Relaxed is equivalent to SeqCst. On ARMv7/v8, things get more complicated. The compiler can also make use of this information for reordering when running optimization passes.²

We recommend reading up on Ordering both in the rust doc and in the documentation for your platform. For the purpose of this post, just assume we used Ordering::SeqCst everywhere. This is often good enough in practice, and switching to a weaker Ordering is only necessary to squeeze out the last bit of performance.

In Andrea’s implementation of the lock-free ring-buffer, spsc-bip-buffer, some of the orderings are relaxed for performance. This has the downside that it can introduce subtle concurrency bugs that may only show up on some platform (ARM, for example): to be a bit more confident that everything’s still fine, Andrea’s has continous integation tests both on x86 and ARM.

Support for embedded systems

In James’ implementation of the lock-free ring-buffer, bbqueue, convenience interfaces are provided for statically allocating instances of the ring-buffer. The queue can be split into Producer and Consumer halves, allowing for use of one half in interrupt context, and the other half in non-interrupt (or a different interrupt) context.

It may be important to cache-align these memory locations and make sure there’s only one per cache line to avoid false sharing. spsc-bip-buffer uses cache_line_size::CacheAligned: https://github.com/utaal/spsc-bip-buffer/blob/master/src/lib.rs#L8-L10. ↩
Thanks to Ralf for pointing out that it’s not just about the hardware. https://www.reddit.com/r/rust/comments/bwr2yg/the_design_and_implementation_of_a_lockfree/eq1zcbk/ ↩

Academics Should Build Their Own Computers to Advance Systems Research

2019-05-13T00:00:00+00:00

Mothy was invited to write a post for the ACM SIGARCH blog, and decided to talk about building hardware designed specifically for system software research (as opposed to run commercial workloads). You can check it out here: https://www.sigarch.org/academics-should-build-their-own-computers-to-advance-systems-research/

A fork() in the road

2019-04-20T00:00:00+00:00

Andrew Baumann, Jonathan Appavoo, Orran Krieger, and I have written a paper for Hot Topics in Operating Systems next month about the Unix fork() system call.

fork() is roughly as old as I am - it was in the original version of Unix - so it seems a bit odd to regard it as a “hot topic”. In the paper we argue that today it should be deprecated, for a variety of reasons.

All four of us have been fascinated by fork() for a long time. We’ve been involved in building various operating systems in industry and academia for many years - including L4 (Andrew), Nemesis (Mothy), Mungi (Andrew), Tornado (Orran, Jonathan), K42 (Orran, Jonathan, Andrew), Barrelfish (Andrew, Mothy), and Drawbridge (Andrew) - none of which really supported fork() well, either by accident or design. This was our attempt to get our collective thoughts down on paper, and also reassess fork() in the light of modern hardware.

The reaction was interesting to say the least. It’s mostly died down by now, but it spurred some interesting discussion:

A lot of these are interesting and highly informed takes on the paper, sometimes agreeing, sometimes disagreeing, but often taking some quite nuanced position in between.

Orran also posted a great conversation between him and his collaborator at Red Hat, Uli Drepper, about the topic:

https://www.bu.edu/rhcollab/2019/04/11/a-fork-in-the-road/

The four of us feel pretty happy with how the paper turned out, but I think it’s worth emphasizing a few points:

We’re not Microsoft! Andrew works at MSR, but the rest of us have little connection with the company. This paper is certainly not representative of, or instigated by, Microsoft.

Quite a few comments assumed this was a “Microsoft whitepaper”, whereas it’s actually an academic position paper written by 4 researchers who know each other, one of whom happens to be at MSR. The rest of us are all at Universities.
We’re not dissing Unix! Unix (and its clones) are an amazing thing, and we all use it daily (I’m writing this on my Linux laptop). More than that, its safe to say that every new OS built since the 1980s has been defined in reference to Unix, either because it is more or less a copy of Unix architecturally, or explicitly an attempt to do something different.

Unix was a minimalist, simple, elegant OS written in a short time by people who deeply understood how these things work. In this respect, fork() was a very minor piece of the puzzle - the least-programming-effort way to create a new process by copying out the state to swap in 27 assembly instructions.

What I find fascinating is how, since then, fork() has come to dominate the OS. As we show in the paper, in a modern Unix-like OS like Linux or BSD, the semantics of fork() touch and constrain the design of every part of the kernel, and quite a bit of user space as well.

In a sense, today, Unix is fork() - once you’ve decided you want fork() to be fast, all the rest follows from good engineering principles.

Conversely, if you really want to explore an alternative OS design for modern hardware, you have to give up on fork()

I’m looking forward to Andrew’s presentation…

String interning and beyond, in differential dataflow

2018-12-10T00:00:00+00:00

Differential dataflow does a great number of interesting bits of data processing, but what about when you want to use complicated types, like strings? In this post we’ll check out how to use differential dataflow to intern strings, replacing them with integer identifiers that will allow the rest of our computation to execute more efficiently. From there, we’ll see how this generalizes to automatically assigning distinct record identifiers to collection elements, much like a database does!

As it has been a while, differential dataflow is a sweet programming framework which automatically updates its computations when you change its inputs. It goes really fast, scales, does twenty-seven things no other systems can do, blah blah.

In the course of using differential dataflow, you may find yourself using types that contain strings. In one recent, interesting case of program analysis, the input records look like tab-separated lines of text:

<sun.font.SunFontManager$14: void <init>()> <init>  ()  sun.font.SunFontManager$14  void    ()V 0
<sun.awt.image.VSyncedBSManager$SingleVSyncedBSMgr: void <init>()>  <init>  ()  sun.awt.image.VSyncedBSManager$SingleVSyncedBSMgr   void    ()V 0
<javax.swing.AbstractAction: boolean shouldReconfigure(java.beans.PropertyChangeEvent)> shouldReconfigure   (java.beans.PropertyChangeEvent)    javax.swing.AbstractAction  boolean (Ljava/beans/PropertyChangeEvent;)Z 1
<sun.font.SunFontManager$14: void <init>()> <init>  ()  sun.font.SunFontManager$14  void    ()V 0
...
<java.awt.geom.AffineTransform: void transform(double[],int,double[],int,int)>  transform   (double[],int,double[],int,int) java.awt.geom.AffineTransform   void    ([DI[DII)V  5
<java.awt.geom.AffineTransform: void transform(double[],int,double[],int,int)>  transform   (double[],int,double[],int,int) java.awt.geom.AffineTransform   void    ([DI[DII)V  5
<java.awt.geom.AffineTransform: void transform(double[],int,double[],int,int)>  transform   (double[],int,double[],int,int) java.awt.geom.AffineTransform   void    ([DI[DII)V  5
...
<org.gjt.sp.jedit.pluginmgr.MirrorList: void readXml()> readXml ()  org.gjt.sp.jedit.pluginmgr.MirrorList   void    ()V 0
<sun.nio.cs.StandardCharsets$Aliases: void init(java.lang.Object[])>    init    (java.lang.Object[])    sun.nio.cs.StandardCharsets$Aliases void    ([Ljava/lang/Object;)V  1
<sun.nio.cs.StandardCharsets$Aliases: void init(java.lang.Object[])>    init    (java.lang.Object[])    sun.nio.cs.StandardCharsets$Aliases void    ([Ljava/lang/Object;)V  1
<org.gjt.sp.jedit.pluginmgr.MirrorList: void readXml()> readXml ()  org.gjt.sp.jedit.pluginmgr.MirrorList   void    ()V 0
...

Barf.

These particular records have a type that is in essence an array of strings six-long, [String; 6], and we need to perform maps, filters, joins, grouping, and even fixed-point iteration, all using types like this (likely with other values of 6).

Strings do work!

Before going anywhere too exotic, like Fiji (sigh), the String type totally works in differential dataflow. You can go and do graph computations where the node type is String, or (Vec<String>, bool), or whatever crazy types you kids use to do your graph processing in Javascript (lol, I know… “types”).

The String type works but it can be a bit inefficient when compared to something like usize, which is just a much simpler type to work with when we want to hash-distribute records, or sort records, or group them by keys. It is also much more efficient to clone a usize than a String—the latter means we must allocate some memory and most likely de-allocate it at some future point—and generally even using String types means we have some memory indirection along our critical path (farewell, branchless sort performance!).

Interning Strings

While String works, it might be interesting to perform something like string interning, in which each of the string instances is replaced by one representative, deduplicating the actual strings (allocations) being used.

Let’s try something same same, but different: replacing each string with a unique integer identifier. We do lose the ability to do string-y operations, like substrings and converting to upper and lower case, but we can still do equality testing and hash-distribution and many of the core things we need to do in our data-parallel compute lifestyle.

In the context of the program analysis project linked above, we can still execute Datalog programs, as they only really care about exact matches between fields.

Single-threaded, string replacement would be super easy:

/// Ensure `string` exists in `map`, return unique identifier.
fn intern_in(string: String, map: &mut HashMap<String, usize>) -> usize {
    let len = map.len();
    *map.entry(string)
        .or_insert(len)
}

Whenever we intern a string, we either return its existing identifier or add it with a new distinct identifier and then return that.

This skips a few fun issues like do we ever remove interned strings (not here), how to we look up strings from their identifiers (a second map, I guess), can we do this in parallel, and maybe a few other issues.

We aren’t going to stick with this approach, but it is good to see that it isn’t all that painful.

Interning Strings IN DIFFERENTIAL DATAFLOW

Let’s write a fairly simple computation that will intern strings using differential dataflow.

Can’t we just use the above single-threaded fragment? Sure, sure. I did that in the project linked up above. It works, it’s totally fine, but it just feels a bit gross and non-robust. For example, we have to do all of the data loading on one thread, because we can’t intern the strings in parallel. If the set of strings changes dramatically, we keep all of the pre-existing strings around grotting up the place. Also c’mon, this is going to be neat and you might learn something. :D

As a first, guess, couldn’t we just intern strings by hashing each of them and using the hash as the integer identifier? Almost, but there could be collisions. How about, if there are collisions, we pick one winner and re-hash the others until they don’t collide?

Bingo!

/// Assigns a unique identifier to each element of `collection`.
fn intern<G, D>(collection: &Collection<G, D>) -> Collection<G, (D, usize)>
where
    G: Scope,
    G::Timestamp: Lattice,
    D: Data,
{
    collection
        // initialize each string at "round" zero.
        .map(|record| (0, record))
        .iterate(|temp| {
            // propose a candidate hash from (round, string),
            // group by hash and pick at most one winner,
            //       defer non-winners to the next round.
            temp.map(|pair| (pair.hashed(), pair))
                .group(|_hash, input, output| {
                    // first (round, string) wins!
                    output.push((input[0].0.clone(), 1));
                    // if any losers, increment their rounds.
                    for ((round, record),_count) in input[1..].iter() {
                        output.push(((*round+1, record.clone()), 1));
                    }
                })
                .map(|(_hash, pair)| pair)
        })
        .map(|pair| (pair.1, pair.hashed()))
}

This works great. Each string in collection gets a fairly random hash in round zero, and if there are any collisions we pick a winner (the earliest round, breaking ties lexicographically by string) and promote the losers to the next round. We don’t really expect many collisions, and we really don’t expect any one string to repeatedly collide with other strings (assuming our .hashed() method is worth anything).

What happens when we have a change in our input strings?

Not very much, which is great news! There was so little interaction of strings, that there are similarly sparse changes to their interactions. Perhaps we could add a string that evicts a winner, but the cascade should be quite limited. Perhaps when we remove a winner we end up finding the string or two it collided with and re-naming them winner. But, most of the time we add or remove a string and its hash value as an identifier.

This is all incrementalized, distributed, streaming, self-compacting, all that good technology that went into doing high-throughput graph computation, now assigning distinct numbers for your strings.

Using interned strings

Way back up there we had some horrible [String; 6] folks that reminded us about Java and that evil still walks the Earth. How do we wire together a string interning fragment (like just above) with our stream of Java six-tuples?

I’ll show you a pretty naive version, and I’ll sketch a more sophisticated version for your homework!

The most naive way to do things is to i. extract the candidate strings to intern, and then ii. repeatedly join the six-tuples with the interned strings to extract identifiers for each coordinate at a time.

// Each six-tuple offers six strings to intern.
let interned = intern(&six_tuples.flat_map(|x| x));

let interned_six =
six_tuples
    // map each to (next_key, ([hashes], [strings])).
    .map(|x| (x[0], ([], x[1..6])))
    .join_map(interned, |_k,h,(hs,ss)| (ss[0], ((hs,h), ss[1..5])))
    .join_map(interned, |_k,h,(hs,ss)| (ss[0], ((hs,h), ss[1..4])))
    .join_map(interned, |_k,h,(hs,ss)| (ss[0], ((hs,h), ss[1..3])))
    .join_map(interned, |_k,h,(hs,ss)| (ss[0], ((hs,h), ss[1..2])))
    .join_map(interned, |_k,h,(hs,ss)| (ss[0], ((hs,h), ss[1..1])))
    .join_map(interned, |_k,h,(hs,ss)| (hs,h));

Ok, now all those join_map closures aren’t actually legit Rust, but I hope you get the gist about what is going on. Six times, we extract the first remaining string and use join_map to find us an integer hash value to replace it, and keep the remaining strings.

Arrangements

As a quick aside, let’s recall that differential dataflow has a concept of an “Arrangement”, which is an indexed representation of a collection that can be re-used without additional cost. In the code above, each time we use interned we re-build an index of the data, and we maintain the six copies as interned changes.

We can skip this by arranging interned once and re-using the arrangement, like so:

// Each six-tuple offers six strings to intern.
let interned = intern(&six_tuples.flat_map(|x| x)).arrange_by_key();

let interned_six =
six_tuples
    // map each to (next_key, ([hashes], [strings])).
    .map(|x| (x[0], ([], x[1..6])))
    .join_core(interned, |_k,h,(hs,ss)| Some((ss[0], ((hs,h), ss[1..5]))))
    .join_core(interned, |_k,h,(hs,ss)| Some((ss[0], ((hs,h), ss[1..4]))))
    .join_core(interned, |_k,h,(hs,ss)| Some((ss[0], ((hs,h), ss[1..3]))))
    .join_core(interned, |_k,h,(hs,ss)| Some((ss[0], ((hs,h), ss[1..2]))))
    .join_core(interned, |_k,h,(hs,ss)| Some((ss[0], ((hs,h), ss[1..1]))))
    .join_core(interned, |_k,h,(hs,ss)| Some((hs,h)));

Technically a few other things changed. We use join_core rather than join_map because that is the method that consumes arranged inputs (and it wants something like an iterator for output, even though we have just output for each match). But, this will maintain just one copy of interned and re-use it six times!

Intermission

It was at this point that I proposed some homework. The industrious reader was invited to take the recently introduce sweet join tech to try and perform those joins above without maintaining six copies of the the six_tuples collection, each with progressively fewer strings and more integers.

Let’s return to our story, and see where the author goes next.

Update (“The next day”)

Hey I tried that homework up above. I bet you did too, right? Pretty hard, huh?

Despite being pretty sure that the dogsdogsdogs project has positive implications, I think you still need to keep around the original [String; 6] data somewhere, because if any string identifiers change you’ll need to push the changed [usize; 6] records.

Of course, if we naively indexed each [String; 6] keyed by each of the six fields we are actually doing worse than above, recording each string six times. So, that sucks. How do we avoid keeping six copies (or more, for numbers larger than six) of the input data around indexed different ways?

How do real databases do this? Can we not just steal from them?

Yes we can.

Row Identifiers IN DIFFERENTIAL DATAFLOW

In a great many database systems and schemas, when you introduce new records to a relation that does not otherwise have a primary key, you automatically get an auto-incremented “row identifier” added in that acts as a primary key. This row identifier lets you speak about the rows without slogging all of their data around all over the place. When performing joins and such a database system can just track the row identifiers involved (and perhaps the subset of attributes) rather than the potentially kilobytes of per-record payload.

Let’s do that IN DIFFERENTIAL DATAFLOW!

Specifically, let’s assign row identifiers in differential dataflow so that we can more efficiently intern strings, but at the same time we’ll come up with a great way to implement relational joins. The first one is probably more important, right?

Let’s imagine we start with some collection of string six-tuples:

let horror: Collection<_, [String; 6]>;

and what we want most is to assign unique identifiers to each six-tuple.

// Use the string interning method up above.
let horror_ids = intern(&horror);

Yeah, it turns out we already invented it. Oops. !#$!ing generic programming.

Decomposing records

We are now sitting on records of the form ([String; 6], usize). What comes next?

Next, let’s break apart these six-tuples into six collections, one for each attribute, each containing elements of the type (usize, String) corresponding to the record identifier and the value for its associated attribute.

The first step is to break down our ([String; 6], usize) collection to one containing six (usize, usize, String) records for each input record, announcing each attribute and row identifier, and the value. Then we’ll partition into six collections using the attribute identifier:

let attr_row_value =
horror_ids
    .flat_map(|(horror, id)|
        horror
            .into_iter()    // six-element iteration.
            .enumerate()    // pre-pend with attr_id.
            .map(|(attr, value)| (attr, (id, value)))
    );

// Partition collections by attr_id (timely magic).
let attr_collections: Vec<Collection<G, (usize, String)>> =
attr_row_value
    .inner                  // inner timely stream
    .partition(6, |((a,(i,v)),t,r)| (a, ((i,v),t,r)))
    .into_iter()
    .map(|stream| stream.as_collection())
    .collect::<Vec<_>>();

We now have six collections. One for each attribute, each containing records like (row_id, value). That was amazingly fun, of course, but why did we do this?

There are at least two reasons that I can think of, string interning and making joins go faster, let’s talk each of them out.

String interning

Obviously this is why we are here. We are all about the strings.

We can independently intern the strings in each of the six collections, like so:

let attr_interned =
attr_collections
    .iter()
    .map(|collection|
        collection
            .map(|(row_id, value)| (value, row_id))
            .join_core(&interned, |_, row_id, hash| Some((row_id, hash)))
    .collect::<Vec<_>>();

Did you notice I used join_core, which is the version that doesn’t re-index interned and instead re-uses the common arrangement? :D

This computation produces six collections each containing (row_id, hash) pairs, and we can use those more-or-less as if they were (row_id, value) pairs, almost. At the same time, it maintains each attribute value once, in indexed form for the first input to join_core. This is way better than six times.

Technically we have an additional copy of each attribute value back where we determined record identifiers (that subcomputation stashes each of the argument records).

Making joins better

Our decomposition of our relation into the attributes allows us to do a bit of a la carte columnar join processing, which can be fun.

Let’s imagine that instead of just forming each of the (row_id, value) collections we also arrange each of them, resulting in a maintained collection indexed by row_id.

let attr_arrangeds =
attr_collections
    .iter()
    .map(|collection| collection.arrange_by_key())
    .collect::<Vec<_>>();

We can now use these arrangements to re-assemble arbitrary subsets of attributes in the record, with surprisingly small cost.

let attrs_034 =
attr_arrangeds[0]
    .join_core(&attr_arrangeds[3], |r,a0,a3| Some((r,(a0,a3))))
    .join_core(&attr_arrangeds[4], |r,(a0,a3),a4| Some((r,(a0,a3,a4))));

The “surprisingly small” cost here (debatable) is because each of the collection arguments to join_core is pre-arranged, meaning sorted by the key (row_id). When join_core gets to work, it does one sequential swing through each arrangement, picking up the attributes we are looking for.

The attentive reader might complain that the first input to the second join_core operator is not arranged, and will cost, and this is where we break out the dogsdogsdogs delta queries. They use the arranged collections but don’t need to maintain intermediate collections (because they respond to change is e.g. attr_arrangeds[4] with a different dataflow).

We can also independently filter the attributes, and then join. This allows us to tap dance through only the relevant records in each of the relations. For example, consider:

let attrs_034 =
attr_collections[0]
    .filter(|(row_id, value)| value.len() > 4)
    .join_core(&attr_arrangeds[3], |r,a0,a3| Some((r,(a0,a3))))
    .join_core(&attr_arrangeds[4], |r,(a0,a3),a4| Some((r,(a0,a3,a4))));

The arrangement going in to the first join_core has been reduced, but the row identifiers are still in order. All off differential dataflow’s join operators are implemented to seek rather than scan between keys, and so a sparse set of filtered keys will allow the operator to zip forward through the keys in attr_arrangeds[3], rather than scanning all of them.

This works less well once we’ve rocked things down to interned string identifiers, but .. well maybe this section should have come before that section. Still working on that.

Performance Update (“A few days later”)

It turns out this computation led me down an interesting performance rabbit hole, and I’ve just returned with fabulous and wonderful stories!

Let’s talk through the intern function up above, and what it does. We will tease out a few performance issues, ones that you might not have even realized were in your power to fix (one still isn’t), and see things get a fair bit better by doing so.

Baseline performance

The core of intern’s iterative computation takes the full set of (round, string) inputs we have and groups them by their candidate hash value, and produces winners and losers as output (where losers get their round incremented):

    temp.map(|pair| (pair.hashed(), pair))
        .group(|_hash, input, output| {
            output.push((input[0].0.clone(), 1));
            for ((round, record),_count) in input[1..].iter() {
                output.push(((*round+1, record.clone()), 1));
            }
        })
        .map(|(_hash, pair)| pair)

This is all well and good. Let’s see how long it takes to intern 1,000,000 integers (not strings, sorry; but this is easier):

Echidnatron% cargo run --release --example test -- 10000000
    Finished release [optimized + debuginfo] target(s) in 0.09s
     Running `target/release/examples/test 10000000`
12.505527384s
^C
Echidnatron%

Hard to know if this is particularly good (it’s about 1.25 microseconds per identifier). The process’s memory use stabilizes at 1.31GB which is around 131 bytes per identifier. That is a bit larger than we might expect.

It turns out that our program is doing something silly, at least silly for this type of computation. Our group operator receives inputs with largely distinct hash values, and for these inputs it simply reproduces them as output. To stay sane, the group operator maintains both its input and output collections, so it maintains two nearly identical copies.

Seems pretty natural, though. Unless we want to invent a new group implementation, which we (I) don’t.

Switching to differences

Our computation determines the “new” pairs (round, string) by incrementing the round field of a relatively small number of records. What if instead we asked group to produce only the changes?

What does that mean? Isn’t differential dataflow already incrementalized? Yes, yes, but I mean something different. What if we compute “how should we change temp to avoid collisions?”, and then concatenate the results with temp:

    temp.map(|pair| (pair.hashed(), pair))
        .group(|_hash, input, output| {
            // output.push((input[0].0.clone(), 1));
            for ((round, record),_count) in input[1..].iter() {
                output.push(((*round+0, record.clone()),-1));
                output.push(((*round+1, record.clone()), 1));
            }
        })
        .map(|(_hash, pair)| pair)
        .concat(&temp)

It is a bit weird to do this ourselves, rather than asking differential dataflow to do it for us, but we are in the special situation where the group operator has the same input as output, and we want to do input v. output differences, rather than time-by-time differences which differential does well.

In some weird interpretation we are doing “spatial differencing” rather than “temporal differencing”, relating different locations in our dataflow. Never believe that just because someone has fancy names for things means they understand what is going on.

For example, watch this:

Echidnatron% cargo run --release --example test -- 10000000
    Finished release [optimized + debuginfo] target(s) in 0.08s
     Running `target/release/examples/test 10000000`
15.039220586s
^C
Echidnatron%

That took longer. What a pity. On the other hand, it stabilizes at a memory footprint of just 578MB, which is about half the size of our prior attempt, which is a success! That is some 57.8 bytes per record, which is basically what we should expect given our data structures (seven eight byte records, including the hash, record, two offsets, a two-dimensional timestamp, and a difference).

So what is going on that makes it go slower, despite using half the memory?

Optimizing iteration

The short version (there is a longer version; it involved cursing) is:

Our iterate method develops all (round, string) pairs, and so at some point (the beginning) we need to bring in these initial values, apply one step of our logic, and then subtract the initial values. We have to do this even when nothing changes, because we have to check that nothing has changed. When nothing changes this difference cancels and we don’t have anything to write down (yay!) but we had to do the work to determine this (oh well…).

To work around this, we will need to write a different computation. A computation that doesn’t iteratively develop the full collection of (round, string) pairs, but perhaps instead iteratively develops the changes needed to apply to the initial collection (rather than changes to temp).

Here is how I wrote that:

/// Assigns a unique identifier to each element of `collection`.
fn intern<G, D>(collection: &Collection<G, D>) -> Collection<G, (D, usize)>
where
    G: Scope,
    G::Timestamp: Lattice,
    D: Data,
{
    // capture the initial (round, record) pairs.
    let init = collection.map(|record| (0, record));

    init.filter(|_| false)
        .iterate(|diff|
            // Bring `init` into scope and add to `diff`.
            // Then determine changes relative to `init`,
            // which means subtracting records at round 0.
            init.enter(&diff.scope())
                .concat(&diff)
                .map(|pair| (pair.hashed(), pair))
                .group(|_hash, input, output| {
                    // keep round-positive records as changes.
                    let ((round, record), count) = &input[0];
                    if *round > 0 {
                        output.push(((0, record.clone()), -*count));
                        output.push(((*round, record.clone()), *count));
                    }
                    // if any losers, increment their rounds.
                    for ((round, record), count) in input[1..].iter() {
                        // Subtract a *round zero* instance of the record.
                        output.push(((0, record.clone()), -*count));
                        output.push(((*round+1, record.clone()), *count));
                    }
                })
                .map(|(_hash, pair)| pair)
        )
        .concat(&init)  // concat init *outside* the loop.
        .map(|pair| { let hash = pair.hashed(); (pair.1, hash) })
}

This computation develops what is very likely a small collection diff, although it does use init which may be not small. Fortunately, it uses init only once, in the input to group. Further, as it does not start from init (our filter fixes that) we don’t end up wasting any time subtracting it back off.

Echidnatron% cargo run --release --example test -- 10000000
    Finished release [optimized + debuginfo] target(s) in 0.08s
     Running `target/release/examples/test 10000000`
6.519266834s
^C
Echidnatron%

This works out to 652 nanoseconds per record (sorting by hash, I think), and the memory footprint ends up at 621MB which is approximately what we had up above. Maybe a touch more for reasons I don’t currently understand, but good enough for us for now.

We were very fortunate with this computation because it was possible to reason about changes relative to init, as we knew that the only thing that happens to round counters is that they are incremented from zero. Our second implementation, slower but with a smaller footprint, is probably the more reasonable style of computation to expect people to be able to write for more general computations.

Physical Adressing on Real Hardware in Isabelle/HOL

2018-11-09T00:00:00+00:00

Modern memory systems are much more complicated than the traditionally assumed virtual and physical address space separation. We explain in this post which effects can not solely expressed by the basic model and are important for correct function of operating systems. We summarize our recent paper. In this work we present a theory for addressing in such modern memory subsystems. We formalize the theory in Isabelle/HOL.

Introduction

Today’s operating system, text books and also verification efforts assume a quite simple memory model: Applications execute in a virtual address space, virtual addresses get translated into the physical address space by a MMU. In this physical address space the systems DRAM and memory mapped registers of devices are located. If there are devices that perform direct memory accesses (DMA), they access the physical address space directly. This model is depicted in Figure 1.

Fig1. The traditional view of memory systems

However, on real systems this simplistic view fails to capture the necessary details for correct operation. In a previous workshop paper we have discussed a variety of real systems and shown that more complicated systems are indeed common. Let me show you one example to illustrate some effects we encounter in Fig2.

Fig2. A real system (excerpt)

This figure shows how different cores can access the GPT5, a memory mapped general purpose timer. The physical address that a core has to issue depends on the core, the A9 core has two option of reaching the device, either using the private interface to the timer using the base address 0x40138000 or accessing the timer through the L3 bus using the address 0x49038000. Address synonyms (two addresses that access the same physical location) are not expressible in the simplistic memory model.

Accessing the same location from the DSP is possible, but a different address has to be used. The M3 core has to issue again a different address, even though the memory accesses passes through the same interconnect. In such a system it does not make sense anymore to talk about a physical address space, every actor has its own view of the system. These effects are quite common, for instance on x86 platforms each CPU has its own local APIC registers, that all appear under the same address.

The decoding net

Since the simplistic model is not powerful enough, we need something better. We noted there is no physical address space anymore, hence the notion of a physical or absolute address does not make sense anymore. Instead we focus on talking about relative addresses. We call a relative address plus the node on which this address is to be interpreted as name. The nodes in our system have two properties: They can accept addresses, and they can translate addresses. To model caches which either accept or translate, depending on the cache state, we permit addresses that translate to multiple output addresses. Our system of nodes forms a directed graph.

A natural question to ask is, where does a certain name end up when issued on this node. A difficulty in this is that our system permits loops. A real system that has memory addressing loops is probably faulty or misconfigured, but we do not want to rule this out in our modeling approach. In fact, showing the absence of such misconfiguration in any real system is one of our future tasks. In the paper we give a set theoretic and an operational definition of the resolve function. We show that they are equivalent, under the assumption that the operational resolution actually terminates. Because of this assumption, we study under which conditions termination actually occurs. The rank function assigns each name a natural number and it must decrease every time a translate step is taken. We found that specifying a rank function is a necessary and sufficient condition for termination.

While the decoding net captures the essential details, it is not a very handy representation for use in an operating system. If we fix one observer, we can derive a simplified decoding net that resembles the two layer simplistic model. We do so by collapsing all the translate steps into one. We formalized this process and show that it preserves view-equivalency for the chosen observer. View equivalent means that the same address resolves to the same address.

In a real operating system, we implement a data structure that refines the decoding net. However, we also must populate this datastructure. How this is done depends on the system we are running on. Certain platforms offer discovery mechanisms, such as ACPI. On other platforms, this information can not be discovered and must be provided by the operating system developer. In this case, we need to be able to express a decoding net in some form of syntax. The syntax presented in the paper adds two syntactical extensions to the plain decoding net that are focused on expressing real systems: First, the syntax works with ranges of addresses, because often whole address blocks are remapped. Second, the introduction of an overlay concept. If a node overlays another, it will map all addresses not otherwise mapped to that node. We show that the syntax refines the decoding net.

Based on this syntax we have developed a system description language that we use as a runtime representation of the memory system and at compile time to generate initial kernel page tables.

A further application is expressing real hardware with respect to the decoding net. To showcase this, we model a simple software loaded translation lookaside buffer (TLB) in Isabelle and map each of its configuration state to a model node.

References

The paper presented in this blog post.
Our prior workshop paper.

A hammer you can only hold by the handle

2018-11-05T00:00:00+00:00

Today we’re looking at the rust borrow checker from a different perspective. As you may know, the borrow checker is designed to safely handle memory allocation and ownership, preventing accessess to invalid memory and ensuring data-race freedom. This is a form of resource management: the borrow checker is tracking who’s in charge of a chunk of memory, and who is currently allowed to read or write to it. In this post, we’ll see how these facilities can be used to enforce higher-level API constraints in your libraries and software. Once you’re familiar with these techniques, we’ll cover how the same principles apply to advanced memory management and handling of other more abstract resources.

This is an extended blog post version of my RustFest Zürich talk (it was recorded). That’s why you’ll find “rustfest” in some of the code examples.

Affine type systems

First, a refresher on affine types. Affine type systems, like Rust’s, only allow a variable to be used once (if it’s not a reference). This is at the core of the ownership semantics, and it’s a significant departure from other mainstream languages (think of using a variable multiple times in C). Here’s an example:

1
2
3
4
5
6
7
8
fn use_name(name: String) { }

fn main() {
    let name = String::from("Andrea");
    use_name(name);

    println!("{}", name);
}

Note that use_name takes name’s ownership (and it’s not pass-by-reference) so name is moved on line 5, and it cannot be used again on line 7. Here’s the compiler output:

error[E0382]: use of moved value: `name`
 --> affine.rs:7:18
  |
5 |   use_name(name);
  |            ---- value moved here
6 | 
7 |   println!("{}", name);
  |                  ^^^^ value used here after move
  |
  = note: move occurs because `name` has type `std::string::String`,
          which does not implement the `Copy` trait

error: aborting due to previous error

For more information about this error, try `rustc --explain E0382`.

If you’d like a more in-depth explanation on ownership (Rust’s lingo for its affine type system), you can take a look at the relevant chapter in the Rust book.

Drop

Now we know what happens if we try to transfer ownership of a variable more than once, but what if we never use it inside a scope?

First of all, Rust is lexically scoped, so variable names are only valid within the scope where they’re defined.

1
2
3
4
5
6
7
8
9
10
11
12
13
struct Thing {
    number: u32,
}

fn main() {
    let a = 4;

    if a > 3 {
        let thing = Thing { number: a };
    } // `thing` dropped here

    println!("{}", thing);
}

error[E0425]: cannot find value `thing` in this scope
  --> drop.rs:12:18
   |
12 |   println!("{}", thing);
   |                  ^^^^^ not found in this scope

error: aborting due to previous error

For more information about this error, try `rustc --explain E0425`.

No thing there, it went out of scope on line 10. Importantly, thing goes out of scope before its ownership is transfered, so Rust drops it: the compiler inserts code to clean up all resources associated with thing and frees its memory. We can hook into this mechanism by providing an implementation of the special Drop trait:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
impl Drop for Thing {
    fn drop(&mut self) {
        eprintln!("dropping thing {}", self.number);
    }
}

fn main() {
    let a = 4;

    if a > 3 {
        let thing = Thing { number: a };
        eprintln!("inside scope");
    } // `thing` dropped here
    eprintln!("outside scope");
}

If we run this we get:

inside scope
dropping thing 4
outside scope

Again, more details are in the book.

Managing resources

We’re going to try to encode higher level API constraints using the linear typing (ownership) semantics of Rust.

The interaction we’re going to describe is pretty simple: sending a letter via a delivery service. One has a written letter they’d like to send: they put it in a pre-stamped envelope, they close the envelope and they hand it to the lorry driver. Of course, all of this applies to many APIs: we’ll see a couple of examples at the end.

Here’s one way to model our protocol in Rust:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
#[derive(Clone)]
pub struct Letter {
    text: String,
}

pub struct Envelope {
    letter: Option<Letter>,
}

pub struct PickupLorryHandle {
    done: bool,
    // references to lorry's resources
}

impl Letter {
    pub fn new(text: String) -> Self { Letter { text: text } }
}

impl Envelope {
    /// Put a letter in the envelope and seal it.
    pub fn wrap(&mut self, letter: &Letter) {
        self.letter = Some(letter.clone());
    }
}

impl PickupLorryHandle {
    /// Give an envelope to the delivery driver.
    pub fn pickup(&mut self, envelope: &Envelope) {
        // (the details here don't matter)
    }

    /// Tell the driver we don't have anything else for them.
    pub fn done(&mut self) {
        self.done = true; println!("sent");
    }
}

pub fn order_pickup() -> PickupLorryHandle {
    // ...
}

With this in place we can write our client code:

1
2
3
4
5
6
7
8
9
// in a separate module
fn main() {
    let rustfest_letter = Letter::new(String::from("Dear RustFest"));
    let mut rustfest_envelope = buy_prestamped_envelope();
    rustfest_envelope.wrap(&rustfest_letter);
    let mut lorry = order_pickup();
    lorry.pickup(&rustfest_envelope);
    lorry.done();
}

Our client code:

writes a new letter (line 3),
buys a prestamped envelope (line 4),
puts the letter in the envelope and seals it (line 5),
orders a pickup from the delivery company (line 6),
hands the closed envelope to the driver (line 7),
tells the driver we don’t have anything else for them (line 8).

1, 2, 3

Our API has three shortcomings we can better address with Rust’s type system:

we’d like to prevent re-use of what we know it’s a finite resource: we only have one physical copy of the letter;
we want to make sure that we perform a series of steps in the right order (and only once): put the letter in the envelope, seal it, and give it to the driver (i.e. avoid giving an empty envelope);
we don’t want to forget to release a resource when we’re done: we ensure we tell the driver they can leave.

Use a resource only once

Here’s some problematic client code:

1
2
3
4
5
6
7
8
9
10
11
fn main() {
    let rustfest_letter = Letter::new(String::from("Dear RustFest"));
    let mut envelopes = vec![
        buy_prestamped_envelope(), buy_prestamped_envelope()];
    let mut lorry = order_pickup();
    for e in envelopes.iter_mut() {
        e.wrap(&rustfest_letter);
        lorry.pickup(&e);
    }
    lorry.done();
}

No compiler error, but somehow the letter was magically duplicated and inserted in both envelopes. Sometimes, this is perfectly fine (copying some memory isn’t a big deal), but sometimes the resource represented by our struct cannot be easily duplicated: in this example, if it’s representing a constraint in our business logic. In general, if our struct represents a handle to a resource out of our control, we may not be able to clone it without breaking some safety or correctness guarantees.

So let’s remove the Clone implementation for Letter, and adjust the client code:

1
2
3
4
5
6
7
8
9
10
#[derive(Clone)]
pub struct Letter {
    text: String,
}

impl Envelope {
    pub fn wrap(&mut self, letter: Letter) { // take ownership of `letter`
        self.letter = Some(letter);
    }
}

1
2
3
4
5
6
7
8
9
10
11
fn main() {
    let rustfest_letter = Letter::new(String::from("Dear RustFest"));
    let mut envelopes = vec![
        buy_prestamped_envelope(), buy_prestamped_envelope()];
    let mut lorry = order_pickup();
    for e in envelopes.iter_mut() {
        e.wrap(rustfest_letter); // give ownership of `rustfest_letter`
        lorry.pickup(&e);
    }
    lorry.done();
}

Here’s the compiler output:

error[E0382]: use of moved value: `rustfest_letter`
  --> letter1.rs:7:16
   |
 7 |         e.wrap(rustfest_letter);
   |                ^^^^^^^^^^^^^^^ value moved here in previous iteration of loop
   |
   = note: move occurs because `rustfest_letter` has type `Letter`,
           which does not implement the `Copy` trait

error: aborting due to previous error

For more information about this error, try `rustc --explain E0382`.

Great! We can now only use each letter once!

Enforce order

One issue down, two to go. We’d like to make sure that the steps of the protocol are carried out in the proper order: we must not forget to put the letter in the envelope before handing it to the lorry driver! And, can we prevent inserting two letters in the same envelope at compile time?

Here’s the broken client code:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
fn main() {
    let rustfest_letter = Letter::new(String::from("Dear RustFest"));
    let mut first_envelope = buy_prestamped_envelope();
    first_envelope.wrap(rustfest_letter);

    let eth_letter = Letter::new(String::from("Dear ETH"));
    first_envelope.wrap(eth_letter);

    let mut lorry = order_pickup();
    lorry.pickup(&first_envelope);

    let another_envelope = buy_prestamped_envelope();
    lorry.pickup(&another_envelope);

    lorry.done();
}

1
2
3
4
5
6
impl Envelope {
    pub fn wrap(&mut self, letter: &Letter) {
        assert!(self.letter.is_none());
        self.letter = Some(letter.clone());
    }
}

This compiles, but of course the assert in wrap will fire, at runtime.

thread 'main' panicked at 'assertion failed: self.letter.is_none()'
note: Run with `RUST_BACKTRACE=1` for a backtrace.

We’d like to prevent this at compile time. And once that’s fixed, what about making sure we don’t send empty envelopes (another_envelope on lines 12-13 of main)?

We can make a couple of classes to represent the current state of the Envelope: EmptyEnvelope is an empty pre-stampted envelope, and ClosedEnvelope is a closed envelope guaranteed to contain a letter. We can then only provide implementations for actions that make sense for that specific state. Then, to make sure we don’t send an empty envelope, we make sure that pickup only takes a ClosedEnvelope (and we make it take ownership, to avoid spurious copies).

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
/// An empty pre-stamped envelope.
pub struct EmptyEnvelope { }

/// A closed envelope containing a letter.
pub struct ClosedEnvelope {
    letter: Letter,
}

impl EmptyEnvelope {
    /// Put a letter in the envelope and seal it.
    pub fn wrap(self, letter: Letter) -> ClosedEnvelope {
        ClosedEnvelope { letter: letter }
    }
}

impl PickupLorryHandle {
    /// Give an envelope to the delivery driver.
    pub fn pickup(&mut self, envelope: ClosedEnvelope) {
        /* give letter */
    }

    ...
}

pub fn buy_prestamped_envelope() -> EmptyEnvelope { EmptyEnvelope { } }

And here’s the updated client code:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
fn main() {
    let rustfest_letter = Letter::new(String::from("Dear RustFest"));
    let mut first_envelope = buy_prestamped_envelope();
    first_envelope.wrap(rustfest_letter);

    let eth_letter = Letter::new(String::from("Dear ETH"));
    first_envelope.wrap(eth_letter);

    let mut lorry = order_pickup();
    lorry.pickup(first_envelope);

    let another_envelope = buy_prestamped_envelope();
    lorry.pickup(another_envelope);

    lorry.done();
}

error[E0308]: mismatched types
  --> letter2.rs:10:18
   |
10 |     lorry.pickup(first_envelope);
   |                  ^^^^^^^^^^^^^^ expected struct `ClosedEnvelope`,
   |                                 found struct `EmptyEnvelope`
   |
   = note: expected type `ClosedEnvelope`
              found type `EmptyEnvelope`

error[E0308]: mismatched types
  --> letter2.rs:13:18
   |
13 |     lorry.pickup(another_envelope);
   |                  ^^^^^^^^^^^^^^^^ expected struct `ClosedEnvelope`,
   |                                      found struct `EmptyEnvelope`
   |
   = note: expected type `ClosedEnvelope`
              found type `EmptyEnvelope`

error: aborting due to 2 previous errors

For more information about this error, try `rustc --explain E0308`.

One problem prevented: no empty envelopes can be sent! Note how the compiler errors point us towards a solution: we need a ClosedEnvelope for the lorry to pickup. Let’s fix the client code again.

1
2
3
4
5
6
7
8
9
10
11
12
13
fn main() {
    let rustfest_letter = Letter::new(String::from("Dear RustFest"));

    let envelope = buy_prestamped_envelope();
    let closed_envelope = envelope.wrap(rustfest_letter);

    let eth_letter = Letter::new(String::from("Dear ETH"));
    let closed_envelope = envelope.wrap(eth_letter);

    let mut lorry = order_pickup();
    lorry.pickup(closed_envelope);
    lorry.done();
}

error[E0382]: use of moved value: `envelope`
  --> letter2.rs:8:27
   |
 5 |     let closed_envelope = envelope.wrap(rustfest_letter);
   |                           -------- value moved here
...
 8 |     let closed_envelope = envelope.wrap(eth_letter);
   |                           ^^^^^^^^ value used here after move
   |
   = note: move occurs because `envelope` has type `EmptyEnvelope`,
           which does not implement the `Copy` trait

error: aborting due to previous error

For more information about this error, try `rustc --explain E0382`.

By making EmptyEnvelope take self’s ownership we can use the linear typing technique from earlier to prevent reuse of envelope: once we’ve put a letter in an envelope, we get back a ClosedEnvelope, that can only be handed over to the lorry driver. Now the compiler can help us make sure we follow the protocol steps in order: put the letter in the envelope, then send it.

We’ll see a more complex example in which we compose an http response in the right order using this technique. For now, here’s the correct client code with the new API:

1
2
3
4
5
6
7
8
9
fn main() {
    let rustfest_letter = Letter::new(String::from("Dear RustFest"));
    let mut envelope = buy_prestamped_envelope();
    let closed_envelope = envelope.wrap(rustfest_letter);

    let mut lorry = order_pickup();
    lorry.pickup(closed_envelope);
    lorry.done();
}

Ensure a resource is released

Another common mistake: forgetting to release a resource. In the following client code, lorry.done() is missing, and we never tell the delivery driver we’re done. And, in this tortured methaphor, we’d never deliver the letter because the driver never leaves…

1
2
3
4
5
6
7
8
fn main() {
    let rustfest_letter = Letter::new(String::from("Dear RustFest"));
    let mut envelope = buy_prestamped_envelope();
    let closed_envelope = envelope.wrap(rustfest_letter);

    let mut lorry = order_pickup();
    lorry.pickup(closed_envelope);
}

In the real world, we may be keeping a connection open, or never completing a process in our business logic; and it’s just because we forgot a single line.

We’ve seen that we can hook into Rust’s drop mechanism; here’s how we’d ensure that we release the lorry:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
impl Drop for PickupLorryHandle {
    fn drop(&mut self) {
        if !self.done {
            self.done();
        }
    }
}

impl PickupLorryHandle {
    /// Tell the driver we don't have anything else for them.
    pub fn done(self) {
        self.done = true; println!("sent");
    }

    ...
}

1
2
3
4
5
6
7
8
9
10
11
fn main() {
    let rustfest_letter = Letter::new(String::from("Dear RustFest"));

    let envelope = buy_prestamped_envelope();
    let closed_envelope = envelope.wrap(rustfest_letter);

    let mut lorry = order_pickup();
    lorry.pickup(closed_envelope);

    // `lorry` dropped here
}

Now when the lorry goes out of scope the drop implementation is called, and we automatically send the driver on their merry way.

In the `std` library

This last pattern is often used in the std library when building abstractions where there’s a need to release a resource or a handle to a resource.

Rc, “a single-threaded reference-counting pointer”, tracks the number of references to a resource, and the ownership of the Rc pointer represents a strong reference: when no references are left (all Rcs are dropped), the underlying resource (memory and other elements of the struct) should be freed.

There’s some unsafe trickery under the hood, but what’s relevant here is that Rc uses Drop to release the underlying memory once the reference-count reaches zero.

Here’s the relevant code: std::rc::Rc.

unsafe impl<#[may_dangle] T: ?Sized> Drop for Rc<T> {
    // ...
    fn drop(&mut self) {
        unsafe {
            self.dec_strong();
            if self.strong() == 0 {
                // destroy the contained object
                ptr::drop_in_place(self.ptr.as_mut());
                // ...
            }
        }
    }
}

Let’s look at another example in std: Mutex, “a mutual exclusion primitive useful for protecting shared data”.

Locking a Mutex returns a MutexGuard, “an RAII implementation of a scoped lock of a mutex”, that is, an object that represents the fact that we’re holding the lock. And Drop is used so we can release the lock just by dropping the guard (either explicitly, with std::mem::drop, or when it goes out of scope).

This is the signature of Mutex::lock:

pub fn lock(&self) -> LockResult<MutexGuard<T>>

And here’s the Drop implementation for MutexGuard: std::sync::MutexGuard.

#[stable(feature = "rust1", since = "1.0.0")]
impl<'a, T: ?Sized> Drop for MutexGuard<'a, T> {
    #[inline]
    fn drop(&mut self) {
        unsafe {
            self.__lock.poison.done(&self.__poison);
            self.__lock.inner.raw_unlock();
        }
    }
}

Example: http response

Now let’s look at an example of how all of these techniques can be combined in a realistic API. Let’s say we’re building an http server library, and we want to make sure that when we write our response we first send all the headers, and only then start writing the body. We may also want to make sure not to forget to flush the buffer at the end.

We use two structs to represent the two states: HttpResponseWritingHeader, and HttpResponseWritingBody. The method body on HttpResponseWritingHeader takes ownership of self, ensuring that the header writer can no longer be used after this call (its ownership is transfered), and we can only append additional chunks to the response body.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
pub struct HttpResponseWritingHeaders { /* connection, … */ }

pub struct HttpResponseWritingBody { /* ... */ }

pub fn start_response() -> HttpResponseWritingHeaders { /* ... */ }

impl HttpResponseWritingHeaders {
    fn header(&mut self, header: Header) { /* ... */ }

    fn body(self) -> HttpResponseWritingBody { /* ... */ }
}

impl HttpResponseWritingBody {
    fn write(&mut self, chunk: Chunk) { /* ... */ }

    fn cease(self) { }
}

impl Drop for HttpResponseWritingBody {
    fn drop(&mut self) {
        self.flush();
    }
}

The Drop implementation ensures that a completed response is always fully written out to the client.

State explosion

Note that the technique of representing the state of a protocol/object with many structs has a significant drawback when the possible state space grows large: we may end up juggling a lot of structs, and the benefit of compile time checks may not be worth the cost in lines-of-code, and maintainability.

Example: streaming engine

At the Systems Group of the ETH Zürich CS department, we’re working on timely dataflow, a low-latency cyclic dataflow computational model: it lets you describe computations as a cyclic graph of nodes (operators), that perform some transformation on incoming data and maintain local state, and edges (channels) that carry data between operators. A computation built this way can be automatically parallelised across cores and computers, over the network.

In timely dataflow, data is transported on channels as tuples, each carrying a logical timestamp.

Timestamps are used to represent logical boundaries between groups of tuples so that operations can be performed on some subset, and timely dataflow tracks which timestamps are in flight in the system. Importantly, for timely dataflow’s correctness, operators are only allowed to send messages with timestamp t when they hold a Capability for the same timestamp t. In addition, they need to report whenever they relinquish one of these capabilities, so that the system can make forward progress (that is, a downstream operator can then process all the tuples associated with the capability’s timestamp).

In timely, we use something like the following to represent a Capability, a resource which grants permission to send data at a certain timestamp.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
/// The capability to send data with a certain timestamp on a dataflow edge.
pub struct Capability<T: Timestamp> {
    time: T,
    internal: Rc<RefCell<ChangeBatch<T>>>,
}

impl<T: Timestamp> Clone for Capability<T> {
    fn clone(&self) -> Capability<T> {
        // … update self.internal …
    }
}

impl<T: Timestamp> Drop for Capability<T> {
    fn drop(&mut self) {
        // … update self.internal …
    }
}

/// Return a channel handle to send data at the timestamp carried by `cap`.
pub fn session(&mut self, cap: &Capability<T>) -> Handle

Capability has a custom Clone implementation, that keeps track of how many copies we’ve made (a bit like Rc), and a Drop implementation, so we can be sure that a Capability is dropped if not explicitly retained. The function that an operator uses to write to its output requires a reference to a valid Capability for a certain timestamp: this way we enforce, at compile time, one of the protocol constraints of timely dataflow.

Thanks for reading! You can send me notes and questions on twitter. There’s a lobste.rs and a reddit post for comments.

RumbleML, a declarative machine learning framework

RumbleML, a declarative machine learning framework

Introduction

Data Independence

Scalability

Enter Rumble

Standing on the Shoulders of Giants: RumbleML

Transformers

Estimators

Conclusion

Why should you try Rumble & RumbleML?

References

Building an inverted index on a large text collection with JSONiq

The Corpus

Reading a large collection of strings

Tokenizing

Getting the distinct terms

Postings lists

Querying the index

JSONiq and data independence

Rumble, an engine to run JSONiq on top of Spark

Data independence

Is the Spark API functional?

Enter data frames

Heterogeneous and nested data

JSONiq: a language that natively processes nested, heterogeneous data

Rumble: pushing down the computations to Spark

Open-source download

The design and implementation of a lock-free ring-buffer with contiguous reservations

Circular buffers

An idealised infinite buffer

A bounded circular buffer

Contiguous writes/reads

A Hardware Interlude

DMA - Direct Memory Access

Stackless Operation

But why is DMA so important?

From embedded to datacenters

A fork in the road

Concurrency design

High watermark for data

Implementation

Writing

Reading

A note on memory ordering

Support for embedded systems

Academics Should Build Their Own Computers to Advance Systems Research

A fork() in the road

String interning and beyond, in differential dataflow

Strings do work!

Interning Strings

Interning Strings IN DIFFERENTIAL DATAFLOW

Using interned strings

Arrangements

Update (“The next day”)

Row Identifiers IN DIFFERENTIAL DATAFLOW

Decomposing records

String interning

Making joins better

Performance Update (“A few days later”)

Baseline performance

Switching to differences

Optimizing iteration

Physical Adressing on Real Hardware in Isabelle/HOL

Introduction

The decoding net

Model refinement

References

A hammer you can only hold by the handle

Affine type systems

Drop

Managing resources

1, 2, 3

Use a resource only once

Enforce order

Ensure a resource is released

In the std library

Example: http response

State explosion

Example: streaming engine

In the `std` library