28 KiB

Raw Permalink Blame History

Author: Erik Johnston
Created: 2018-07-20
Updated:
- #1693: Clarify how to handle rejected events ─ Erik Johnston, 2018-10-30

State Resolution: Reloaded

Thoughts on the next iteration of the state resolution algorithm that aims to mitigate currently known attacks

Background

The state of a room at an event is a mapping from key to event, which is built up and updated by sending state events into the room. All the information about the room is encoded in the state, from metadata like the name and topic to membership of the room to security policies like bans and join rules.

It is therefore important that─wherever possible─the view of the state of the room is consistent across all servers. If different servers have different views of the state then it can lead to the room bifurcating, due to differing ideas on who is in the room, who is allowed to talk, etc.

The difficulty comes when the room DAG forks and then merges again (which can happen naturally if two servers send events at the same time or when a network partition is resolved). The state after the merge has to be resolved from the state of the two branches: the algorithm to resolve this is called the state resolution algorithm.

Since the result of state resolution must be consistent across servers, the information that the algorithm can use is strictly limited to the information that will always be available to all servers (including future servers that may not even be in the room at that point) at any point in time where the resolution needs to be calculated. In particular, this has the consequence that the algorithm cannot use information from the room DAG, since servers are not required to store events for any length of time.

As such, the state resolution algorithm is effectively a pure function from sets of state to a single resolved set of state.

The final important property for state resolution is that it should not allow malicious servers to avoid moderation action by forking and merging the room DAG. For example, if a server gets banned and then forks the room before the ban, any merge back should always ensure that the ban is still in the state.

Current Algorithm

The current state resolution is known to have some undesirable properties, which can be summarized into two separate cases:

Moderation evasion ─ where an attacker can avoid e.g. bans by forking and joining the room DAG in particular ways.
State resets ─ where a server (often innocently) sends an event that points to disparate parts of the graph, causing state resolution to pick old state rather than later versions.

These have the following causes:

Conflicting state must pass auth checks to be eligible to be picked, but the algorithm does not consider previous (superseded) state changes in a fork. For example, where Alice gives Bob power and then Bob gives Charlie power on one branch of a conflict, when the latter power level event is authed against the original power level (where Bob didn't have power), it fails.
The algorithm relies on the deprecated and untrustable depth parameter to try and ensure that the "most recent" state is picked. Without having a copy of the complete room DAG the algorithm doesn't know that e.g. one topic event came strictly after another in the DAG. For efficiency and storage reasons servers are not required (or expected) to store the whole room DAG.
The algorithm always accepts events where there are no conflicting alternatives in other forks. This means that if an admin changed the join rules to private, then new joins on forks based on parts of the DAG which predate that change would always be accepted without being authed against the join_rules event.

Desirable Properties

As well as the important properties listed in the "Background" section, there are also some other properties that would significantly improve the experience of end users, though not strictly essential. These include:

Banning and changing power levels should "do the right thing", i.e. end users shouldn't have to take extra steps to make the state resolution produce the "right" results.
Minimise occurrences of "state resets". Servers will sometimes point to disparate parts of the room DAG (due to a variety of reasons), which ideally should not result in changes in the state.
Be efficient; state resolution can happen a lot on some large rooms. Ideally it would also support efficiently working on "state deltas" - i.e. the ability to calculate state resolution incrementally from snapshots rather than having to consider the full state of each fork each time a conflict is resolved

Ideas for New Algorithm

Auth Chain

The auth events of a given event is the set of events which justify why a given event is allowed to be sent into a room (e.g. an m.room.create, an m.room.power_levels and the sender's m.room.membership). The auth chain of an event is its auth events and their auth events, recursively. The auth chains of a set of events in a given room form a DAG.

"Auth events" are events that can appear as auth events of an event. These include power levels, membership etc.¹

Servers in a room are required to have the full auth chain for all events that they have seen, and so the auth chain is available to be used by state resolution algorithms.

Unconflicted State

The current algorithm defines the notion of "unconflicted state" to be all entries that for each set of state either has the same event or no entry. All unconflicted state entries are included in the resolved state. This is problematic due to the fact that any new entries introduced on forks always appear in the resolved state, regardless of if they would pass the checks applied to conflicted state.

The new algorithm could redefine "unconflicted state" to be all entries which both exist and are the same in every state set (as opposed to previously where the entry didn't need to exist in every state set).

Replacing Depth

Since depth of an event cannot be reliably calculated without possessing the full DAG, and cannot be trusted when provided by other servers, it can not be used in future versions of state resolution. A potential alternative, however, is to use "origin_server_ts". While it cannot be relied on to be accurate─an attacker can set it to arbitrary values─it has the advantage over depth that end users can clearly see when a server is using incorrect values. (Note that server clocks don't need to be particularly accurate for the ordering to still be more useful than other arbitrary orderings).

It can also be assumed that in most cases the origin_server_ts for a given benign server will be mostly consistent. For example, if a server sends a join and then a leave in the vast majority of cases the leave would have a greater origin_server_ts.

This makes "origin_server_ts" a good candidate to be used as a last resort to order events if necessary, where otherwise a different arbitrary ordering would be used. However, it's important that there is some other mechanism to ensure that malicious servers can't abuse origin_server_ts to ensure their state always gets picked during resolution (In the proposal below we use the auth DAG ordering to override users who set state with malicious origin_server_ts.)

Ordering and Authing

Roughly, the current algorithm tries to ensure that moderation evasion doesn't happen by ordering conflicted events by depth and (re)authing them sequentially. The exact implementation has several issues, but the idea of ensuring that state events from forks still need to pass auth subject to e.g. bans and power level changes is a powerful one, as it reduces the utility of maliciously forking.

For that to work we need to ensure that there is a suitable ordering that puts e.g. bans before events sent in other forks. (However events can point to old parts of the DAG, for a variety of reasons, and ideally in that case the resolved state would closely match the recent state).

Similarly care should be taken when multiple changes to e.g. power levels happen in a fork. If Alice gives Bob power (A), then Bob gives Charlie power (B) and then Charlie, say, changes the ban level (C). If you try and resolve two state sets one of which has A and the other has C, C will not pass auth unless B is also taken into account. This case can be handled if we also consider the difference in auth chains between the two sets, which in the previous example would include B.

(This is also the root cause of the "Hotel California" issue, where left users get spontaneously rejoined to rooms. This happens when a user has a sequence of memberships changes of the form: leave (A), join (B) and then another leave (C). In the current algorithm a resolution of A and C would pick A, and a resolution of A and B would then pick B, i.e. the join. This means that a suitably forked graph can reset the state to B. This is fixed if when resolving A and C we also consider B, since its in the auth chain of C.)

Power Level Ordering

Actions that malicious servers would try and evade are actions that require greater power levels to perform, for example banning, reducing power level, etc. We define "power events" as events that have the potential to remove the ability of another user to do something.² (Note that they are a subset of auth events.)

In all these cases it is desirable for those privileged actions to take precedence over events in other forks. This can be achieved by first considering "power events", and requiring the remaining events to pass auth based on them.

Mainline

An issue caused by servers not storing the full room DAG is that one can't tell how two arbitrary events are ordered. The auth chain gives a partial ordering to certain events, though far from complete; however, all events do contain a reference to the current power levels in their auth events. As such if two state events reference two different power levels events, and one power levels' auth chain references the other, then there is a strong likelihood that the event referencing the latter power level came after the other event.

A "mainline" is a list of power levels events created if you pick a particular power levels event (usually the current resolved power level) and recursively follow each power level referenced in auth_events back to the first power level event.

The mainline can then be used to induce an ordering on events by looking at where the power level referenced in their auth_events is in the mainline (or recursively following the chain of power level events back until one is found that appears in the mainline). This effectively partitions the room into epochs, where a new epoch is started whenever a new power level is sent.

If this mainline ordering is combined with ordering by origin_server_ts, then it gives an ordering that is correct for servers that don't lie about the time, while giving a mechanism that can be used to deal if a server lied (by room admins starting a new epoch).

The natural course of action for a room admin to take when noticing a user/server is misbehaving is to ban them from the room, rather than changing the power levels. It would therefore be useful if banning a user or server started a new epoch as well. This would require being able to create a mainline that includes power level events and bans³, which would suggest that power level and ban events would need to point to the latest ban event as well. (This would be significantly easier if we maintained a list of bans in a single event, however there is concern that would limit the number of possible bans in a room.)

Proposed Algorithm

First we define:

"State sets" are the sets of state that the resolution algorithm tries to resolve, i.e. the inputs to the algorithm.
"Power events" are events that have the potential to remove the ability of another user to do something. These are power levels, join rules, bans and kicks.
The "unconflicted state map" is the state where the value of each key exists and is the same in every state set. The "conflicted state map" is everything else. (Note that this is subtly different to the definition used in the existing algorithm, which considered the merge of a present event with an absent event to be unconflicted rather than conflicted)
The "auth difference" is calculated by first calculating the full auth chain for each state set and taking every event that doesn't appear in every auth chain.
The "full conflicted set" is the union of the conflicted state map and auth difference.
The "reverse topological power ordering"⁴ of a set of events is an ordering of the given events, plus any events in their auth chains that appear in the auth difference, topologically ordered by their auth chains with ties broken such that x < y if:
1. x's sender has a greater power level than y (calculated by looking at their respective auth events, or if
2. x's origin_server_ts is less than y's, or if
3. x's event_id is lexicographically less than y's
This is also known as a lexicographical topological sort (i.e. this is the unique topological ordering such that for an entry x all entries after it must either have x in their auth chain or be greater than x as defined above). This can be implemented using Kahn's algorithm.
The "mainline ordering" based on a power level event P of a set of events is calculated as follows:
1. Generate the list of power levels starting at P and recursively take the power level from its auth events. This list is called the mainline, ordered such that P is last.
2. We say the "closest mainline event" of an event is the first power level event encountered in mainline when iteratively descending through the power level events in the auth events.
3. Order the set of events such that x < y if:
  1. The closest mainline event of x appears strictly before the closest of y in the mainline list, or if
  2. x's origin_server_ts is less than y's, or if
  3. x's event_id lexicographically sorts before y's
The "iterative auth checks" algorithm is where given a sorted list of events, the auth check algorithm is applied to each event in turn. The state events used to auth are built up from previous events that passed the auth checks, starting from a base set of state. If a required auth key doesn't exist in the state, then the one in the event's auth_events is used if the auth event is not rejected. (See Variations and Attack Vectors below).

The algorithm proceeds as follows:

Take all power events and any events in their auth chains that appear in the full conflicted set and order them by the reverse topological power ordering.
Apply the iterative auth checks algorithm based on the unconflicted state map to get a partial set of resolved state.
Take all remaining events that weren't picked in step 1 and order them by the mainline ordering based on the power level in the partially resolved state.
Apply the iterative auth checks algorithm based on the partial resolved state.
Update the result with the unconflicted state to get the final resolved state⁵. (Note: this is different from the current algorithm, which considered different event types at distinct stages)

An example python implementation can be found on github here.

Note that this works best if we also change which events to include as an event's auth_events. See the "Auth Events" section below.

Discussion

Essentially, the algorithm works by producing a sorted list of all conflicted events (and differences in auth chains), and applies the auth checks one by one, building up the state as it goes. The list is produced in two parts: first the power events and auth dependencies are ordered by power level of the senders and resolved, then the remaining events are ordered using the "mainline" of the resolved power levels and then resolved to produce the final resolved state.

(This is equivalent to linearizing the full conflicted set of events and reapplying the usual state updates and auth rules.)

Variations

There are multiple options for what to use as the base state for iterative auth checks algorithm; while it needs to be some variation of auth events and unconflicted events, it is unclear exactly which combination is best (and least manipulatable by malicious servers).

Care has to be taken if we want to ensure that old auth events that appear in the auth chain difference can't supersede unconflicted state entries.

Due to auth chain differences being added to the resolved states during iterative auth checks, we therefore need to re-apply the unconflicted state at the end to ensure that they appear in the final resolved state. This feels like an odd fudge that shouldn't be necessary, and may point to a flaw in the proposed algorithm.

State Resets

The proposed algorithm still has some potentially unexpected behaviour.

One example of this is when Alice sets a topic and then gets banned. If an event gets created (potentially much later) that points to both before and after the topic and ban then the proposed algorithm will resolve and apply the ban before resolving the topic, causing the topic to be denied and dropped from the resolved state. This will result in no topic being set in the resolved state.

Auth Events

The algorithm relies heavily on the ordering induced by the auth chain DAG.

There are two types of auth events (not necessarily distinct):

Those that give authorization to do something
Those that revoke authorization to do something.

For example, invites/joins are in the former category, leaves/kicks/bans are in the latter and power levels are both.

Assuming⁶ revocations always point to (i.e., have in their auth chain) the authorization event that they are revoking, and authorization events point to revocations that they are superseding, then the algorithm will ensure that the authorization events are applied in order (so generally the "latest" authorization state would win).

This helps ensure that e.g. an invite cannot be reused after a leave/kick, since the leave (revocation) would have the invite in their auth chain.

This idea also relies on revocations replacing the state that granted authorization to do an action (and vice versa). For example, in the current model bans (basically) revoke the ability for a particular user from being able to join. If the user later gets unbanned and then rejoins, the join would point to the join rules as the authorization that lets them join, but would not (necessarily) point to the unban. This has the effect that if a state resolution happened between the new join and the ban, the unban would not be included in the resolution and so the join would be rejected.

The changes to the current model that would be required to make the above assumptions true would be, for example:

By default permissions are closed.
Bans would need to be a list in either the join rules event or a separate event type which all membership events pointed to.
Bans would only revoke the ability to join, not automatically remove users from the room.
Change the defaults of join_rules to be closed by default

Efficiency and Delta State Resolution

The current (unoptimised) implementation of the algorithm is 10x slower than the current algorithm, based on a single, large test case. While hopefully some optimisations can be made, the ability to incrementally calculate state resolution via deltas will also mitigate some of the slow down.

Another aspect that should be considered is the amount of data that is required to perform the resolution. The current algorithm only requires the events for the conflicted set, plus the events from the unconflicted set needed to auth them. The proposed algorithm also requires the events in the auth chain difference (calculating the auth chain difference may also require more data to calculate).

Delta state resolution is where if you have, say, two state sets and their resolution, then you can use that result to work out the new resolution where there has been a small change to the state sets. For the proposed algorithm, if the following properties hold true then the result can be found by simply applying steps 3 and 4 to the state deltas. The properties are:

The delta contains no power events
The origin_server_ts of all events in state delta are strictly greater than those in the previous state sets
Any event that has been removed must not have been used to auth subsequent events (e.g. if we replaced a member event and that user had also set a topic)

These properties will likely hold true for most state updates that happen in a room, allowing servers to use this more efficient algorithm the majority of the time.

Full DAG

It's worth noting that if the algorithm had access to the full room DAG that it would really only help by ensuring that the ordering in "reverse topological ordering" and "mainline ordering" respected the ordering induced by the DAG.

This would help, e.g., ensure the latest topic was always picked rather than rely on origin_server_ts and mainline. As well as obviate the need to maintain a separate auth chain, and the difficulties that entails (like having to reapply the unconflicted state at the end).

Rejected Events

Events that have been rejected due to failing auth based on the state at the event (rather than based on their auth chain) are handled as usual by the algorithm, unless otherwise specified.

Note that no events rejected due to failure to auth against their auth chain should appear in the process, as they should not appear in state (the algorithm only uses events that appear in either the state sets or in the auth chain of the events in the state sets).

This helps ensure that different servers' view of state is more likely to converge, since rejection state of an event may be different. This can happen if a third server gives an incorrect version of the state when a server joins a room via it (either due to being faulty or malicious). Convergence of state is a desirable property as it ensures that all users in the room have a (mostly) consistent view of the state of the room. If the view of the state on different servers diverges it can lead to bifurcation of the room due to e.g. servers disagreeing on who is in the room.

Intuitively, using rejected events feels dangerous, however:

Servers cannot arbitrarily make up state, since they still need to pass the auth checks based on the event's auth chain (e.g. they can't grant themselves power levels if they didn't have them before).
For a previously rejected event to pass auth there must be a set of state that allows said event. A malicious server could therefore produce a fork where it claims the state is that particular set of state, duplicate the rejected event to point to that fork, and send the event. The duplicated event would then pass the auth checks. Ignoring rejected events would therefore not eliminate any potential attack vectors.

Rejected auth events are deliberately excluded from use in the iterative auth checks, as auth events aren't re-authed (although non-auth events are) during the iterative auth checks.

Attack Vectors

The main potential attack vector that needs to be considered is in the iterative auth checks algorithm, and whether an attacker could make use of the fact that it's based on the unconflicted state and/or auth events of the event.

Appendix

The following are some worked examples to illustrate some of the mechanisms in the algorithm. In each we're interested in what happens to the topic.

Example 1 - Mainline

The following is an example room DAG, where time flows down the page. We shall work through resolving the state at both Message 2 and Message 3.

(Note that green circles are events sent by Alice, blue circles sent by Bob and black arrows point to previous events. The red arrows are the mainline computed during resolution.)

First we resolve the state at Message 2. The conflicted event types are the power levels and topics, and since the auth chains are the same for both state sets the auth difference is the empty set.

Step 1: The full conflicted set are the events _P2, P3, Topic 2 _and Topic 3, of which P2 and P3 are the only power events. Since Alice (the room creator) has a greater power level than Bob (and neither _P2 _and P3 appear in each other's auth chain), the reverse topological ordering is: [P2, P3].

Step 2: Now we apply the auth rules iteratively, P2 trivially passes based on the unconflicted state, but P3 does not pass since after P2 Bob no longer has sufficient power to set state. This results in the power levels resolving to P2.

Step 3: Now we work out the mainline based on P2, which is coloured in red on the diagram. We use the mainline to order Topic 2 and Topic 3. Topic 2 points to_ P1_, while the closest mainline to Topic 3 is also P1. We then order based on the origin_server_ts of the two events, let's assume that gives us: [Topic 2, Topic 3].

Step 4: Iteratively applying the auth rules results in Topic 2 being allowed, but _Topic 3 _being denied (since Bob doesn't have power to set state anymore), so the topic is resolved to Topic 2.

This gives the resolved state at Message 2 to be _P2 _and Topic 2. (This is actually the same result as the existing algorithm gives)

Now let's look at the state at Message 3.

Step 1: The full conflicted set are simple: Topic 2 and Topic 4. There are no conflicted power events.

Step 2: N/A

Step 3: Topic 2 points to P1 in the mainline, and Topic 4 points to P2 in its auth events. Since P2 comes after P1 in the mainline, this gives an ordering of [Topic 2, Topic 4].

Step 4: Iteratively applying the auth rules results in both topics passing the auth checks, and so the last topic, Topic 4, is chosen.

This gives the resolved state at Message 3 to be Topic 4.

Example 2 - Rejected Events

The following is an example room DAG, where time flows down the page. The event D is initially rejected by the server (due to not passing auth against the state), but does pass auth against its auth chain.

(Note that the blue lines are the power levels pointed to in the event's auth events)

At F we first resolve the power levels, which results in E. When we then go to resolve the topics against the partially resolved state, Bob has ops and so the resolved state includes the topic change D, even though it was initially rejected.

Notes

In the current room protocol these are: the create event, power levels, membership, join rules and third party invites. See the spec. ↩︎
In the current protocol these are: power levels, kicks, bans and join rules. ↩︎
Future room versions may have a concept of server ban event that works like existing bans, which would also be included ↩︎
The topology being considered here is the auth chain DAG, rather than the room DAG, so this ordering is only applicable to events which appear in the auth chain DAG. ↩︎
We do this so that, if we receive events with misleading auth_events, this ensures that the unconflicted state at least is correct. ↩︎
This isn't true in the current protocol ↩︎

28 KiB Raw Permalink Blame History