matrix-doc/proposals/3706-partial-state-in-send-join.md

MSC3706: Extensions to /_matrix/federation/v2/send_join/{roomId}/{eventId} for partial state

Background

It is well known that joining large rooms over federation can be very slow (see, for example, synapse#1211).

Much of the reason for this is the large number of events which are returned by the /send_join API, and must be validated and stored.

MSC3902 gives an overview of changes to the matrix protocol to improve this situation. This MSC focuses on a specific aspect of those suggestions by proposing specific changes to the /send_join API.

Proposal

PUT /_matrix/federation/v2/send_join/{roomId}/{eventId} is extended to support "partial state" in its responses. This involves the following changes.

New query parameter

omit_members is added as a new query parameter. This can take the values true or false; other values should be rejected with an HTTP 400 error with matrix error code M_INVALID_PARAM.

Calling servers use this parameter to indicate support for partial state in send_join. If it is not set to true, receiving servers continue to behave as they do today.

Changes to the response

Receiving servers are not obliged to implement partial state: they are also free to support it for some rooms and not others.

The following changes are made to the response:

• members_omitted: a new boolean field is added. This should be set to true to indicate that m.room.member events have been omitted from the response. It must otherwise be set to false or omitted.

• state: if partial state is being returned, then state events with event type m.room.member are omitted from the response. All other room state is returned as normal. (See 'Alternatives' for discussion on why only m.room.member events are omitted.)

• auth_chain: The spec currently defines this as "The auth chain for the entire current room state". We instead rephrase this as:

  All events in the auth chain for the returned join event, as well as those in the auth chains for any events returned in state.

  (Note that in the case that full state is being returned, the two definitions are equivalent.)

• If the omit_members query parameter was set, we make a further optimisation to auth_chain:

  Any events returned within state can be omitted from auth_chain.

  For example: the m.room.create event is part of the room state, so must be included in state. However, it also forms part of the auth chain for all of the returned events, so in the current spec, must also be included in auth_chain. However, this is redundant, so we should omit it for calling servers which opt into that via the omit_members query param.

• servers_in_room: A new field of type [string], listing the servers active in the room (ie, those with joined members) before the join.

  This is to be used by the joining server to send outgoing federation transactions while it synchronises the full state, as outlined in MSC3902.

  This field is required if the members_omitted response field is true; it is otherwise optional.

Potential issues

None at present.

Alternatives

• In future, the list of event types to omit could be expanded. (Some rooms may have large numbers of other state events).

  Currently, m.room.member events are by far the biggest problem. For example, a /send_join request for Matrix HQ returns approximately 85000 events in state, of which all but 44 are m.room.member.

  Additionally: the client-server API already provides a mechanism for omitting m.room.member events from the /sync response, via lazy loading, which means that this change can be implemented without changes to the client-server API. Extending the list of event types to be omitted would require changes to the client-server API and therefore be significantly more involved.

  In order to reduce the scope of the change, we have therefore decided to focus on m.room.member events for now. Future MSCs might provide a mechanism for omitting other event types.

Security considerations

No security issues are currently foreseen with this specific MSC, though the larger topic of incremental synchronisation of state has several concerns; these will be discussed in other MSCs such as MSC3902.

Unstable prefix

The following mapping will be used for identifiers in this MSC during development:

Proposed final identifier	Purpose	Development identifier
omit_members	query parameter	org.matrix.msc3706.partial_state
members_omitted	response field	org.matrix.msc3706.partial_state
servers_in_room	response field	org.matrix.msc3706.servers_in_room

Dependencies

This MSC does not build on any existing unaccepted MSCs.