matrix-doc/proposals/2781-down-with-the-fallbacks.md

MSC2781: Remove reply fallbacks from the specification

Currently the specification suggests clients should send and strip a fallback representation of a replied to message. The fallback representation was meant to simplify supporting replies in a new client, but in practice they add complexity, are often implemented incorrectly and block new features.

This MSC proposes to remove those fallbacks from the specification.

Some of the known issues include:

• The content of reply fallback is untrusted.
• Reply fallbacks may leak history. (#368)
• Parsing reply fallbacks can be tricky. (#350)
• It is unclear how to handle a reply to a reply. (#372)
• Localization of replies is not possible when the content is embedded into the event.
• It is not possible to fully redact an event once it is replied to. This causes issues with Trust & Safety where spam or other removed content remains visible, and may cause issues with the GDPR Right to be Forgotten.
• There are a variety of implementation bugs related to reply fallback handling.

More details and considerations are provided in the appendices, but these are provided for convenience and aren't necessary to understand this proposal.

Proposal

Remove the rich reply fallback from the specification. Clients should stop sending them and should consider treating <mx-reply> parts as they treat other invalid html tags.

Clients are not required to include a fallback in a reply since version 1.3 of the specification. For this reason the reply fallback can be removed from the specification without any additional deprecation period.

A suggestion for the spec PR: An info box could be included to mention the historical use of the reply fallback, suggesting that clients may encounter such events sent by other clients and that clients may need to strip out such fallbacks.

Given clients have had enough time to implement replies completely, the overall look & feel of replies should be unchanged or even improved by this proposal. Implementing replies in a client should also be a bit easier with this change.

An extended motivation is provided at the end of this document.

Potential issues

Old events and events sent by clients implementing an older version of the Matrix specification might still contain a reply fallback. So for at least some period of time clients will still need to strip reply fallbacks from messages.

Clients which don't implement rich replies may see messages without context, confusing users. However, most replies are in close proximity to the original message, making context likely to be nearby. Clients should also have enough information in the event to render helpful indications to users while they work on full support.

Clients which aren't using intentional mentions may cause some missed notifications on the receiving side. MSC3664 and similar aim to address this issue, and MSC4142 tries to improve the intentional mentions experience for replies generally. Because intentional mentions are already part of the Matrix specification since version 1.7, clients can be expected to implement those first, which should make the impact on notifications minimal in practice.

Alternatives

MSC2589: This adds the reply text as an additional key. While this solves the parsing issues, it doesn't address the other issues with fallbacks.

One could also just stick with the current fallbacks and make all clients pay the cost for a small number of clients actually benefitting from them.

Lastly one could introduce an alternative relation type for replies without fallback and deprecate the current relation type (since it does not fit the new format for relations anyway). We could specify, that the server is supposed to send the replied_to event in unsigned to the client, so that clients just need to stitch those two events together, but don't need to fetch the replied_to event from the server. It would make replies slightly harder to implement for clients, but it would be simpler than what this MSC proposes.

Security considerations

Overall this should reduce security issues as the handling of untrusted HTML is simplified. For an example security issue that could be avoided, see https://github.com/vector-im/element-web/releases/tag/v1.7.3 and the appendix.

Unstable prefix

No unstable prefix should be necessary as clients aren't required to send reply fallbacks for all messages since version 1.3 of the Matrix specification, which changed the wording from "MUST" to "SHOULD".

Appendix A: Support for rich replies in different clients

Clients without rendering support for rich replies

Of the 23 clients listed in the Matrix client matrix 16 are listed as not supporting replies (updated January 2022):

• Element Android: Relies on the reply fallback.
• Element iOS: Does not support rich replies
• weechat-matrix: Actually has an implementation to send replies although it seems to be broken. Doesn't render rich replies. Hard to implement because of the single socket implementation, but may be easier in the Rust version.
• Quaternion: Blocked because of fallbacks.
• matrixcli: Doesn't support formatted messages.
• Ditto Chat: Seems to rely on the fallback
• Mirage: Supports rich replies, but doesn't strip the fallback correctly and uses the fallback to render them.
• Nio: Unsupported.
• Pattle: Client is not being developed anymore.
• Seaglass: Doesn't support rich replies, but is unhappy with how the fallback looks?
• Miitrix: Somewhat unlikely to support it, I guess?
• matrix-commander: No idea, but doesn't look like it.
• gotktrix: Seems to rely on the reply fallback
• Hydrogen: Seems to use the reply fallback
• kazv: Doesn't seem to support replies at all
• Syphon: Uses the reply fallback in body

So in summary, 3/4 of the listed clients don't support replies. At least one client doesn't support it because of the fallback (Quaternion). 3 of the command line clients probably won't support replies, since they don't support formatted messages and replies require html support for at least sending.

Only one client implemented rich replies in the last 1.5 years after the original list was done in October 2020. Other clients are either new in my list or didn't change their reply rendering. I would appreciate to hear, why those client developers decided not to support rich reply rendering and if dropping the reply fallback would be an issue for them.

Changes from 1.5 years ago as of January 2022:

• Fractal: Seems to support replies now!
• Commune: Seems to support rich reply rendering and style them very nicely.
• NeoChat: Supports rich replies
• Cinny: Seems to support rich replies
• gomuks: Strips the reply fallback
• Lots of other new clients!

Results of testing replies without fallback

So far I haven't found a client that completely breaks without the fallback. All clients that support rendering rich replies don't break, when there is no fallback according to my tests (at least Nheko, Element/Web, FluffyChat and NeoChat were tested and some events without fallback are in #nheko:nheko.im and I haven't heard of any breakage). Those clients just show the reply as normal and otherwise seem to work completely fine as well. Element Android and Element iOS just don't show what message was replied to. Other clients haven't been tested by the author, but since the content of an event is untrusted, a client should not break if there is no reply fallback. Otherwise this would be a trivial abuse vector.

Appendix B: Issues with the current fallbacks

This section was moved to the back of this MSC, because it is fairly long and exhaustive. It lists all the issues the proposal author personally experienced with fallbacks in their client and its interactions with the ecosystem.

Stripping the fallback

To reply to a reply, a client needs to strip the existing fallback of the first reply. Otherwise replies will just infinitely nest replies. While the spec doesn't necessarily require stripping the fallback in replies to replies (only for rendering), not doing so risks running into the event size limit, but more importantly, it just leads to a bad experience for clients actually relying on the fallback.

Stripping the fallback is not trivial. Multiple implementations had bugs in their fallback stripping logic. The edge cases are not covered in the specification in detail and some clients have interpreted them differently. Common mistakes include:

• Not stripping the fallback in body, which leads to a very long nested chain.
• Not dealing with mismatched <mx-reply> tags, which can look like you were impersonating someone.

For the body extra attention needs to be paid to only strip lines starting with > until the first empty line. Implementations either only stripped the first line, stripped all lines starting with > until the first non empty line, that does not start with > or stripped only the formatted_body. While those are implementation bugs, they can't happen if you don't need to strip a fallback.

Creating a new fallback

To create a new fallback, a client needs to add untrusted html to its own events. This is an easy attack vector to inject your own content into someone elses reply. While this can be prevented with enough care, since Riot basically had to fix this issue twice, it can be expected that other clients can also be affected by this.

Requirement of html for replies

The spec requires rich replies to have a fallback using html:

Rich replies MUST have a format of org.matrix.custom.html and therefore a formatted_body alongside the body and appropriate msgtype.

This means you can't reply using only a body and you can't reply with an image, since those don't have a formatted_body property currently. This means a text only client, that doesn't want to display html, still needs to support html anyway and that new features are blocked, because of fallbacks.

Format is unreliable

While the spec says how a fallback "should" look, there are variations in use which further complicates stripping the fallback or are common mistakes, when emitting the fallback. Some variations include localizing the fallback, missing suggested links or tags, using the body in replies to files or images or using the display name instead of the matrix id.

As a result the experience in clients relying on the fallback or stripping the fallback varies depending on the sending client.

Replies leak history

A reply includes the body of another event. This means a reply to an event can leak data to users, that joined this room at a later point, but shouldn't be able to see the event because of visibility rules or encryption. While this isn't a big issue, there is still an issue about it: https://github.com/matrix-org/matrix-doc/issues/1654

This history leak can also cause abusive or redacted messages to remain visible to other room members, depending on the client implementation of replies.

Historically clients have also sometimes localized the fallbacks. In those cases they leak the users language selection for their client, which may be personal information.

Using the unmodified fallback in clients and bridges

The above issues are minor, if reply fallbacks added sufficient value to clients. Bridges usually try to bridge to native replies, so they need to strip the reply fallback (https://github.com/matrix-org/matrix-doc/issues/1541). Even the IRC bridge seems to send a custom fallback, because the default fallback is not that welcome to the IRC crowd, although the use cases for simple, text only bridges is often touted as a good usecase for the fallback (sometimes even explicitly mentioning bridging to IRC). As a result there are very few bridges, that benefit from the fallback being present.

Some clients do choose not to implement rich reply rendering, but the experience tends to not be ideal, especially in cases where you reply to an image and now the user needs to guess, what image was being replied to.

As a result the fallbacks provide value to only a subset of the Matrix ecosystem.

Fallbacks increase integration work with new features

• Edits explicitly mention that a reply fallback should not be sent in the m.new_content. This causes issues for clients relying on the fallback, because they won't show replies once a message has been edited (see Element Android as a current example) and similar edge cases.
• Extensible events require an update to the specification for fallbacks (because there is no body or formatted_body anymore after the transition period). The current proposal also intends to just drop the fallbacks in extensible events.

Localization

Since the fallback is added as normal text into the message, it needs to be localized for the receiving party to understand it. This however proves to be a challenge, since users may switch languages freely in a room and it is not easy to guess, which language was used in a short message. One could also use the client's language, but that leaks the user's localization settings, which can be a privacy concern and the other party may not speak that language. Alternatively a client can just send english fallbacks, but that significantly worsens the experience for casual users in non-english speaking countries. The specification currently requires them to not be translated (although some clients don't follow that), but not sending a fallback at all completely sidesteps the need for the spec to specify that and clients relying on an english only fallback.