6.6 KiB
MSC4231: Backwards compatibility for media captions
Problem
MSC2530 introduced the ability to use the body field
on file transfers as a caption. This merged and was shipped in Matrix 1.10, and we're now seeing more clients sending
captions in the wild.
Unfortunately, any client which is not "caption-aware" (i.e. has yet to implement
MSC2530 or Matrix 1.10) does not know to display the
body field as a caption - and so these messages effectively get silently dropped, fragmenting Matrix as a
communication medium. Given captions typically contain as much important information as any other message, this can
result in bad communication failures, and a very negative perception of Matrix's reliability.
We should have specified a means of backwards compatibility to avoid breaking communication between newer and older clients during the window in which we wait for clients to upgrade to Matrix 1.10.
Proposal
Clients should send a separate m.room.message event after the captioned media, including the caption as the body,
and replying to the media event. This is referred to as a caption fallback event.
The content block of the caption fallback event includes an m.caption_fallback: true field, so that caption-aware
clients do not display this event, instead displaying the media event's body field as a caption per
MSC2530.
However, caption-unaware clients will display the event as a reply to the media and so avoid discarding the contents of the caption, while associating it visually with the original media via the reply.
If a user on a caption-aware client edits their caption, their client should update both the media event and the caption fallback with the edit.
If a user on a caption-aware client redacts their media, their client should redact its caption fallback too.
If a user on a caption-unaware client edits or redacts a caption fallback sent on a caption-aware client, then the fallback will drift out of sync with the caption on the media event - see Outstanding Issues below.
The event contains an m.relates_to field of type m.caption_fallback in order to associate the fallback to the media
event, and so make it easy to locate when a caption-aware client applies edits or redactions. This also stops clients
trying to start threads from the caption fallback, as the server will reject the invalid thread. The end result looks
like this:
"type": "m.room.message",
"content": {
"body": "Caption text",
"msgtype": "m.text",
"m.relates_to": {
"event_id": "$(some image event)",
"rel_type": "m.caption_fallback",
"m.in_reply_to": {
"event_id": "$OYKwuL..."
},
}
},
If non-caption-aware users reply to a caption fallback, then caption-aware clients should display the media event as the event being replied to.
Outstanding issues
If a user on a caption-unaware client edits a caption fallback sent on a caption-aware client, then this change will not be visible to caption-aware clients, causing inconsistent history between caption-aware and unaware clients.
If a user on a caption-unaware client redacts a caption fallback sent on a caption-aware client, then the caption in the media event won't be redacted, potentially leaking the redacted content.
Clients or bridges that are caption-aware but not MSC4231-aware capable will display or transport the text content twice, displaying double content to the user.
Potential issues
It's a bit ugly and redundant to duplicate the caption in the fallback event as well as the media event. However, it's way worse to drop messages.
The fact that caption fallback events will be visible to some clients and invisible to others might highlight unread state/count problems. However, given we need to handle invisible events already, it's not making the problem worse - and in fact by making it more obvious, might help fix any remaining issues in implementations.
Alternatives
Captions should be provided by extensible events. However, until extensible events are fully rolled out, we're stuck with fixing up the situation with MSC2530, and this is a problem which is playing out right now on the public network.
Alternatively, we could ignore the issue and go around upgrading as many clients as possible to speak MSC2530. However, this feels like incredibly bad practice, given we have a trivial way to provide backwards compatibility, and in practice we shouldn't be forcing clients to upgrade in order to avoid losing messages when we could have avoided it in the first place.
This has ended up combining both MSC2530 and MSC2529. There's a world where the fallback event could be the primary source of truth for the caption, and meanwhile the field on the media event be the 'fallback' for the convenience of bridges.
Alternatively, we could change to sending captions entirely as relations, as in MSC2529, and require bridges to wait for the caption event (if flagged on the media event) before they send on the media event. This would avoid needing a dedicated caption fallback event - as the caption would have its own event anyway. It would also avoid the risk of edits and redactions getting out of sync between the media event and the caption fallback. This feels like it might be a preferable approach, given the outstanding issues above. It does however travel in the opposite direction to extensible events (where the caption would be a mixin on the media event).
Security considerations
The caption in the fallback may not match the caption in the media event, causing confusion between caption-aware and caption-unaware clients. From a trust & safety perspective, the caption in the fallback might contain abusive content not visible to human moderators because their caption-aware clients hide the fallback (and vice versa, for caption-unaware clients).
Sending two events (media + caption) in quick succession will make event-sending rate limits kick in more rapidly. In practice this feels unlikely to be a problem.
Unstable prefix
m.caption_fallback would be org.matrix.msc4231.caption_fallback until this merges.
Dependencies
None, given MSC2530 has already merged.