matrix-doc/proposals/1951-custom-emoji-and-stickers.md

Custom emoji and sticker packs in Matrix

This proposal is an iteration on MSC1256 which uses pre-existing Matrix concepts (namely rooms) to achieve the same goals.

Having the ability to use custom emojis and stickers are often requested in various support channels, and the show of support on MSC1256 in +1's alone reinforces just how much this feature is desired by the community. Through some experimentation by the author, it is already possible to create custom sticker packs when using the author's integration manager (Dimension) and dedicated bot - this experimentation is a large driving factor for this proposal.

For scope, this proposal aims to give users the ability to define their own custom emoji and sticker packs, share them, and actually use them. Rooms should also be able to recommend custom emoji and sticker packs for members of the room to use. In future, communities/groups should be able to do the same thing.

Some issues, related projects, and documentation on the matter are:

• https://github.com/vector-im/riot-web/issues/2950
• https://github.com/vector-im/riot-web/issues/2648
• https://github.com/vector-im/riot-web/issues/5157#issuecomment-340364320
• https://github.com/matrix-org/matrix-doc/pull/1174 - n.b. this is incorporated in this proposal.
• https://github.com/turt2live/matrix-dimension/blob/master/docs/custom_stickerpacks.md - documentation on using Dimension to achieve this, with links to matrix-sticker-manager which handles the curation aspect. Implementation is within the same repo.

Proposal

Custom emoji and sticker packs are the same thing technically, just handled differently in the client. Stickers in this proposal are the same m.stickers found in the current Client-Server API specification. Emoji in this proposal are intended to be rendered by clients in a similar fashion to how they'd render regular unicode emoji in terms of sizing, positioning, etc. A "pack" is a collection of either emoji or stickers.

Packs are to be mapped to dedicated rooms with the following semantics:

• The room name is the pack's display name.
• The room topic is the pack's description.
• The room avatar is a thumbnail for the pack. If not present, the first sticker/emoji should be used.
• The join rules and guest access control who is able to see the pack. For example, an invite only room would not be sharable without inviting people directly.
• Specific state events (defined later) represent other metadata for the pack, such as the stickers/emoji and who created it.
• The power levels define who is able to edit the pack.
• Deleting a pack would be done by making the room invite only and kicking everyone.

The room is just a plain Matrix room, therefore concepts like ACLs and upgrades still apply. As a note for implementations, pack metadata should be transferred when room upgrades happen to preserve the pack. Implementations should use the event timeline as a sort of comment section for the pack.

Sharing packs with other users

Sharing is achieved by sharing a link to the room, or inviting someone. There is no explicit grammar for what the room's alias should be as the presence of the pack metadata state events is an indicator that the room is a pack.

Sharing can also be done through distributing plain http or https URLs which are able to point to the room in which the pack is defined. The intent is that 3rd party services can create sticker packs and distribute user friendly or SEO optimized URLs, such as https://packs.example.org/1234/official-sheltie-pack. When approached with Accept: application/json or have .json appended to the end of the URL, the following JSON body should be returned:

{
    "room_uri": "https://matrix.to/#/%23_stickerpack_example%3Aexample.org"
}

The room_uri is a URI which references a Matrix room. Currently that means a matrix.to URI however with future proposals that could be mx:// or similar style schemes. When not approached with the intent of JSON, the response is up to the application serving the content. For example, this would be a good opportunity to render the sticker pack for viewing by a human.

Recommending which packs to use in a room

Rooms can recommend which packs to use for members viewing the room by setting a state event which looks like the following:

{
  "type": "m.room.recommended_packs",
  "sender": "@travis:t2l.io",
  "content": {
    "packs": [
      "https://matrix.to/#/%23_stickerpack_sample1%3Aexample.org",
      "https://matrix.to/#/%23_stickerpack_sample2%3Aexample.org"
    ]
  },
  "state_key": "",
  "origin_server_ts": 1554344486692,
  "event_id": "$D4h43bYVTf48tDTfkja6ROV13yJrzGUNGj7pySOVj6E",
  "room_id": "!gdRMqOrTFdOCYHNwOo:example.org"
}

If a client is making use of an embedded picker (such as the stickerpicker widget), it will need to send this information to the picker when it is opened. Building on MSC1236, when a widget is approved the m.sticker capability, a toWidget action of recommended_packs is sent roughly around the same time a visibility action would be sent to show the picker. The data of recommended_packs is:

{
    "room_id": "!gdRMqOrTFdOCYHNwOo:example.org",
    "packs": [
      "https://matrix.to/#/%23_stickerpack_sample1%3Aexample.org",
      "https://matrix.to/#/%23_stickerpack_sample2%3Aexample.org"
    ]
}

The room_id is the room ID where the packs are recommended and packs is verbatim from the m.room.recommended_packs state event content. The picker is then responsible for showing the packs to the user. The picker must acknowledge the recommended_packs action with an empty response object. Clients making use of their own picker are responsible for showing the packs to the user. In either case (built-in picker or widget), the application is not required to automatically show the whole pack to the user - a step where the user approves the use of the pack (potentially enabling it for all rooms) is encouraged.

There is no current specification for an emoji picker widget, however future proposals which add this are encouraged to support the recommended_packs action.

Tracking changes in sticker packs

To track changes to packs, the interested application would join the pack's room. In the case of clients using their own picker, the logged in user would join the room (potentially with the client's assistance through dedicated UI/UX) - the client should hide the pack's room from the user where applicable, such as by not showing it in the room list. Pickers would perform the same action, however instead of using the logged in user's account they would use a bot account in the background to keep track of the pack. How this is accomplished is left as an implementation detail.

How the application tracks changes is up to the application. For instance, this may be done with the /sync API or via Application Services.

State events in the pack room

There are two kinds of special events which appear in a pack's dedicated room: m.pack.metadata for the pack's metadata and zero or more m.pack.item for the stickers/emoji themselves. Both event types are state events. The presence of a m.pack.metadata event indicates that it is a pack room.

A m.pack.item event looks as follows:

{
  "type": "m.pack.item",
  "sender": "@travis:t2l.io",
  "content": {
    "uri": "mxc://example.org/media_id",
    "description": "This is where a short sentence explaining the sticker goes",
    "shortcodes": [
        ":sample:",
        ":cool_sticker:"
    ]
  },
  "state_key": "ThisIsTheItemIdToDistinguishItFromAnotherItem",
  "origin_server_ts": 1554344486692,
  "event_id": "$D4h43bYVTf48tDTfkja6ROV13yJrzGUNGj7pySOVj6E",
  "room_id": "!gdRMqOrTFdOCYHNwOo:example.org"
}

shortcodes are best explained as tags which can be used to find the item in the pack. They are arbitrary strings which clients (and pickers) should use to recommend things to the user. In the case of emoji, these will typically be surrounded in colons whereas stickers may simply have emoji like 🙂 and 😥. shortcodes are optional.

The description is required and should be a human representation of the emoji or sticker. The description should be a single short sentence, but has no length limit.

The uri is required and is a Matrix Content URI to the represented emoji or sticker.

The state_key of the item is the ID for the item. This is referenced by the m.pack.metadata event for listing which items are active.

Items which have empty content or do not meet the schema should be considered inactive, even if the m.pack.metadata event recommends otherwise. This allows for an item to be deleted more easily from the pack, with the understanding that state in Matrix is permanent. Redacted m.pack.item events should be handled in this same way.

A m.pack.metadata event looks as follows:

{
  "type": "m.pack.metadata",
  "sender": "@travis:t2l.io",
  "content": {
    "active_items": [
        "ThisIsTheItemIdToDistinguishItFromAnotherItem-1",
        "ThisIsTheItemIdToDistinguishItFromAnotherItem-2"
    ],
    "creator": "https://matrix.to/#/%40alice%3Aexample.org",
    "author": "https://matrix.to/#/%40alice%3Aexample.org",
    "license": "CC-BY-NC-SA-4.0",
    "kind": "m.stickers"
  },
  "state_key": "ThisIsTheItemIdToDistinguishItFromAnotherItem",
  "origin_server_ts": 1554344486692,
  "event_id": "$D4h43bYVTf48tDTfkja6ROV13yJrzGUNGj7pySOVj6E",
  "room_id": "!gdRMqOrTFdOCYHNwOo:example.org"
}

active_items (required) are the state_keys of the m.pack.item events which should be considered active in the pack, provided they meet the criteria previously mentioned.

The creator (required) is the person who created the pack, but not necessarily authored it. This is generally used by third party services which allow users to upload items which they have not created. This must be a valid http, https, or matrix.to (in future mx://) URI.

The author is optional and if not provided is assumed to be the same as the creator. This is the person who authored the items contained within the pack. For example, if the pack was commissioned by the creator, this would be the person who did the artwork. This must be a valid http, https, or matrix.to (in future mx://) URI.

The license is required and must be a valid SPDX identifier. This applies to all the items contained within the pack.

The kind must be either m.stickers for a sticker pack or m.emoji for emoji. This is the hint used by applications to determine how best to handle items.

While the presence of an m.pack.metadata event signifies that the room is a pack room, if the event is invalid then it should be treated as an error by applications.

Representation of packs on m.sticker events

To cover matrix-doc#1174, a new optional field is to be added to m.sticker events: pack_url. This is the sharable URI for the pack which contains the sticker, if the sticker is contained within a known pack. For example, this may be a matrix.to (or mx:// in future) URI or a URL which the application can resolve to a room as per earlier in this proposal.

Because access control is handled by Matrix itself, applications should not omit the field if the pack is private and instead send it anyways.

Representation of custom emoji in messages

Emoji are intended to be represented inline as <img> elements in the HTML body. For the plain text representation, the shortcode the user used (or the first one) should be used. Using the mxc:// URI for the item, the img element should look similar to:

<img src="mxc://example.org/media_id" width="16" height="16"
     alt="This is the item's description"
     data-mx-pack="https://matrix.to/#/%23_stickerpack_sample1%3Aexample.org" />

The src is the mxc:// URI of the item, and the alt is the description of the item. The width and height are provided as a fallback for clients which do not support custom emoji. Clients which do support custom emoji can ignore the dimensions and use whatever is appropriate for their purposes. The data-mx-pack attribute has the same requirements as the pack_url for stickers mentioned above.

Tradeoffs

This proposal does not define an emoji picker widget similar to the existing sticker picker widget. An emoji picker would be best served as a dedicated proposal due to the complexity involved in defining its behaviour, such as how it interacts with a client's potential autocomplete mechanics. Similarly, the sticker picker widget is minimally extended in this proposal to push the problem of enhanced client interaction to a future proposal. Clients are likely to want to handle custom emoji themselves anyhow for integration with their autocomplete mechanisms and for general performance.

Security considerations

Due to the heuristics of m.pack.metadata events, it is possible for moderators to hide rooms from existing members by applying the state event. Moderators already have several opportunities to damage a room, and are already considered in a place of trust to not do so maliciously - therefore, this risk is mitigated by having trusted moderators and administrators in a room, just like with any other new state event which would be added.

The sharable URL for a pack could leak information about who created it or what it contains, potentially subjecting the user to personal risk. It is expected that implementations handle this appropriately where possible, such as by giving the creator the option to create private packs with randomly generated room aliases to hide their contents from outsiders.