378 lines
12 KiB
Markdown
378 lines
12 KiB
Markdown
# MSC2448: Using BlurHash as a Placeholder for Matrix Media
|
|
|
|
[BlurHash](https://blurha.sh) is a compact representation of a placeholder
|
|
for an image (or a frame of video). Currently, in Matrix, clients must
|
|
display a placeholder image in the message timeline while a piece of media is
|
|
loading. Some clients, such as Element, simply display an empty space.
|
|
|
|
While thumbnails exist to combat this to some degree, they still need to be
|
|
downloaded from a homeserver, which is not instantaneous.
|
|
|
|
Instead, a BlurHash can be sent inside events, which upon
|
|
receipt other clients can render for a pretty media preview while the actual
|
|
thumbnail downloads. They also do not contain any `"` characters, making them
|
|
simple to stick inside existing JSON blobs.
|
|
|
|
To be clear: A BlurHash does not replace a thumbnail - rather it is a placeholder
|
|
and should be shown before the thumbnail is downloaded. For a familiar messaging
|
|
user experience, clients are recommended to first render and display a blurhash,
|
|
then download the thumbnail of the media. Once the thumbnail file is downloaded,
|
|
display it. Finally, the user can optionally view the full media item by selecting it.
|
|
|
|
## Proposal
|
|
|
|
### m.room.message
|
|
|
|
An optional field is added in `m.room.message`'s `content.info` dictionary
|
|
with the key `blurhash`. It is a BlurHash of the original piece of media.
|
|
Clients could then render this using [one of the available BlurHash
|
|
implementations](https://github.com/woltapp/blurhash).
|
|
|
|
This would be optionally displayed while the thumbnail of the media is loaded
|
|
in parallel.
|
|
|
|
Example `m.room.message` content:
|
|
|
|
```json
|
|
{
|
|
"body": "image.png",
|
|
"info": {
|
|
"size": 149234,
|
|
"mimetype": "image/png",
|
|
"thumbnail_info": {
|
|
"w": 301,
|
|
"h": 193,
|
|
"mimetype": "image/png",
|
|
"size": 72958
|
|
},
|
|
"w": 301,
|
|
"h": 193,
|
|
"thumbnail_url": "mxc://example.org/abcdefg",
|
|
"blurhash": "JadR*.7kCMdnj"
|
|
},
|
|
"msgtype": "m.image",
|
|
"url": "mxc://example.org/abcde"
|
|
}
|
|
```
|
|
|
|
Note that a BlurHash representation is really only applicable to media, and
|
|
as such should only be used in conjunction with the following
|
|
`m.room.message` msgtypes:
|
|
|
|
* `m.image`
|
|
* `m.video`
|
|
|
|
### m.sticker
|
|
|
|
An optional field is added to `m.sticker`'s `content.info` dictionary with
|
|
the key `blurhash`. Its value is a BlurHash of the sticker media.
|
|
|
|
Example `m.sticker` content:
|
|
|
|
```json
|
|
{
|
|
"body": "Landing",
|
|
"info": {
|
|
"h": 200,
|
|
"mimetype": "image/png",
|
|
"size": 73602,
|
|
"thumbnail_info": {
|
|
"h": 200,
|
|
"mimetype": "image/png",
|
|
"size": 73602,
|
|
"w": 140
|
|
},
|
|
"thumbnail_url": "mxc://matrix.org/sHhqkFCvSkFwtmvtETOtKnLP",
|
|
"w": 140,
|
|
"blurhash": "JadR*.7kCMdnj"
|
|
},
|
|
"url": "mxc://matrix.org/sHhqkFCvSkFwtmvtETOtKnLP"
|
|
}
|
|
```
|
|
|
|
### m.room.avatar
|
|
|
|
Room avatars having BlurHashes available will be especially useful when
|
|
viewing a server's Public Rooms directory.
|
|
|
|
An optional field is added to `m.room.avatar`'s `content.info` dictionary with the
|
|
key `blurhash`. Its value is a BlurHash of the media that is pointed to by
|
|
`url`.
|
|
|
|
Example `m.room.avatar` content:
|
|
|
|
```json
|
|
{
|
|
"url": "mxc://matrix.example.com/a59ee02f180677d83d1b57d366127f8e1afdd4ed",
|
|
"info": {
|
|
"blurhash": "JadR*.7kCMdnj"
|
|
}
|
|
}
|
|
```
|
|
|
|
### m.room.member
|
|
|
|
Much like room avatars, user avatars can have BlurHashes as well. There is a
|
|
little more required to implement this, but the outcome of no longer having
|
|
missing avatars upon opening a room is worthwhile.
|
|
|
|
An optional field is added to `m.room.member`'s `content` dictionary with
|
|
the key `blurhash`. Its value is a BlurHash of the media that is pointed
|
|
to by `avatar_url`.
|
|
|
|
Note that `blurhash` MUST be omitted if `avatar_url` is not present.
|
|
|
|
Example `m.room.member` event content:
|
|
|
|
```json
|
|
{
|
|
"avatar_url": "mxc://example.org/SEsfnsuifSDFSSEF",
|
|
"displayname": "Alice Margatroid",
|
|
"membership": "join",
|
|
"blurhash": "JadR*.7kCMdnj"
|
|
}
|
|
```
|
|
|
|
### Profile endpoints
|
|
|
|
Endpoints that return profile information, and thus MXC URLs to user avatars, are
|
|
extended to optionally include BlurHashes as well.
|
|
|
|
[`GET /_matrix/client/v3/profile/{userId}`](https://spec.matrix.org/v1.3/client-server-api/#get_matrixclientv3profileuserid)
|
|
has an optional field added with
|
|
the key `blurhash`. Its value is a BlurHash of the media that is pointed to
|
|
by `avatar_url`. `blurhash` MUST be omitted if `avatar_url` is not present.
|
|
The same applies
|
|
to [`GET /_matrix/client/v3/profile/{userId}/avatar_url`](https://spec.matrix.org/v1.3/client-server-api/#get_matrixclientv3profileuseridavatar_url)
|
|
, and to the federation
|
|
endpoint [`GET /_matrix/federation/v1/query/profile`](https://spec.matrix.org/v1.3/server-server-api/#get_matrixfederationv1queryprofile)
|
|
.
|
|
|
|
#### Example responses:
|
|
|
|
`GET /_matrix/client/v3/profile/{userId}`
|
|
|
|
```json5
|
|
{
|
|
"avatar_url": "mxc://matrix.org/SDGdghriugerRg",
|
|
"displayname": "Alice Margatroid",
|
|
"blurhash": "oyp8ky2BWn7VHEL"
|
|
}
|
|
```
|
|
|
|
`GET /_matrix/federation/v1/query/profile`
|
|
|
|
```json5
|
|
{
|
|
"avatar_url": "mxc://matrix.org/SDGdghriugerRg",
|
|
"displayname": "Alice Margatroid",
|
|
"blurhash": "oyp8ky2BWn7VHEL"
|
|
}
|
|
```
|
|
|
|
`GET /_matrix/client/v3/profile/{userId}/avatar_url`
|
|
|
|
```json5
|
|
{
|
|
"avatar_url": "mxc://matrix.org/SDGdghriugerRg",
|
|
"blurhash": "oyp8ky2BWn7VHEL"
|
|
}
|
|
```
|
|
|
|
Separately, [`PUT /_matrix/client/v3/profile/{userId}/avatar_url`](https://spec.matrix.org/v1.3/client-server-api/#put_matrixclientv3profileuseridavatar_url)
|
|
has an optional field added
|
|
to the request body with the key `blurhash`. Its value is a BlurHash of the media that is pointed to by the value of
|
|
the `avatar_url` field in the same request.
|
|
|
|
#### Example request/response interaction
|
|
|
|
Request:
|
|
```
|
|
PUT /_matrix/client/v3/profile/{userId}/avatar_url HTTP/1.1
|
|
|
|
{
|
|
"avatar_url": "mxc://matrix.org/SDGdghriugerRg",
|
|
"blurhash": "oyp8ky2BWn7VHEL"
|
|
}
|
|
```
|
|
|
|
Successful response:
|
|
|
|
```
|
|
200 OK
|
|
|
|
{}
|
|
```
|
|
|
|
### URL previews
|
|
|
|
An optional attribute is added to the OpenGraph data returned by a call
|
|
to
|
|
[`GET /_matrix/media/v3/preview_url`](https://spec.matrix.org/v1.3/client-server-api/#get_matrixmediav3preview_url)
|
|
called `matrix:image:blurhash`. The value
|
|
of this attribute is the blurhash representation of the media specified
|
|
by `og:image`. Note that we place this value under the `matrix` namespace as
|
|
the [OpenGraph protocol](https://ogp.me/) does not (yet) have a spec'd field
|
|
for BlurHashes.
|
|
|
|
Note that `matrix:image:blurhash` MUST be omitted if `og:image` is not present
|
|
in the response.
|
|
|
|
Example response to `GET /_matrix/media/v3/preview_url`:
|
|
|
|
```json
|
|
{
|
|
"og:title": "Matrix Blog Post",
|
|
"og:description": "This is a really cool blog post from matrix.org",
|
|
"og:image": "mxc://example.com/ascERGshawAWawugaAcauga",
|
|
"og:image:type": "image/png",
|
|
"og:image:height": 48,
|
|
"og:image:width": 48,
|
|
"matrix:image:size": 102400,
|
|
"matrix:image:blurhash": "oyp8ky2BWn7VHEL"
|
|
}
|
|
```
|
|
|
|
### Inline images
|
|
|
|
An optional attribute is added to `<img>` tags in messages:
|
|
`data-mx-blurhash`, where the value of the attribute is the blurhash
|
|
representation of the inline image.
|
|
|
|
This would be optionally displayed while the inline image itself is loaded in
|
|
parallel.
|
|
|
|
Example `m.room.message.formatted_body`:
|
|
|
|
```
|
|
"formatted_body": This is awesome <img alt=\"flutterjoy\" title=\"flutterjoy\" height=\"32\" src=\"mxc://matrix.example.org/abc\" data-mx-blurhash=\"LEHV6nWB2yk8pyo\" />
|
|
```
|
|
|
|
## Calculating a blurhash on the server
|
|
|
|
BlurHashes are inserted into events by the client, however some clients may not
|
|
be able to implement the BlurHash library for whatever reason. In this case, it
|
|
would be nice to allow the media repository to calculate the BlurHash of a piece
|
|
of media for the client, similar to how thumbnails are calculated by media
|
|
repositories today.
|
|
|
|
A new boolean query parameter, `generate_blurhash`, is added to
|
|
[`POST /_matrix/media/v3/upload`](https://spec.matrix.org/v1.3/client-server-api/#post_matrixmediav3upload)
|
|
which allows clients to ask the server to generate a BlurHash for them. This
|
|
may be useful for clients that are designed to run on very low-power hardware,
|
|
or otherwise cannot generate a BlurHash itself.
|
|
If set to `true`, the server SHOULD generate a BlurHash of the uploaded media
|
|
and return it as the value of the `blurhash` key in the response.
|
|
|
|
If the server cannot generate a BlurHash of the media - perhaps because it is
|
|
not a file that a BlurHash can be derived from or it is too expensive to process
|
|
a very large piece of media - then the server SHOULD NOT return a `blurhash` key
|
|
in the response. Additionally, if `generate_blurhash` is not `true`, then the
|
|
server SHOULD NOT return a `blurhash` key in the response.
|
|
|
|
Fundamentally, this means that clients SHOULD NOT assume that a server will always
|
|
return a BlurHash in the response to `/_matrix/media/v3/upload`, even if they have
|
|
set the `generate_blurhash` query parameter to `true` in the request.
|
|
|
|
An example request from a client that would like the server to generate a
|
|
blurhash would look like:
|
|
|
|
```
|
|
POST /_matrix/media/v3/upload?generate_blurhash=true&filename=My+Family+Photo.jpeg HTTP/1.1
|
|
Content-Type: image/jpeg
|
|
|
|
<bytes>
|
|
```
|
|
|
|
Example response:
|
|
|
|
```
|
|
{
|
|
"content_uri": "mxc://example.com/abcde123",
|
|
"blurhash": "LKO2?U%2Tw=w]~RB"
|
|
}
|
|
```
|
|
|
|
We explicitly make this behaviour opt-in as it is assumed that the majority of
|
|
clients that end up supporting BlurHashes will be capable of generating them
|
|
locally. Thus, the less load we can put on the homeserver (by not making
|
|
blurhash generation the default) the better.
|
|
|
|
Note that media servers will not be able to return a BlurHash string for
|
|
encrypted media; that must be left to the client.
|
|
|
|
The server could additionally return the BlurHash string for an image when
|
|
given an MXC URL. This would be through something like the Media Information
|
|
API (specified in
|
|
[MSC2380](https://github.com/matrix-org/matrix-doc/pull/2380)), or similar.
|
|
|
|
## Visualisation
|
|
|
|
Viewing an image message that is loading:
|
|
|
|
|
|
|
|
Once the image loads:
|
|
|
|
|
|
|
|
As a sample, this is Element's behaviour prior to this MSC's introduction:
|
|
|
|
|
|
|
|
## Alternatives
|
|
|
|
We could include a base64 thumbnail of the image in the event, but blurhash
|
|
produces much more efficient textual representations.
|
|
|
|
## Backwards compatibility
|
|
|
|
Older clients would ignore the new `blurhash` parameter.
|
|
|
|
Newer clients would only show it if it exists.
|
|
|
|
Users who have not specified `blurhash` in their `m.room.member` event yet may
|
|
stand out from users who have while both are loading. This is entirely up to
|
|
clients to handle, though a suggestion may be to "fake" a blurhash by
|
|
blurring some placeholder image (perhaps something with variation between
|
|
users like Element's coloured backgrounds with letters in them, or [an
|
|
identicon](https://en.wikipedia.org/wiki/Identicon) derived from the user's
|
|
ID) until the user's actual avatar loads.
|
|
|
|
## Unstable prefixes
|
|
|
|
Implementations wishing to add this before this MSC is merged can do so with
|
|
the following:
|
|
|
|
* The `blurhash` key in any events, request or response bodies should be
|
|
replaced with `xyz.amorgan.blurhash`.
|
|
|
|
* `/_matrix/media/v3/upload`'s new `generate_blurhash` query parameter
|
|
should instead be `xyz.amorgan.generate_blurhash`. And instead of the
|
|
key `blurhash`, the endpoint should return `xyz.amorgan.blurhash`.
|
|
|
|
* The `data-mx-blurhash` attribute in `<img>` tags should be replaced with
|
|
`data-xyz-amorgan-blurhash`.
|
|
|
|
## Security considerations
|
|
|
|
BlurHash entries in encrypted events, be it as part of the `info` property,
|
|
or `<img>` tags, should be encrypted along with the rest of the event
|
|
content.
|
|
|
|
A [discussion in
|
|
#matrix-spec](https://matrix.to/#/!NasysSDfxKxZBzJJoE:matrix.org/$Cfa0dtF3DenIUAbC5aeg3Xo10gAF54mAJLZ6VzvYNfo?via=matrix.org&via=amorgan.xyz&via=pixie.town)
|
|
questioned whether massive BlurHashes may be a potential DoS vector for
|
|
clients. The discussion found that only a maximum of 100 x 100 components can
|
|
be defined by a BlurHash. This may be in the higher range for low-resource
|
|
(or unoptimised) clients, and clients are free to refuse to render a BlurHash
|
|
with a large component count, but it shouldn't be a cause for concern.
|
|
|
|
Invalid BlurHashes should not be rendered.
|
|
|
|
## Links
|
|
|
|
BlurHash's algorithm description can be found
|
|
[here](https://github.com/woltapp/blurhash/blob/master/Algorithm.md), which
|
|
also includes the full output character set.
|