matrix
/
matrix.org
mirror of https://github.com/matrix-org/matrix.org.git
0
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
matrix.org/content/blog/2022/03/2022-03-04-this-week-in-mat...

33 KiB
Raw Permalink Blame History

+++ title = "This Week in Matrix 2022-03-04" path = "/blog/2022/03/04/this-week-in-matrix-2022-03-04"

[taxonomies] author = ["Thib"] category = ["This Week in Matrix"]

[extra] image = "https://matrix.org/blog/img/WwUSbKWYkIUPEOELmeJiPlYO.png" +++

Matrix Live 🎙

{{ youtube_player(video_id="IlOsvFNu88M") }}

Dept of Status of Matrix 🌡️

TravisR says

t2bot.io - a public integrations network for Matrix - has passed 1 Million known rooms and 8.3 Million bridged users. 10,000 of these rooms are attributed to Voyager (a bot which actively goes out to find rooms to map Matrix, with the map currently down for maintenance), leaving the remaining ones either bridged, previously bridged, or using a different integration offered by t2bot.io for free.

The 8.3 Million users are mainly Discord and Telegram users which have been brought over to Matrix through bridges. The stats say "excluding Twitter-bridged" because there's 424,832 old accounts from back in the day when t2bot.io had a free Twitter bridge available. To further break this down, about 6.8M are Telegram users (12% of Telegram) and 1.3M are Discord (<1% of Discord).

For perspective, t2bot.io has about 569 Million events stored in its database and sees approximately 30 thousand people bridged daily from the wider world into Matrix through its bridges.

This post is just a milestone update, but it also serves as a reminder that running your own server/bridges is also possible. In fact, it's even recommended to have better control over your own data and avoid latency issues that large providers, like t2bot.io, can unintentionally introduce. Synapse is relatively easy to set up with minimal sysadmin knowledge (guide), and there's always paid offerings like Element Matrix Services (for home and also for fun) and Beeper for a richer bridging experience than t2bot.io can feasibly provide.

TravisR adds

in the eternal battle of "t2bot.io should be successful" and "please oh god self-host", the above^

Thib reports

It is to be noted that eQualitie has set-up public Matrix and Element instances for Ukrainian people who are struggling right now. If you are in Ukraine or are in touch with Ukrainian people who need secure communications, you can show them https://kyiv.dcomm.net.ua, https://odessa.dcomm.net.ua, or https://kharkiv.dcomm.net.ua depending on their location so they get instructions on how to create an account on Matrix and stay in touch

Dept of Spec 📜

Kegan reports

MSC3575 Sliding Sync work is progressing, and a number of new features have been added to the MSC with implementations in the built-in web client in the proxy. A list of changes include:

  • spec/proxy: the ability to filter the room list by room name.
  • spec/proxy: Aninitial flag on the room to distinguish between updates and the initial sync for a given room. This can be used as a hint to clients to determine if they should update/replace their local data for that room.
  • proxy: more efficient algorithm for determining overlapping sets of ranges, resulting in fewer bytes over the wire when scrolling.
  • proxy: the client implementation now supports sending basic text messages as well as issuing/join,/invite,/leave commands.
  • proxy: the client implementation now has a developer HUD which tracks the Tx/Rx bytes as well as a visual representation of the sliding window.

The net result is a basic but incredibly low bandwidth syncing client: 60.79KB to download an entire initial sync on matrix.org. Further improvements will be done on the client to make sure it doesn't scale with the number of rooms on the user's account (it currently does because it naively adds a placeholder for each room in the list) to ensure it remains extremely snappy and a vision for what Sliding Sync can do in practice, right now. Please continue giving feedback on MSC3575 or in #sliding-sync:matrix.org as the API is still in development and will change depending on what clients require.

Andrew Morgan (Element) announces

Here's your weekly spec update! The heart of Matrix is the specification - and this is modified by Matrix Spec Change (MSC) proposals. Learn more about how the process works at https://matrix.org/docs/spec/proposals.

MSC Status

Created MSCs:

MSCs with newly proposed Final Comment Period:

  • No FCPs were proposed this week.

MSCs in Final Comment Period:

Closed MSCs:

Merged MSCs:

  • No MSCs were merged this week.

Spec Updates

Heads up! You may have noticed some changes to the matrix-org/matrix-doc github repository this week. The first major one being that it no longer exists!

We have separated it out into two repositories:

  • matrix-org/matrix-spec - the source text of the spec. Issues and pull requests filed here should relate to the source of the spec.
  • matrix-org/matrix-spec-proposals - all Matrix Spec Change proposals. New MSCs should be filed against this repository as pull requests. Issues should not be filed against this repository (and no, the issue tracker cannot be disabled, as it would cause transferred issue link redirects to 404).

The primary motivation for this was to separate concerns of interacting with matrix-org/matrix-doc. Those interacting with the spec's source would have to wade through long-lived spec proposals, while those only interested in spec proposals would need to filter out spec source files and other issues and PRs related to them. This also helps writing tooling against the repo either as it no longer needs to filter MSCs via issue labels.

Existing links to matrix-doc should be automatically redirected by github to the right place. MSC authors and developers should update their git remote settings to point to the new repos.

What happened in detail to achieve this was:

  • matrix-doc was renamed tomatrix-spec-proposals
  • matrix-spec was created
  • All files other than the "proposals" folder were transferred frommatrix-spec-proposals tomatrix-spec
  • All open issues frommatrix-spec-proposals were transferred tomatrix-spec

Further details are available here: https://github.com/matrix-org/matrix-spec/issues/927

Random MSC of the Week

The random MSC of the week is... MSC3383: Include destination in X-Matrix Auth Header!

This MSC asks for the simple change of adding a "destination" field to the Authorization header of federation requests. This would have the benefit of allowing reverse proxies - potentially handling traffic for multiple homeservers - to know where requests should be sent. Or for forward proxies, which could validate outgoing requests from a homeserver before they're sent.

There is currently some discussion on the fundamental idea of the MSCs, which will need to be resolved first before it can move forwards.

Dept of Servers 🏢

Synapse (website)

Synapse is the reference homeserver for Matrix

richvdh announces

Quite a varied bag this week! Firstly we cut a release candidate for Synapse 1.54.0, which we hope to release next week. As ever, testing and feedback is much appreciated.

Our switch to Poetry for dependency management has come a few steps closer, by removing tox from our Continuous Integration, and converting some of our ad-hoc scripts to proper Python entry points.

Meanwhile we've been continuing to knock out blockers to improving room join speeds, including better exception handling and logging, and some exciting work to reduce wasted work when a client disconnects while we're still processing a request.

We've also been taking a wider look at performance problems, such as slow /sync and /login times.

Dendrite (website)

Second generation Matrix homeserver

Kegan reports

I have been asked by numerous people over months if there are any kind of graphs comparing Synapse and Dendrite. Up until now, my answer has been "as far as I know, no". However, this got me thinking whether I can use Complement to engineer some kind of performance testing infrastructure. After hacking on it for a day or two, I can now share some of the work I've done in this area. There now exists two new binaries in the Complement repo: perftest and perfgraph. The former will run some tests and output a .json file. The latter will consume these files to generate .svg graphs. The tests performed include:

  • Creating a bunch of rooms with different users. (currently 10 users and 50 rooms)
  • Joining a bunch of rooms with different users.
  • Sending messages into rooms with different users. (currently 100 messages)
  • Doing an initial sync for all users.
  • Updating the display name of all users.
  • Doing an incremental sync for all users.

The metrics are collected using docker stats. This means the entire container is measured, not just the homeserver binary (e.g any daemons or assistive processes are included in the metrics) and includes:

  • CPU time consumed
  • Memory consumed
  • Bytes transmitted/received
  • Disk read/writes
  • Time taken

Here are some results:

This was a surprising find and probably indicates that Dendrite has a memory leak somewhere when creating events. However, the fun doesn't stop there. We can also use this tool to compare different versions of the same homeserver:

This allows us to quantify any performance improvements we've been working on, and you can clearly see how we're being more efficient with CPU in newer versions. But the fun doesn't stop there either. We can also compare different flavours of the same version of a homeserver:

You can quantify the cost of running sqlite or postgres for your particular installation.

Care must be taken when interpreting results. Lower doesn't always mean better. For example, a low readout on "create_users" may be a cause for concern as it might indicate that the server is not hashing passwords correctly (e.g using bcrypt/scrypt which are designed to consume CPU/memory). Servers which don't implement all of the specification will also be abnormally low on CPU/memory (e.g not having to calculate unread counts or handle push notifications naturally means less work to do so less resources consumed). Furthermore, these tests do not yet test federation, so expensive remote joins are not measured (remote joins require the joining server to check signatures, hashes, etc of a lot of events which doesn't happen for local servers). That being said, I have hopefully illustrated how useful this tool can be for both server developers trying to improve their software and server admins who want to use the right server for their hardware. The graphs are still a work-in-progress and there's a lot more that can be done in this area beyond a few days work, but it's a start.

neilalexander says

Today we've released Dendrite 0.6.5 which contains early push notification support as well as a number of fixes and improvements. This release includes the following changes:

  • Early support for push notifications has been added, with support for push rules, pushers, HTTP push gateways and the /notifications endpoint (contributions by danpe, PiotrKozimor and tommie)
  • Spaces Summary (MSC2946) is now correctly supported (when msc2946 is enabled in the config)
  • All media API endpoints are now available under the /v3 namespace
  • Profile updates (display name and avatar) are now sent asynchronously so they shouldn't block the client for a very long time
  • State resolution v2 has been optimised further to considerably reduce the number of memory allocations
  • State resolution v2 will no longer duplicate events unnecessarily when calculating the auth difference
  • The create-account tool now has a -reset-password option for resetting the passwords of existing accounts
  • The /sync endpoint now calculates device list changes much more quickly with less RAM used
  • The /messages endpoint now lazy-loads members correctly
  • Read receipts now work correctly by correcting bugs in the stream positions and receipt coalescing
  • Topological sorting of state and join responses has been corrected, which should help to reduce the number of auth problems when joining new federated rooms
  • Media thumbnails should now work properly after having unnecessarily strict rate limiting removed
  • The roomserver no longer holds transactions for as long when processing input events
  • Uploading device keys and cross-signing keys will now correctly no-op if there were no changes
  • Parameters are now remembered correctly during registration
  • Devices can now only be deleted within the appropriate UIA flow
  • The /context endpoint now returns 404 instead of 500 if the event was not found
  • SQLite mode will no longer leak memory as a result of not closing prepared statements

Spec compliance, as measured by Sytest, currently sits at:

  • Client-server APIs: 76%, up from 65% last time
  • Server-server APIs: 95%, up from 94% last time

As always, you can join us in #dendrite:matrix.org for Dendrite discussion and announcements.

Dept of Bridges 🌉

matrix-hookshot (website)

A multi purpose multi platform bridge, formerly known as matrix-github

Half-Shot reports

matrix-hookshot reaches 1.2.0

Hey folks, I'm back from holidays and I'm proud to say that hookshot is kicking out release after release. This release is especially large and contains a bounty of new features and fixes. Of note are:

  • JIRA Datacenter (On Premise) is now supported.
  • You can now configure fine grained permissions for users on the bridge by userId, homeserver domain or room membership (think spaces).
  • Generic webhooks has sprouted a versioned API, with v2 allowing for finer control over the output.
  • GitHub connections now include the closing comment when issues and PRs are closed.
  • GitHub connections will also notify a room when an existing issue has been relabled to one filtered by that room.
  • Figma now uses MSC3440 for comment threads.

But seriously go check out the release, there is way more there than I can include in this TWIM post. Happy webhookin, matrix gang!

In case you did not know, Hookshot can be installed via spantaleev's Ansible playbook. In addition to updates for this new release, the role has recently gotten some improvements and fixes by the community, so in case you had issues before, now is the time to try again!

P.S oh and I almost forgot, we rehomed to https://github.com/matrix-org/matrix-hookshot. This doesn't change much, other than some redirects in your browsers cache.

Half-Shot adds

permissions:
* actor: '*'
  services:
  - level: commands
    service: github
* actor: '!the-internal-element-room:matrix.org'
  services:
  - level: manageConnections
    service: '*'
* actor: '@random-on-the-side-user:homeserver.org'
  services:
  - level: manageConnections
    service: '*'

Allows anyone to run github commands in rooms. User part of the room listed can actually go and create connections to anywhere (we use it internally at $work), and then there is a final random user who gets those permissions because they refuse to join that room :p

matrix-appservice-irc (website)

Node.js IRC bridge for Matrix

Half-Shot reports

matrix-appservice-irc 0.33.0

Two bridge updates from matrix.org in one week? You would be forgiven for thinking it was FOSDEM Christmas.

The highlights are:

  • Support splitting users from different homeservers into different IPv6 blocks. We used this at FOSDEM to split attendeeds into a different ILINE.
  • Added a new metric clientpool_by_homeserver which lists the states of IRC clients, by the top 25 homeservers. Useful to know who is using your bridge.
  • Add support for subscribing to moderation policies. This allows a bridge admin to subscribe to Matrix "ban lists" and filter who can use their bridge.

This release also now requires Node 14.x as Node 12.x will be out of support very soon.

See the changelog for yourself over at https://github.com/matrix-org/matrix-appservice-irc/releases/tag/0.33.0

Dept of Clients 📱

Neochat (website)

A client for matrix, the decentralized communication protocol

Tobias Fella says

This week, we've fixed a few annoying issues in NeoChat:

  • On platforms with a system tray, NeoChat no quits correctly when the system tray integration is disabled
  • Similar state events are now aggregated into a single message (so no more scrolling through "Moderation has changed the access control lists" 🙂)
  • Sending messages starting with a ':' works correctly again

We've also improved NeoChat's startup time from cache by a factor of several hundred, depending on the situation

Element Web/Desktop (website)

Secure and independent communication, connected via Matrix. Come talk with us in #element-web:matrix.org!

kittykat says

  • Released v1.10.5 and also v1.10.6 as a hotfix for a crash
  • Surveyed users and collated feedback from search result ordering for improvements to the new search experience (enable it in Settings -> Labs in production)
  • In labs (you can enable labs in settings on develop.element.io or on Nightly):
    • Pin drop location sharing has been added
    • Fixed message ordering issues in threads
    • Fixed threads discovery when scrolling back in a room
    • Added a couple of new ways to access pinned messages, in the room header and info panel
    • Improved the reliability of pinned messages with edits

Element iOS (website)

Secure and independent communication for iOS, connected via Matrix. Come talk with us in #element-ios:matrix.org!

Manu announces

  • New release, v1.8.3, is out
  • Still improving the activity indicator
  • Working on missing messages bugs
  • Making progress on the registration flow by setting name and avatar
  • In development:
    • We will be trying out Spaces on iOS at 16:00 UTC / 17:00 CET on Tuesday, 8th of March. Head over to #element-community-testing:matrix.org to hear the latest on all testing sessions!
    • Live location sharing has started
  • In labs:
    • An option to use the last member avatar and name in the timeline

Element Android (website)

Secure and independent communication for Android, connected via Matrix. Come talk with us in #element-android:matrix.org!

adam reports

  • Our newest release, v1.4.2, is out!
  • Progress is being made on fixing “unable to decrypt” errors (UISI) - re-architected legacy code.
  • Our language capabilities have expanded with improved Japanese translations (special thanks to the contributor for this!)
  • The team is working on process improvements to reduce incoming PR backlog and getting to those faster.
  • We’re actively looking into the reported increased memory usage.
    • As always, thank you for feedback and reports, it helps a lot.
  • For those using the SDK, we’ve deprecated the main Matrix entrypoints, more information here

Dept of Non Chat Clients 🎛️

matrix-streamchat (website)

Matrix powered stream overlay for OBS, to integrate live chat in your favorite (selfhosted) streaming setups.

f0x says

a very simple invite-accepting bot as part of matrix-streamchat to more easily facilitate peeking into rooms over federation. You run one on your guest-access enabled homeserver, invite it to your remote room, and now all guest accounts from your server know they are in fact allowed to publicly read the room over federation without joining themselves. https://git.pixie.town/f0x/matrix-streamchat/src/branch/main/autojoin-bot

Dept of Widgets 🧩

Mjolnir Widget (website)

A widget for moderating with mjolnir. Highly WIP. Use at your own risk!

MTRNord (they/them) reports

Mjölnir Moderation Widget

There have been a few changes to the widget:

Dept of VoIP 🤙

Element Call (website)

Native Decentralised End-to-end Encrypted Group Calls in Matrix, as a standalone web app

Matthew reports

Element Call has entered beta! Head over to https://call.element.io (formerly matrixvoip.dev) to play with in-browser native Matrix voice/video calling powered by MSC3401, supporting all webrtc-capable mobile & desktop browsers. See all the details over at https://element.io/blog/introducing-native-matrix-voip-with-element-call/

  • Offers high quality video calls with around 6 or 7 participants as a small & simple, standalone app that allows anyone to drop into a video conference easily.
  • This is beta - please file bugs at https://github.com/vector-im/element-call/issues and be aware that currently all participants need good bandwidth
  • This is the first implementation of MSC3401: a spec for secure, federated voice and video conferences.
  • Integrations into other clients are coming soon, Hydrogen is working on it right now
  • We’ll also be developing a server to mix media streams so calls can scale up to much larger number of participants. The server will only ever see your encrypted media, so calls will stay secure and confidential.

Dept of SDKs and Frameworks 🧰

Ruma (website)

A set of Rust library crates for working with the Matrix protocol. Ruma’s approach to Matrix emphasizes correctness, security, stability and performance.

Jonas Platte reports

We're back (to TWIM)! Here's some highlights from the past few months:

All of these changes are available in version 0.5.0 which was released about two weeks ago.

Additionally, @zecakeh recently started implementing extensible events and doing some pretty heavy internal refactoring that will make our release process faster and hopefully make contributing to Ruma easier as well.

With all these changes we're very close to supporting all of Matrix v1.1 (and Matrix v1.2 too!), with the only major omission being Secure Secret Storage and Sharing (SSSS). Some work on that was done as far back as last year's GSoC but it was blocked on a medium-sized refactoring of how we handle account data. I'm planning to get that over the finish line over the next weeks.

Dept of Ops 🛠

Synatainer (website)

Synapse Maintenance Container – Docker container with tools for synapse & postgres database maintenance

saces announces

Start it without command and let it do its magic :)

What it does by default:

  • daily:
    • purge all rooms without local members
    • run the state autocompressor (small setting 500/100)
  • weekly:
    • delete old remote media (>90 days)
    • delete old message history from public joinable rooms (>180 days)
  • monthly:
    • run the state autocompressor (big setting 1500/300)
    • vacuum the database

Source: https://gitlab.com/mb-saces/synatainer

Room: #synatainer:c-base.org

Compose example: https://gitlab.com/mb-saces/synatainer/-/snippets/2264828

Dept of Bots 🤖

Matrix Scribe

JulianF announces

I'm working on a little hobby project, The Matrix Scribe.

The Matrix Scribe helps us re-post or transcribe messages into matrix that we received from somewhere else, posing as different ghost users to represent the original authors.

Scribe will post this:

Ann I'm Ann.

Bob Hello! I'm Bob.

when I write this to @scribe-bot:

@julian /scribe as Ann

@julian I'm Ann.

@julian /scribe as Bob

@julian Hello! I'm Bob.

That is an illustration of the goal. The output part works already; the bot part isn't implemented yet.

Why?

  • I want to encourage people to experiment and play with extending Matrix in creative ways.
  • As a building block for myself towards building more usual matrix bridges. Scribe is the input half of the matrix half of a bridge, so literally one quarter of a typical matrix bridge.
  • I hope that I myself and maybe some other people will find Scribe fun or useful in itself.

If you find this interesting...

Scribe is not intended to mislead users. It should be deployed in such a way that its use is clear, and require appropriate permissions.

maubot_azuracast

boris reports

a simple bot https://github.com/borisrunakov/maubot_azuracast built with maubot, that you can use to request data from your azuracast self hosted instance. It's not much but ok...

Dept of Events and Talks 🗣️

J. Ryan Stinnett says

I'll be speaking at the new HYTRADBOI conference (29 Apr) about building collaborative, open software on Matrix. All of the talks sound super interesting, highly recommended! 😄 The talks will be public after the conference as well.

Dept of Ping 🏓

Here we reveal, rank, and applaud the homeservers with the lowest ping, as measured by pingbot, a maubot that you can host on your own server.

#ping:maunium.net

Join #ping:maunium.net to experience the fun live, and to find out how to add YOUR server to the game.

Rank Hostname Median MS
1 converser.eu 347
2 envs.net 377
3 aria-net.org 466
4 maescool.be 573
5 pavot.ca 742
6 rollyourown.xyz 1294.5
7 canchat.org 1473
8 targodan.de 1658
9 2808.nl 1949
10 mtrx.fail 2093.5

#ping-no-synapse:maunium.net

Join #ping-no-synapse:maunium.net to experience the fun live, and to find out how to add YOUR server to the game.

Rank Hostname Median MS
1 cry.is 288
2 nognu.de 319
3 rustybever.be 383.5
4 conduit.grich.sk 465
5 bogus.ee 568
6 matrix.sum7.eu 650.5
7 matrix.awesomesheep48.me 653
8 dendrite.matrix.org 1544
9 xethos.net 1756

That's all I know 🏁

See you next week, and be sure to stop by #twim:matrix.org with your updates!