John Livingston

Peertube chat plugin: quick feedbacks about a live stress test

John Livingston — 2023-01-04T17:37:59Z

Sommaire

Introduction

On the last friday, the 30th of december, the “Au Poste” team — a french media started by David Dufresne — organised a Peertube live test. It was the opportunity for Framasoft, Octopuce and I to test Peertube as well as my chat plugin “at scale”.

This is something I had been waiting for a long time. As an independant developper, it is hard for me to gather hundreds of people to make stress tests. David Dufresne has a community of multiple hundreds of people who follow him on Twitch. Most of whom, for political reasons, are trying to free themselves from GAFAMs. This full-size test was long-awaited by many people of his community.

So I had several meetups with David and his team the week before to organize this live event. From his side, he also contacted Octopuce (his hoster) and Framasoft (which is behind the Peertube project).

Note: it was foreseen that I would be connected to the chat to answer questions, but as it was very active it was difficult to follow any of it. I ended up joining the live half-way. We had many interesting discussions: on politics, or tech, ... The replay is available here: Peerturbateurs, unissez-vous! Test d'un live Peertube avec 400 viewers.

Feedback

Octopuce has published a very complete article (in french) which details how servers handled the test. Peertube live load testing with Auposte (fr).

For people that doesn't understand french, in two words: 450 viewers tried to connect at the same time to the live and the chat.
We had some issues with the chat plugin. Some issues that I was expecting, some others not. I will try to come back to them in this article ; and explain what happens and why I think it wasn't working the best it could. I think that some of the issues are a little more complex than what the Octopuce article made it sound like. This - quickly written- article is also the opportunity for me to explain what I have in mind to improve the situation.

Technical details

We're now going to look at what happened during this stress test. I will explain technical details that could IMHO explain observed metrics.

I think some problems are due to many little things that were more or less predictable. The sum of them caused some troubles.

1. Architecture

The Peertube chat plugin is a bit special. The goal is to be able to install it from the Peertube interface, without any server configuration required, and without root rights.

This means I can't:

Use official packages (debian, ...),
Assume necessary XMPP ports are open on the server
Access a database. Peertube's plugin system only gives basic access to the PostgreSQL database. Adding tables from there doesn't look very clean.

This lead me to make choices that are not optimal and I am aware of it. I already have a very long TODO list with lots of possible improvements. It's only a matter of time.

Parts of this TODO is already described in the article 2023 will be full of new features, available in multiple languages.

The rest is registered on the github tracker. These issues are also sorted in a github project.

For the frontend, I use ConverseJS which allows connecting to XMPP servers from a web browser, using Javascript.

Let's dig into constraints I listed above and see what they imply.

1.A AppImage

The chat plugin uses the Prosody XMPP server as a backend. I need to be able to use Prosody without installing anything on the server. No “root” access.

I created an AppImage of Prosody based on Debian stable packages. This AppImage contains Lua (language used for Prosody), Prosody itself, and a small launcher script used to start either prosody or prosodyctl (to avoid having two different AppImages).

The AppImage is created with appimage-builder, from this configuration file: appimage yml.

Then Peertube generates a configuration file for Prosody (from plugin parameters), and launches the AppImage. It's not launched as a daemon, but as a child process of Peertube. This way if Peertube is stopped, we're sure Prosody is too.

1.B Proxy

As ports on the machine could be closed by firewalls, we need to proxify everything through Peertube. It's ugly but it works out of the box.

How does it work?
I use Websocket or BOSH connections which go through 2 successive reverse proxies:

nginx in front of Peertube
Peertube itself

Peertube is written in NodeJS, so I use a NodeJS library which allows me to reverse proxy. It's not optimal, and I probably should rework it to improve performances.

Note: I plan to provide a way to have advanced configuration in the plugin, which would allow us to optimise it, especially by giving direct access to the XMPP server from the outside. It will require root access on the server though.

1.C Database

No database: Prosody uses the the internal storage module. Which means everything is stored as files: user files, chat history, ...

2. Virtual Hosts and XMPP Components

In the generated Prosody configuration we can find:

A Virtual host for connected Peertube users. If I have a Peertube account “@john instance.tld”, then I will have an XMPP user “john instance.tld”.
A Virtual host for anonymous users. Users without an account can join the chat just by choosing a nickname. In the live test on the 30th, we had more than 400 people in this situation.
A “room” component, to handle groupchats.

3. Prosody specific modules

I have specific Prosody modules. There's different types: those with “livechat” in the name:
Chat plugin Prosody modules.

Here you'll find:

A module to list rooms (to use in moderation tools)
A test module, to check Peertube and Prosody can communicate. This is used in a diagnostic tool included in the plugin.
Two vCards modules: One to get connected Peertube users' avatars, and one to generate random avatars for anonymous users.

I also use existing modules:

mod_muc_http_default to verify rooms can be created and configure them
mod_auth_http to authenticate connected Peertube users

All these modules make requests to Peertube's web API.

This is where part of the problems Octopuce had last week comes from (IMHO): because of lacking features in Peertube's plugin system, whenever Prosody needs to call Peertube, it needs to go through nginx again. An issue has been opened, it needs to be discussed and implemented.
Note: it was implemented today, and will be available in next Peertube version.

In the meantime, this can be configured by hand in advanced plugin settings, but it's not very well documented.

4. vCards

An issue that had already been identified is that we use vCards in ConverseJS. vCards are like «electronic business cards» for chat users. They're also used to store avatars and that's where I get them from.

As soon as someone connects, ConverseJS will fetch every other connected users' vcards. And when a new person joins, everyone already connected will ask for their vcard.

For those already connected to Peertube, Peertube's avatar will be fetched. In last week's test only 4 or 5 people had Peertube accounts and were in this case.

For anonymous vcards, a random one will be choosen among a list of a few dozens and get displayed.
Code handling this is here: mod_vcard_peertubelivechat->https://github.com/JohnXLivingston/peertube-plugin-livechat/blob/main/prosody-modules/mod_vcard_peertubelivechat/mod_vcard_peertubelivechat.lua].
I'm new at Lua so this may not be the best code there is.

Last week we had 400 people joining around the same time. 400 people asking for 400 vcards. It was really sluggish, and on top of that people refreshed the page multiple times instead of waiting a bit for things to settle down.

In addition to this, each avatar is about 4.2Ko in JPG. This JPG is base64 encoded before being sent via XMPP, which makes it about 6Ko per avatar.
400 * 400 * 6Ko = 960Mo, at once, in addition to the XMPP enveloppe and other vCard data.

This also doesn't take into account CPU load and disk on Prosody, which was at a 100% for some reason. I think vcards are the culprit (as mentioned in the Octopuce article, Prosody is single-threaded).
Which implies timeouts, people refreshing the page even more, etc.

I have in my TODO list to compute avatars on the frontend, and stop fetching vCards. This should greatly help. I also hesitate to disable avatars completely to replace them with a simple colored nickname, which could use XEP-0392.

5. Chat history, pruning, ...

Until last friday's test, I didn't think handling chat history would be an issue. This is why I haven't tried to optimise it, and I configured ConverseJS to fetch it when a user joins.

It turns out, with 400 users at once, it's X messages * 400 users to fetch. All of this being stored in files (Prosody's internal storage). It was bound to be an issue.

Another completely unexpected issue: I don't prune messages in ConverseJS. So they all pile up and the browser starts struggling for each key stroke.

This again made people refresh the page even more during the live stream causing multiple issues. This is something I will be looking into very quickly.

6. Network load caused by fetching ConverseJS

The way ConverseJS is currently integrated in the plugin makes that:

ConverseJS is included in its entirety, even modules I don't need
It goes through Nginx and then NodeJS to load files.

Everytime someone opens the page or refreshes it, the browser downloads multiple MB through Nginx and NodeJS. This must explain part of the huge bandwidth consumption observed by Octopuce.

I have already started cleaning all of this. If everything goes to plan, ConverseJS will be optimised and will load more efficiently in the next version of the plugin.

7. Join/Part messages and activity

I have enabled join/part messages' display in chat (“Juliet joined the room”), as well as activity messages (“Max is writing...”). With 400 people joining and quitting multiple times, and with all the refreshes, it can also quickly add up and generate traffic (and make JS go sluggish again).

I will remove this. Maybe only leave join / part messages for moderators, but messages such as “Jack is writing” have little interest, if not to generate hundreds of unnecessary XMPP messages.

8. Changing nicknames

For anybody without a Peertube account to be able to see live chat without even having to choose a nickname, I assign to them a random one such as “Anonymous 123456”.

To be able to chat, they must first choose a nickname, and ConverseJS is going to display “Anonymous 123456 becomes Camille” each time. This again generates lots of XMPP traffic for this number of users and lots of JavaScript running for little gain.

I may end up hiding this type of messages as well. At least when the original nickname is « Anonymous 123456 ». I am also thinking about removing those that haven't yet chosen a nickname from the list of participants in the room, only to display a counter « X anonymous » which can only reduce traffic and the browser load.

In Conclusion

As I mentioned at the beginning of this article, this is just a first shot. I put down what came to mind while writing and it's possible I have forgotten things. You can find all the improvments mentionned in this article in this github issue tracker.

There are still many possible improvements for this plugin. I only need to get back to it!

Thanks

I want to thank the XMPP community. Many people working on Prosody, ConverseJS, XEPs, etc. are looking at my work and regularly provide help.
Thanks to all, it reassures me in the choice of technology I made when I started this project.

Many thanks also to David and his team, Octopuce, and Framasoft. It was a fun live event, and there is plenty of things to learn from.

Peertube chat plugin: 2023 will be full of new features!

John Livingston — 2022-12-15T10:19:17Z

Sommaire

This article is a translation from the french version

The chat plugin

One of my most important free software projects, and without a doubt the one I care about the most, is the Peertube livechat plugin.

Peertube is a free software, allowing to deploy a video on demand and live streaming platform. The different installations of Peertube (called "instances") can be federated together, being part of what is called the fédiverse, contraction of "federation" and "universe". It is a set of software that can communicate with each other. There are video platforms, music platforms, micro-blogging tools, publishing tools, and all types of social networks.

Framasoft is piloting the Peertube project, but did not want to add a chat function natively in the software. They preferred to make it possible via the "plugins" system that Peertube offers.

And this is where I come in!

I was first contacted by ritimo, and I developed in 3 days a first functional version.
I chose to rely on an existing, proven and scalable technology, XMPP. This choice was motivated by a long term vision: I had to be able to make the software evolve easily, and add moderation, federation, robots, ... In short, the XMPP protocol, and the XMPP server Prosody were just right.

Afterwards, the plugin has evolved a lot. At the time of writing, it is in its 6th major version!

To see a demo of the plugin, you can go here.
There is also a demo room where I test future versions here.

The search for funding

It's been 3 years now that I decided to become a free software developer on my own. It is above all an ideological choice, wanting to get out of the capitalist market, its relations of subordination, and wanting to work for the creation of "digital commons".

The development of this plugin requires time, a lot of time. And I'm trying to make a living out of it. So to be able to work on it, I need to find funding. Unfortunately, I have to prioritize my projects according to the financial support I find.

2021/2022

In the past I have received funding from ritimo, Framasoft and Code Lutin.

But a large part of my 2022 year has been spent looking for budgets to work on this plugin. I also spent a lot of time working on it "for free" (ie: I didn't find anyone to fund the time spent).

It was only at the end of this year that I was able to find solid funding, and work on a roadmap.

Donations

I didn't mention it in the previous paragraph, but I also received some donations, especially through my Liberapay page. From this platform, I receive on average 8.90€/month.
I also received a donation via Github Sponsor.

These donations may seem anecdotal, but in reality they are very motivating. To think that a stranger is willing to give me 10, 20 or 50€, just because what I did is useful to them, is extremely gratifying.

So "big <3" to all the people who donate via this kind of platform, whether to me or to others. Thank you, thank you, THANK YOU!

Copie Publique

Code Lutin is a company specialized in free software. They have decided, several years ago now, to redistribute at least 1% of their annual turnover to one or several free software projects. The decision is taken democratically among the team members. This is a fantastic initiative, which in addition to helping financially, gives a lot of visibility to the chosen projects.

I had already been lucky enough to be selected last year.

And for 2022, they decided it was time to move up a gear, and encourage other structures (companies, developers, ...) to do the same. So they set up the Public Copy Retribution platform. The idea is simple: you benefit from free and open source tools, it's a fair return to give some money in return. Any structure wanting to join the initiative can define its own conditions. I strongly invite you to visit their site, and have a look at the past beneficiaries.

And I'm very happy to have been selected again for a grant in 2022 (for the Peertube chat plugin, and more broadly for the other Peertube plugins I'm working on).

Framasoft

At the end of this year, Framasoft has also decided to release a budget for the chat plugin.

Framasoft has always supported me in this project, and Chocobozzz (the creator of Peertube) has always been very reactive when I needed changes in the Peertube plugin system. It's a real pleasure to work with people like them.

I have to specify the following points:

This funding was made possible thanks to donations (mainly from individuals) made to Framasoft in previous years.
Framasoft is financed only by donations, so "more donations to Framasoft => more capacity to finance free services around PeerTube & other Framasoft projects" (and Framasoft has a lot of projects for the years to come!).
Framasoft is in the middle of a donation campaign for 2023. They are still missing a big part of what they need for 2023, so go support them!

If you want a little more information, it just so happens that the two co-directors of Framasoft were live at the station to talk about it this morning (this is a Twitch link, but I'll replace it with a Peertube link as soon as it becomes available).

NlNet

NlNet is a Dutch foundation that funds Free Software projects. They launch several fundraising campaigns per year, on different themes. The money comes from sponsors, among others the European Commission.

After I applied once without being selected, I tried again for the NGI Zero call in August 2022. And this time, I can now announce it publicly, we have signed!

This will allow me to work on the plugin for several weeks, and to make some much needed changes! (see further in this article).

And for 2023?

After this long search for funding, I can now finally tackle the evolutions of the plugin!

I have already published a version 6.0.0 which is detailed here.

I have reordered all the tasks to be done in the form of Github tickets. I've added milestones to give a rough timeline (I still have to juggle between different projects, so I hope you'll be tolerant of my possible delays 😉).

I also created a Github project that allows you to see the status of different tasks (and filter by milestone).

Finally, I added tags on the Github tickets specifying the sponsor related to the task (if any). So you can filter the tickets by sponsor, thanks to whom the features will be released.
NB: Code Lutin is not associated with any particular task, but their funding does exist, and helped me a lot!

I will now detail the biggest features to come.

Display of the chat in the "embed"

It is possible to embed a Peertube video on a third party site. But until recently, it was not possible for Peertube plugins to add features to the included videos.

Today, it is already possible to include the chat, by building "by hand" the "iframe" tag that goes well. If you don't understand what I'm talking about, well... you understand the problem ;).

The first thing I'm going to do is to add an option to easily generate the necessary to copy and paste on your website. Just like you would do to include the video. You will then only have to generate the "iframe" for the video, and the one for the chat, and place both at your convenience.

Next, I will also propose another option. When you generate the iframe for the video, you will have an option to include/exclude the chat directly in the same iframe as the video (and/or a button to open the chat in a new window, the decision is not taken).

I will propose both functional solutions, because I think both are useful. Indeed, there is no point in displaying the chat on top of the video if you are in a very small thumbnail, and vice versa.

These features are sponsored by Framasoft.

UI/UX redesign

Framasoft asked me to work on the interface (UI, User Interface) and the user experience (UX, User eXperience).
So I'm going to re-work entirely the plugin integration. Ideally, this one will integrate better with Peertube (same fonts, borders, ...). And I'll see if I can make it possible to move the chat window freely.

I'll also make the moderation functions more visible, and more convenient to use. Because yes, the plugin already allows advanced moderation, but... it's a bit hidden. Because I didn't have time to do better in the past.

Finally, I will also improve the integration of the chat in the streaming tools (especially OBS). Currently, it is already possible (and documented). But I've been given some ideas on how to make it much better.

Federation

Peertube is a federated technology. You can create "bridges" between "instances". For example, I can see a live broadcast on the server of association A, from my personal server B. But for the moment, the plugin doesn't know how to manage the federation at all. The chat is only displayed on the original instance (A).

Thanks to a funding from NlNet, I'll make the chat display on the instance from where you're looking, and that you're connected with your account on this one (if necessary).
The fact to have chosen the XMPP technology will make my life easier, this one was already made to be decentralized. I just have some technical difficulties to overcome (Peertube has to act as a web proxy in front of the service, so I have to "proxy" the communications through a web layer).

Of course, this will have to be accompanied with moderation features, to avoid hostile behaviors (like "bot raid" or "spam"). The streamers will need to be able to choose whether or not to open to remote instances, mass ban, etc.

In a second step, I would also try to standardize these chat mechanisms on the fediverse, to allow connecting to it from other software! (why not Mastodon for example). My work will consist in proposing an "ActivityPub" standard to announce the chat(s) associated to an object, and to implement it in Peertube and the chat plugin.

Moderation/interaction bot

I have a lot of ideas about bots features (non exhaustive list):
– automated publication of text in the chat (for funding links, chat rules, ...)
– automated moderation (blocking of certain words, ban, etc)
– commands

To summarize: a lot of features you may have already seen on Twitch and so on. I know it is very expected, and good news, it will be done in 2023 thanks to NlNet!

I'm also thinking of an interface for the streamers, with a list of action buttons. For example to ban in one click, or publish pre-filled messages.

And finally, I'm also thinking of interaction tools in the other direction: an action in the chat would cause a display/sound in the stream. This is for example what we can see on Twitch when there is a donation.

Don't hesitate to propose your ideas here if you have an account.

Translations

I will also work on the plugin translations, and try to support a maximum of languages.

Accessibility

NlNet provides an accessibility audit service to its beneficiaries. So I will be able to make sure - with the help of competent people - that the chat plugin is accessible (for visually impaired, colorblind, motor handicap, ...), and correct issues if necessary.

Other

There are other evolutions planned, but I won't detail them here and now. The Github project gives a pretty complete overview.

Conclusion

That's it, that's the news about this Peertube chat plugin. I hope you will be as excited as I am. I can't wait to tackle all these features!

If you want to discuss it, you can go through Github, the Peertube forum which I read almost daily, or my Mastodon account.

PS: the illustration image of this article is Sepia, the Peertube mascot. Published under a free license CC-BY-SA by David Revoy