One of the most important features of decentralised social networks based on open standards is that they are much easier to own and control, while also being highly intercompatible between software versions, clients, and maintainers.

These features make them desirable especially to people who enjoy free software, and therefore decentralised protocols are essentially the norm in circles of experienced open source programmers and users, Linux user groups, and generally people who appreciate free software.

First there was IRC

IRC (Internet Relay Chat) was the pioneering chat protocol on the Internet. It offered a basic form of communication, limited to text and simple features like chat rooms, chat room privileges (including admins, moderators, etc.), private messaging, and basic file sending and receiving through DCC (Direct Client-to-Client). Documented in RFC 1459, and although strictly not a standard, the IRC protocol has remained largely unchanged since its inception. While some client programs offer additional features like previewing images sent as links, extending IRC is challenging. Any modification to the core protocol risks breaking compatibility with older clients, limiting its adaptability.

Then XMPP came along

Enter XMPP. As an avid user, I witnessed its evolution. XMPP, also an open standard, enhances IRC in several key aspects, primarily through extensions known as XEPs (XMPP Enhancement Proposals).

XMPP extensibility

Many basic features like user-specified nicknames, or user-specified profile pictures are implemented as XEPs, which work similarly to RFCs, being technical documents that describe proposed implementations for specific features or functionalities.

However, it’s important to understand that XEPs are, as the name suggests, proposals. This means they’re subject to change and may not be universally implemented.

XMPP has a set of core features that are part of its basic protocol, not requiring any additional XEPs. These basic features include:

1. JID (Jabber ID) Identifiers: XMPP uses a unique identifier format known as Jabber ID or JID, which follows the format user@domain.tld. This format is similar to email addresses and allows for easy identification and addressing of users on different servers.
2. User-to-User Private Messaging: XMPP supports direct, one-on-one messaging between users. This feature allows users to send and receive messages in real-time. Crucially, offline message handling is described in a separate XEP: https://xmpp.org/extensions/xep-0160.html, even though it is a basic feature in any modern messaging application.
3. Presence Information: Users can broadcast their availability status, such as online, away, or do not disturb, along with custom status messages. This lets others know their current availability for communication.
4. Contact Lists (Roster Management): XMPP allows users to maintain a list of contacts, known as a roster. Users can add, remove, or organize contacts, and view the presence status of these contacts.
5. Basic “Chatroom” MUCs (Multi-User Chat): XMPP supports multi-user chat rooms, allowing multiple users to join and communicate in a single chatroom. However, more advanced features of multi-user chats like persistent rooms and room administration are defined in XEPs.
6. Server-to-Server Communication: XMPP servers can communicate with each other, allowing users on different servers to interact seamlessly. This is a fundamental feature for the decentralized nature of XMPP.
7. Resource Binding: XMPP allows a user to be connected from multiple devices or locations simultaneously, with each device/location being identified as a different resource. For example, a user might be connected from both a phone and a laptop, with each device having its own resource identifier.
8. Basic Delivery Requests: XMPP can notify the sender when a message has been delivered to the recipient’s client. This is a basic acknowledgment that the message reached the client, not necessarily that it was seen by the user.
9. Privacy Lists: Basic privacy controls are available, allowing users to block communications from specific other users or to manage who can see their presence information.
10. XML Stanzas for Messaging: The protocol uses XML stanzas (structured data formats) for sending and receiving messages, presence information, and IQ (Info/Query) stanzas for control-type requests and responses.

For almost any sort of functionality beyond the above, implementation of XEPs is required. At the time of writing, almost 500 documents are available. Some are retracted, rejected, deprecated, or obsolete.

A critical feature in modern communication is end-to-end encryption, as exemplified by XEP-0384: OMEMO Encryption. This is, at the time of writing, the only end-to-end encryption XEP, but in an experimental phase, which means it is not mandatory to implement. There are two other options for message encryption that a multitude of clients like Pidgin and Gajim have implemented, such as PGP and OTR encryption, however, these two encryption mechanisms are not defined as XEPs at all, and thus cannot be considered part of the XMPP specification.

Another taken-for-granted feature in contemporary messaging solutions is the ability to send files to multiple recipients, including when some are offline. This is outlined in XEP-0363: HTTP File Upload. Fortunately, this time, XEP-0363 is considered stable. In addition to HTTP file transfer, there exist two more file transfer mechanisms. Jingle, as defined by XEP-0166, XEP-0234, XEP-0260, and XEP-0261 (yes, a total of four XEPs for implementing Jingle file transfer), which is optional, and Stream Initiation as defined by XEP-0096 which predates Jingle, and is now deprecated.

XEP-0459: XMPP Compliance Suites 2022 defines the set of XEPs any server or client must implement in order to be considered “Core” or “Advanced”. Notably, XEP-0384 is not yet a requirement, but this may change. However, considering that Signal, a widely popular end-to-end encrypted chat application, was released in 2010, it’s significant that XMPP has yet to mandate end-to-end encryption, even 12 years later. This delay highlights a critical area where XMPP lags behind contemporary security standards in messaging applications.

Another basic feature is XEP-0357: Push Notifications. This XEP describes the implementation of notifications, crucial for ensuring timely message delivery in mobile environments. The process involves a flow between the XMPP client, server, app client and app server, and the push service, as illustrated in this helpful diagram:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

+-------------+         +-------------+
|             |         |             |
| XMPP Client |         | XMPP Server +--------+
|             |         |             |        |
+-----^-------+         +-------------+        |
      .                                        |
      .                                        |
+-----+------+          +------------+---------v---------+
|            |          |            |                   |
| App Client |          | App Server < XMPP Push Service |
|            |          |            |                   |
+-----^------+          +------+-----+-------------------+
      |                        |
      |                        |
+-----+------+          +------v--------+
|            |          |               |
| User Agent <----------+ Push Service  |
|            |          |               |
+------------+          +---------------+

However, XEP-0357 is currently in a “deferred” status, which presents a significant challenge. In an era where smartphones often have intermittent network connectivity and limited battery life, the absence of stable push notifications means users cannot receive real-time alerts unless their XMPP client is actively running. Continuously running the client, however, leads to rapid battery drain, and most modern mobile operating systems will automatically terminate apps after a period of inactivity.

Several commendable efforts have been taking place to make XMPP suitable for today’s world, like Modern XMPP, and the Compliance Suites XEPs released since 2019. Some clients like Conversations, and servers like Prosody show the XEPs they implement, so that you can check for yourself what features are available. Nevertheless, for functionalities like push notifications in XEP-0357, support is required from both the server and client. This particular feature is not yet implemented in Prosody. Additionally, the uncertainty about the software and its version used by XMPP server providers means users cannot be certain about the availability of specific features.

A significant challenge remains: proprietary closed-source chat applications are innovating at a pace far beyond what XMPP and other open-source efforts can match. This rapid development in features and user experience often leaves XMPP lagging in comparison. On the other hand, XMPP has found success in unexpected domains. Notably, famous videogames like League of Legends, Fortnite, or the Nintendo Switch console use XMPP. This adoption is a testament to XMPP’s flexibility as a free and open-source protocol. Its ability to be tailored for specific needs showcases the protocol’s inherent power and versatility. But it is not enough for personal messaging.

Matrix will save us

Matrix, a messaging standard introduced in 2014, brings a solution to many of XMPP’s shortcomings, such as the overwhelming number of XEPs. A notable feature of Matrix is its default end-to-end encryption for private chats, addressing a key concern in modern digital communication. However, Matrix’s journey hasn’t been without challenges.

Too heavy

Synapse, the most prevalent Matrix server implementation, is known for its high demand on system resources. This requirement significantly limits accessibility for individuals and smaller organizations with constrained resources. In contrast, XMPP is more mature and boasts a range of extraordinarily lightweight server implementations. The protocol’s accessibility also simplifies the process of building a custom server, tailored with only the desired XEPs.

Protocol complexity

Unlike XMPP, which has hundreds of feature proposals, the Matrix protocol is more rigidly defined. This well-defined structure, paradoxically, adds to its complexity. The intricate nature of Matrix leads to a steeper learning curve for developers, potentially impacting the diversity of server software as fewer developers might be willing to engage with such a complex system.

UX issues

Matrix is designed for scalability, but this doesn’t come without its own set of user experience issues. Particularly in larger rooms with numerous participants, users have reported performance-related problems, including message delays and server load challenges. These issues underscore the need for ongoing optimization in the Matrix ecosystem.

Federation complexity

While one of Matrix’s strengths is its federated architecture, setting up and maintaining federation can be complex and sometimes lead to issues where rooms or messages are not properly synced across different servers.

Where do we go from here?

I don’t know. You can probably tell I’m not exactly optimistic about the state of open source, federated instant messaging.

One could make the argument that these technologies don’t need mass adoption in order to be successful. You can say they already are successful simply because they exist. And while it is true these technologies have contributed massively to the Internet and society in general, the way I see it is that these improvements and contributions have merely served to improve the software of massive corporations who have a strong incentive to keep their platforms closed and exclusive to their users only.

Therefore, it doesn’t seem very helpful that in order to enjoy the benefits of these systems, I am more or less forced to use proprietary walled garden implementations, because it is not possible to, for example, talk to WhatsApp users from Signal or XMPP, even though WhatsApp is built on top of XMPP.

This situation presents a paradoxical landscape in the world of instant messaging. The ideals of open-source, federated systems – transparency, user control, privacy, and interoperability – stand in stark contrast to the closed, proprietary ecosystems that dominate the market. This divide poses significant challenges, not only in terms of user convenience but also in shaping the future direction of communication technologies.

The harsh truth is that closed-source systems typically benefit from greater funding and a singular, market-driven objective: to dominate their sector, extract value from users, and convert this into profits for stakeholders. As these proprietary platforms grow, they attract more investors, who pour in funds that enable the hiring of more dedicated engineers. This influx of talent often leads to the development of advanced features, drawing users from simpler applications. This cycle perpetuates, continually reinforcing these platforms’ market dominance.

Yet, there is a silver lining. The very existence and ongoing development of open-source protocols like XMPP and Matrix demonstrate a persistent commitment to these ideals. They embody a resistance to the trend of centralized control over communication platforms, offering an alternative path that prioritizes user freedom and open standards.

In the end, the future of instant messaging – open-source or otherwise – will likely be shaped by a complex interplay of technological innovation, market forces, and social groups and their preferences.