Add README
Mark Haines
commit
855f8628e1
|
|
@ -0,0 +1,102 @@
|
|||
# Dendrite
|
||||
|
||||
Dendrite will be a matrix homeserver written in go.
|
||||
|
||||
# Design
|
||||
|
||||
*
|
||||
|
||||
|
||||
## Log Based Architecture
|
||||
|
||||
### Decomposition and Decoupling
|
||||
|
||||
A matrix homeserver can be built around append-only event logs built from the
|
||||
messages, receipts, presence, typing notifications, device messages and other
|
||||
events sent by users on the homeservers or by other homeservers.
|
||||
|
||||
The server would then decompose into two categories: writers that add new
|
||||
entries to the logs and readers that read those entries.
|
||||
|
||||
The event logs then serve to decouple the two components, the writers and
|
||||
readers need only agree on the format of the entries in the event log.
|
||||
This format could be largely dervived from the wire format of the events used
|
||||
in the client and federation protocols:
|
||||
|
||||
|
||||
C-S API +---------+ Event Log +---------+ C-S API
|
||||
---------> | |+ (e.g. kafka) | |+ --------->
|
||||
| Writers || =============> | Readers ||
|
||||
---------> | || | || --------->
|
||||
S-S API +---------+| +---------+| S-S API
|
||||
+---------+ +---------+
|
||||
|
||||
However the way matrix handles state events in a room creates a few
|
||||
complications for this model.
|
||||
|
||||
1) Writers require the room state at an event to check if it is allowed.
|
||||
2) Readers require the room state at an event to determine the users and
|
||||
servers that are allowed to see the event.
|
||||
3) A client can query the current state of the room from a reader.
|
||||
|
||||
The writers and readers cannot extract the necessary information directly from
|
||||
the event logs because it would take to long to extract the information as the
|
||||
state is built up by collecting individual state events from the event history.
|
||||
|
||||
The writers and readers therefore need access to something that stores copies
|
||||
of the event state in a form that can be efficiently queried. One possibility
|
||||
would be for the readers and writers to maintain copies of the current state
|
||||
in local databases. A second possibility would be to add a dedicated component
|
||||
that maintained the state of the room and exposed an API that the readers and
|
||||
writers could query to get the state. The second has the advantage that the
|
||||
state is calculated and stored in a single location.
|
||||
|
||||
|
||||
C-S API +---------+ Log +--------+ Log +---------+ C-S API
|
||||
---------> | |+ ======> | | ======> | |+ --------->
|
||||
| Writers || | Room | | Readers ||
|
||||
---------> | || <------ | Server | ------> | || --------->
|
||||
S-S API +---------+| Query | | Query +---------+| S-S API
|
||||
+---------+ +--------+ +---------+
|
||||
|
||||
|
||||
The room server can annotate the events it logs to the readers with room state
|
||||
so that the readers can avoid querying the room server unnecessarily.
|
||||
|
||||
[This architecture can be extended to cover most of the APIs.](WIRING.md)
|
||||
|
||||
# TODO
|
||||
|
||||
- [ ] gomatrixlib
|
||||
- [x] Canonical JSON.
|
||||
- [x] Signed JSON.
|
||||
- [ ] Event hashing.
|
||||
- [ ] Event signing.
|
||||
- [x] Federation server discovery.
|
||||
- [x] Federation key lookup.
|
||||
- [ ] Federation request signing.
|
||||
- [x] Event authentication.
|
||||
- [ ] Event visibility.
|
||||
- [ ] State resolution.
|
||||
- [ ] Third party invites authentication.
|
||||
- [ ] Room Server
|
||||
- [ ] Inputing new events from logs.
|
||||
- [ ] Inputing backfilled events from logs.
|
||||
- [ ] Outputing events and current state to logs.
|
||||
- [ ] Querying state at an event.
|
||||
- [ ] Querying current forward extremities and state.
|
||||
- [ ] Querying message history.
|
||||
- [ ] Exporting/importing messages from other servers.
|
||||
- [ ] Other Room Server stuff.
|
||||
- [ ] Client Room Send
|
||||
- [ ] Handling /client/r0/room/... HTTP PUTs
|
||||
- [ ] Talk to remote servers for joins to remote servers.
|
||||
- [ ] Outputting new events to logs.
|
||||
- [ ] Updating the last active time in presence.
|
||||
- [ ] Other Client Room Send stuff.
|
||||
- [ ] Client Sync
|
||||
- [ ] Inputing new room events and state from the logs.
|
||||
- [ ] Handling /client/r0/sync HTTP GETs
|
||||
- [ ] Outputing whether the client is syncing to the logs.
|
||||
- [ ] Other Client Sync Stuff.
|
||||
- [ ] Other Components.
|
||||
|
|
@ -0,0 +1,192 @@
|
|||
# Wiring
|
||||
|
||||
The diagram is incomplete. The following things aren't shown on the diagram:
|
||||
|
||||
- [ ] Device Messages
|
||||
- [ ] User Profiles
|
||||
- [ ] Notification Counts
|
||||
- [ ] Sending federation.
|
||||
- [ ] Querying federation.
|
||||
- [ ] Other things that aren't shown on the diagram.
|
||||
|
||||
Diagram:
|
||||
|
||||
|
||||
W -> Writer
|
||||
S -> Server/Store/Service/Something/Stuff
|
||||
R -> Reader
|
||||
|
||||
+---+ +---+ +---+
|
||||
+----------| W | +----------| S | +--------| R |
|
||||
| +---+ | Receipts +---+ | Client +---+
|
||||
| Federation |>=========================================>| Server |>=====================>| Sync |
|
||||
| Send | | | | |
|
||||
| | +---+ | | | |
|
||||
| | +--------| W | | | | |
|
||||
| | | Client +---+ | | | |
|
||||
| | | Receipt |>=====>| | | |
|
||||
| | | Updater | | | | |
|
||||
| | +----------+ | | | |
|
||||
| | | | | |
|
||||
| | +---+ +---+ | | +---+ | |
|
||||
| | +------------| W | +------| S | | | +--------| R | | |
|
||||
| | | Federation +---+ | Room +---+ | | | Client +---+ | |
|
||||
| | | Backfill |>=====>| Server |>=====>| |>=====>| Push | | |
|
||||
| | +--------------+ | | +------------+ | | | |
|
||||
| | | | | | | |
|
||||
| | | |>==========================>| | | |
|
||||
| | | | +----------+ | |
|
||||
| | | | | |
|
||||
| | | | +---+ | |
|
||||
| | | | +--------| R | | |
|
||||
| | | | | Client +---+ | |
|
||||
| |>========================>| |>==========================>| Search | | |
|
||||
| | | | | | | |
|
||||
| | | | +----------+ | |
|
||||
| | | | | |
|
||||
| | | |>==========================================>| |
|
||||
| | | | | |
|
||||
| | +---+ | | +---+ | |
|
||||
| | +--------| W | | | +----------| S | | |
|
||||
| | | Client +---+ | | | Presence +---+ | |
|
||||
| | | Room |>=====>| |>=====>| Server |>=====================>| |
|
||||
| | | Send | +--------+ | | | |
|
||||
| | | | | | | |
|
||||
| | | |>======================>| |<=====================<| |
|
||||
| | +----------+ | | | |
|
||||
| | | | | |
|
||||
| | +---+ | | | |
|
||||
| | +--------| W | | | | |
|
||||
| | | Client +---+ | | | |
|
||||
| | | Presence |>=====>| | | |
|
||||
| | | Setter | | | | |
|
||||
| | +----------+ | | | |
|
||||
| | | | | |
|
||||
| | | | | |
|
||||
| |>=========================================>| | | |
|
||||
| | +------------+ | |
|
||||
| | | |
|
||||
| | +---+ | |
|
||||
| | +----------| S | | |
|
||||
| | | Typing +---+ | |
|
||||
| |>=========================================>| Server |>=====================>| |
|
||||
+------------+ | | +----------+
|
||||
+---+ | |
|
||||
+--------| W | | |
|
||||
| Client +---+ | |
|
||||
| Typing |>=====>| |
|
||||
| Setter | | |
|
||||
+----------+ +------------+
|
||||
|
||||
|
||||
# Component Descriptions
|
||||
|
||||
Many of the components are logical rather than physical. For example it is
|
||||
possible that all of the client API writers will end up being glued together
|
||||
and always deployed as a single unit.
|
||||
|
||||
Outbound federation requests will probably need to be funnelled through a
|
||||
choke-point to implement ratelimiting and backoff correctly.
|
||||
|
||||
## Federation Send
|
||||
|
||||
* Handles `/federation/v1/send/` requests.
|
||||
* Fetches missing ``prev_events`` from the remote server if needed.
|
||||
* Fetches missing room state from the remote server if needed.
|
||||
* Checks signatures on remote events, downloading keys if needed.
|
||||
* Queries information needed to process events from the Room Server.
|
||||
* Writes room events to logs.
|
||||
* Writes presence updates to logs.
|
||||
* Writes receipt updates to logs.
|
||||
* Writes typing updates to logs.
|
||||
* Writes other updates to logs.
|
||||
|
||||
## Client Room Send
|
||||
|
||||
* Handles puts to `/client/v1/rooms/` that create room events.
|
||||
* Queries information needed to process events from the Room Server.
|
||||
* Talks to remote servers if needed for joins and invites.
|
||||
* Writes room event pdus.
|
||||
* Writes presence updates to logs.
|
||||
|
||||
## Client Presence Setter
|
||||
|
||||
* Handles puts to whatever the client API path for presence is?
|
||||
* Writes presence updates to logs.
|
||||
|
||||
## Client Typing Setter
|
||||
|
||||
* Handles puts to whatever the client API path for typing is?
|
||||
* Writes typing updates to logs.
|
||||
|
||||
## Client Receipt Updater
|
||||
|
||||
* Handles puts to whatever the client API path for receipts is?
|
||||
* Writes typing updates to logs.
|
||||
|
||||
## Federation Backfill
|
||||
|
||||
* Backfills events from other servers
|
||||
* Writes the resulting room events to logs.
|
||||
* Is a different component from the room server itself cause it'll
|
||||
be easier if the room server component isn't making outbound HTTP requests
|
||||
to remote servers
|
||||
|
||||
## Room Server
|
||||
|
||||
* Reads new and backfilled room events from the logs written by FS, FB and CRS.
|
||||
* Tracks the current state of the room and the state at each event.
|
||||
* Probably does auth checks on the incoming events.
|
||||
* Handles state resolution as part of working out the current state and the
|
||||
* state at each event.
|
||||
* Writes updates to the current state and new events to logs.
|
||||
* Shards by room ID.
|
||||
|
||||
## Receipt Server
|
||||
|
||||
* Reads new updates to receipts from the logs written by the FS and CRU.
|
||||
* Somehow learns enough information from the room server to workout how the
|
||||
current receipt markers move with each update.
|
||||
* Writes the new marker positions to logs
|
||||
* Shards by room ID?
|
||||
* It may be impossible to implement without folding it into the Room Server
|
||||
forever coupling the components together.
|
||||
|
||||
## Typing Server
|
||||
|
||||
* Reads new updates to typing from the logs written by the FS and CTS.
|
||||
* Updates the current list of people typing in a room.
|
||||
* Writes the current list of people typing in a room to the logs.
|
||||
* Shards by room ID?
|
||||
|
||||
## Presence Server
|
||||
|
||||
* Reads the current state of the rooms from the logs to track the intersection
|
||||
of room membership between users.
|
||||
* Reads updates to presence from the logs writen by the FS and the CPS.
|
||||
* Reads when clients sync from the logs from the Client Sync.
|
||||
* Tracks any timers for users.
|
||||
* Writes the changes to presence state to the logs.
|
||||
* Shards by user ID somehow?
|
||||
|
||||
## Client Sync
|
||||
|
||||
* Handle /client/v2/sync requests.
|
||||
* Reads new events and the current state of the rooms from logs written by the Room Server.
|
||||
* Reads new receipts positions from the logs written by the Receipts Server.
|
||||
* Reads changes to presence from the logs written by the Presence Server.
|
||||
* Reads changes to typing from the logs written by the Typing Server.
|
||||
* Writes when a client starts and stops syncing to the logs.
|
||||
|
||||
## Client Search
|
||||
|
||||
* Handle whatever the client API path for event search is?
|
||||
* Reads new events and the current state of the rooms from logs writeen by the Room Server.
|
||||
* Maintains a full text search index of somekind.
|
||||
|
||||
## Client Push
|
||||
|
||||
* Pushes unread messages to remote push servers.
|
||||
* Reads new events and the current state of the rooms from logs writeen by the Room Server.
|
||||
* Reads the position of the read marker from the Receipts Server.
|
||||
* Makes outbound HTTP hits to the push server for the client device.
|
||||
Loading…
Reference in New Issue