Commit Graph

1489 Commits (8b22c4270da28bb6e968f0804da8e2dc4a865651)

Author SHA1 Message Date
Kegsay 1414922026
sytest: Make 'Inbound federation can backfill events' pass (#1051)
* sytest: Make 'Inbound federation can backfill events' pass

This breaks 'Outbound federation can backfill events' because now
we are returning the right number of events, which the previous
test was relying on.

Previously, /messages was backfilling the membership event, causing
the test to pass. Now we are no longer backfilling the membership
event due to the change in this commit, causing the test to fail.

The test should instead be returning the membership event locally
from synacpis database, but it doesn't do it fast enough, resulting
in a no-op /sync response with a next_batch=s0_0 which will never
pick up the local membership event when it rolls in. The test
does attempt to retry, but doesn't take the new next_batch=s1_0
resulting in it missing from the /messages response.

* Linting
2020-05-20 16:04:31 +01:00
Kegsay 260e69d138
Make "Outbound federation can backfill events" pass sytest (#1049)
- Use a backfill limit of 100 regardless of what was asked.
- Special case the create event for `StateIDsBeforeEvent`
- Trim to the limit in `syncapi`
2020-05-19 18:42:55 +01:00
Kegsay 5faecdac82
Bake in git commit into dendritejs binary (#1048) 2020-05-19 16:42:30 +01:00
Kegsay 8b3100935c
dendritejs: Persist ed25519 keys in localforage (#1046) 2020-05-19 12:09:52 +01:00
Kegan Dougal b730e21892 bump go-http-js-libp2p 2020-05-18 18:12:11 +01:00
Neil Alexander 9ef30bb13b
Tweaks to latest events updater (#1045)
* Comment out updaters a bit, add overwrite flag to latest events

* Make sure we don't send fast-forwarded state changes over federation, start with empty set when overwriting

* Remove redundant check for overwrite
2020-05-18 17:49:24 +01:00
Kegan Dougal 5c221f0655 Latest go-http-js-libp2p 2020-05-18 15:51:49 +01:00
Neil Alexander dce4f436f7
Add -api flag to monolith (#1044)
* Add flag for enabling HTTP APIs in monolith mode

* Flag -api

* Only start HTTP APIs if needed
2020-05-18 10:56:43 +01:00
Kegan Dougal bfb954519b Only keep keys valid for 100 years rather than the heat death of the universe
WASM builds really dislike thinking that far into the future, causing
WASM to trap with "float unrepresentable in integer range", which corrupts
the Go stack in wasm_exec which then leads to a segfault and the program
exiting.
2020-05-15 17:39:20 +01:00
Kegsay 8db60c90bb
Fix a bug whereby backfilling could leak events across rooms (#1043)
* Fix a bug whereby backfilling could leak events across rooms

Caused by a faulty SQL query. With tests now.

* comment
2020-05-15 16:27:34 +01:00
Neil Alexander 5f6f8adaa5
Don't prematurely stop trying to join using servers (#1041)
* Don't prematurely stop trying to join using servers

* Factor out performJoinUsingServer
2020-05-15 13:55:14 +01:00
Kegsay f4f032381b
Set canonical alias on room creation (#1042) 2020-05-15 13:39:01 +01:00
Neil Alexander 773d5bb9f9
Return user_id and stream_id in federated devices query (#1040) 2020-05-15 12:07:51 +01:00
Neil Alexander f0e0a6668f
Prometheus metrics for LRU cache (#1039)
* Add prom metrics for the in-memory LRU cache

* Increase cache sizes
2020-05-15 11:27:10 +01:00
Kegsay 2b5052eccf
Add Range (#1037)
* Add Range

* Use Range
2020-05-15 09:41:12 +01:00
Neil Alexander 419ff150d4
Implement key caching directly (#1038)
* Use gomatrixserverlib key caching

* Implement key caching wrapper

* Add caching wrapper in BaseComponent

* Review comments
2020-05-15 09:32:40 +01:00
Kegsay 7ca230e931
Cleanup syncapi topology logic (#1035)
* Cleanup syncapi topology logic

* Variable renaming

* comments
2020-05-14 17:30:16 +01:00
Neil Alexander 3cb04e8004
Update INSTALL.md, move docs (#1034)
* Update INSTALL.md

* Move some things

* Clean up

* Move some more things

* Don't build all the things for the monolith

* Update INSTALL.md

* Nuke hooks
2020-05-14 16:49:18 +01:00
Kegsay 1b34130a5b
Finish merging syncserver.go (#1033)
* Refactor all postgres tables; start work on sqlite

* wip sqlite merges; database is locked errors to investigate and failing tests

* Revert "wip sqlite merges; database is locked errors to investigate and failing tests"

This reverts commit 26cbfc5b75ae2dc4fb31a838b917aa39d758f162.

* convert current room state table

* port over sqlite topology table

* remove a few functions

* remove more functions

* Share more code

* factor out completesync and a bit more

* Remove remaining code
2020-05-14 16:11:37 +01:00
Neil Alexander 640a0265df
Fix some cases where accepting invites over federation doesn't work (#1028)
* Handle cases where accepting invites doesn't work for historic rooms

* Rewrite pairUpChanges

* Review comments
2020-05-14 14:58:47 +01:00
Neil Alexander 8adc128225
Keyserver skeleton (#1032)
* Keyserver skeleton

* Indentation
2020-05-14 14:05:14 +01:00
Kegsay 9ed68a3125
Factor out account data and events table (#1031)
* Factor out account data

* Factor out events table and EDU cache

* linting

* fix npe
2020-05-14 09:53:55 +01:00
Kegsay a25d477cdb
Initial syncapi storage refactor to share pq/sqlite code (#1030)
* Initial syncapi storage refactor to share pq/sqlite code

This goes down a different route than https://github.com/matrix-org/dendrite/pull/985
which tried to even reduce the boilerplate of `ExecContext` etc. The previous pattern
fails badly when there are subtle differences in parameters and hence the shared
boilerplate to read from `QueryContext` breaks. Rather than attacking it at that level,
the main place where we want to reuse code is for the `syncserver.go` itself - the
database implementation which has lots of complex logic. So instead, this commit:
 - Makes `invites_table.go` an interface.
 - Makes `SyncServerDatasource` use that interface
 - This means some functions are now identical for pq/sqlite, so factor them out
   to a temporary `shared.Database` struct which will grow until it replaces all of
   `SyncServerDatasource`.

* Missing files
2020-05-13 17:28:42 +01:00
Kegan Dougal bdddd83753 Spurious logging 2020-05-13 15:00:47 +01:00
Neil Alexander 334680a216
Get profile content for join request again (#1027) 2020-05-13 14:53:25 +01:00
Kegsay 37b2299b71
Fix #1015 (#1026)
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2020-05-13 13:02:34 +01:00
Neil Alexander ee140c9d6a
Reduce 500s (#1017)
* Try to avoid returning 500s on /send

* Don't return 500s from media API download requests

* Don't 500 on context errors

* Update sytest-whitelist

* Fix lint, add comments
2020-05-13 13:01:45 +01:00
Kegsay 5e9dce1c0c
syncapi: Rename and split out tokens (#1025)
* syncapi: Rename and split out tokens

Previously we used the badly named `PaginationToken` which was
used for both `/sync` and `/messages` requests. This quickly
became confusing because named fields like `PDUPosition` meant
different things depending on the token type. Instead, we now have
two token types: `TopologyToken` and `StreamingToken`, both of
which have fields which make more sense for their specific situations.

Updated the codebase to use one or the other. `PaginationToken` still
lives on as `syncToken`, an unexported type which both tokens rely on.
This allows us to guarantee that the specific mappings of positions
to a string remain solely under the control of the `types` package.
This enables us to move high-level conceptual things like
"decrement this topological token" to function calls e.g
`TopologicalToken.Decrement()`.

Currently broken because `/messages` seemingly used both stream and
topological tokens, though I need to confirm this.

* final tweaks/hacks

* spurious logging

* Review comments and linting
2020-05-13 12:14:50 +01:00
Sid Karunaratne 31e6a7f193
Enforce `mediaIDRegex` to be only valid `mediaIDCharacters` (#1020)
Error messages indicate that:
> mediaId must be a non-empty string using only characters in `mediaIDCharacters`

However the regex used only required that some characters in the filename match
the restriction, not that the entire filename does. This commit ensures that
the filename must entirely fullfill the `mediaIDCharacters` restriction

Signed-off-by: Sid Karunaratne <sid@karunaratne.net>

Co-authored-by: Kegsay <kegan@matrix.org>
2020-05-13 12:04:54 +01:00
Neil Alexander 106a335b7d Update sytest-whitelist 2020-05-12 17:49:04 +01:00
Kegsay ce5dfbebf9
Implement /get_missing_events (#1022)
* WIP get_missing_events work

* More WIP get_missing_events work

* First working /get_missing_events implementation

Flakey currently due to racing between /sync and /send

* Final tweaks

* Remove log lines

* Linting

* go mod tidy

* Clamp min depth to 0

* sort events by depth because sytest makes me sad

Specifically I think it's
4172585c25/lib/SyTest/Federation/Client.pm (L265)
to blame here.
2020-05-12 16:24:28 +01:00
Neil Alexander 32624697fd
Add PPROFLISTEN (#1019)
* Add PPROFLISTEN env var

* Direct logging to more useful places

* Space
2020-05-11 18:21:39 +01:00
Neil Alexander 0c892d59fa
Prevent panic in membership updater (#1021) 2020-05-11 18:21:25 +01:00
Neil Alexander 99e0a7dff2 Update gomatrixserverlib for even more memory-efficient state res v2 2020-05-11 16:43:50 +01:00
Neil Alexander 615de25347 Update gomatrixserverlib for more memory-efficient state res v2 2020-05-11 16:02:23 +01:00
Neil Alexander 6e643860b1 Update sytest-whitelist 2020-05-11 14:01:15 +01:00
Neil Alexander 4fd97df2c5 Don't return 500s from media API download requests 2020-05-11 14:01:07 +01:00
Will Hunt d6e18a33ce
Add registration_disabled to dendrite-config.yaml (#1013)
Missed this when first setting up my instance, and it feels important enough that it should be part of the sample config
2020-05-08 12:00:32 +01:00
Neil Alexander 17d27331a3 Fix 'input to Unique() must be sorted' panic 2020-05-07 17:14:32 +01:00
Neil Alexander c8e11dfe53
Direct messages (#1012)
* Initial DM support, include invite event in stripped state for regular invites

* Update go.mod, go.sum, test list
2020-05-07 16:46:11 +01:00
Neil Alexander a16db1c408
Improve federation sender performance, implement backoff and blacklisting, fix up invites a bit (#1007)
* Improve federation sender performance and behaviour, add backoff

* Tweaks

* Tweaks

* Tweaks

* Take copies of events before passing to destination queues

* Don't accidentally drop queued messages

* Don't take copies again

* Tidy up a bit

* Break out statistics (tracked component-wide), report success and failures from Perform actions

* Fix comment, use atomic add

* Improve logic a bit, don't block on wakeup, move idle check

* Don't retry sucessful invites, don't dispatch sendEvent, sendInvite etc

* Dedupe destinations, fix other bug hopefully

* Dispatch sends again

* Federation sender to ignore invites that are destined locally

* Loopback invite events

* Remodel a bit with channels

* Linter

* Only loopback invite event if we know the room

* We should tell other resident servers about the invite if we know about the room

* Correct invite signing

* Fix invite loopback

* Check HTTP response codes, push new invites to front of queue

* Review comments
2020-05-07 12:42:06 +01:00
Kegsay 3b98535dc5
only send new events to RS; add tests for /state_ids and /event (#1011)
* only send new events to RS; add tests for /state_ids and /event

* Review comments: send in auth event order

* Ignore order of state events for this test as RespState.Events is non-deterministic
2020-05-06 18:03:25 +01:00
Kegsay 1294852270
Add tests around federationapi's txnReq (#1010)
* Add necessary stubs for testing txnReq

* Add basic tests
2020-05-06 14:27:02 +01:00
Kegsay 1db5dfe4d0
Fetch events by ID rather than use current state as this includes auth events (#1009) 2020-05-05 16:46:22 +01:00
Kegan Dougal a06511cae8 Remove debug line 2020-05-05 15:54:12 +01:00
Kegsay 31d3b0d4a5
Prefer /state_ids when missing state across federation (#1008)
* Prefer /state_ids when missing state across federation

* Linting

* Better logging
2020-05-05 15:48:37 +01:00
Neil Alexander 9d15312ef6
Fix RespState/RespSendJoin (#1005)
* Update gmsl

* Update gomatrixserverlib

* Add link to spec
2020-05-05 10:53:38 +01:00
Neil Alexander 9b1b095b49
Roomserver perform leave (#1004)
* First pass at PerformLeave

* Fix SQLite bulkSelectEventStateKey

* Update gomatrixserverlib

* Fix bugs

* Tidy a bit

* Satisfy King Linter

* Review comments

* Review comments

* Fix constants in SQLite event state keys table
2020-05-04 18:34:09 +01:00
Neil Alexander df80e42408 Update gomatrixserverlib 2020-05-04 15:30:36 +01:00
Andrew Morgan 65c6fbddeb
Fix newlines between white/blacklist test names in buildkite annotations (#1003) 2020-05-04 14:48:49 +01:00