Commit Graph

29 Commits (48850d9995aae9b526784ea2d9c5ff8ce3a8938e)

Author SHA1 Message Date
Neil Alexander bcdf9577a3
Support for server ACLs (#1261)
* First pass at server ACLs (not efficient)

* Use transaction origin, update whitelist

* Fix federation API test

It's sufficient for us to return nothing in response to current state, so that the server ACL check returns no ACLs.

* More efficient server ACLs - hopefully

* Fix queries

* Fix queries

* Avoid panics by nil pointers

* Bug fixes

* Fix state event type

* Fix mutex

* Update logging

* Ignore port when matching servername

* Use read mutex

* Fix bugs

* Fix sync API test

* Comments

* Add tests, tweaks to behaviour

* Fix test output
2020-08-11 18:19:11 +01:00
Neil Alexander 4b09f445c9
Configuration format v1 (#1230)
* Initial pass at refactoring config (not finished)

* Don't forget current state and EDU servers

* More shifting around

* Update server key API tests

* Fix roomserver test

* Fix more tests

* Further tweaks

* Fix current state server test (sort of)

* Maybe fix appservices

* Fix client API test

* Include database connection string in database options

* Fix sync API build

* Update config test

* Fix unit tests

* Fix federation sender build

* Fix gobind build

* Set Listen address for all services in HTTP monolith mode

* Validate config, reinstate appservice derived in directory, tweaks

* Tweak federation API test

* Set MaxOpenConnections/MaxIdleConnections to previous values

* Update generate-config
2020-08-10 14:18:04 +01:00
Neil Alexander 5dd5a41119
Tweak log levels of some federation logging (#1248)
* Tweak log levels of some federation logging

* Update go.mod/go.sum for matrix-org/util#22 and matrix-org/gomatrixserverlib#215
2020-08-07 15:00:23 +01:00
Kegsay 642f9cb964
Process inbound device list updates from federation (#1240)
* Add InputDeviceListUpdate

* Unbreak unit tests

* Process inbound device list updates from federation

- Persist the keys in the keyserver and produce key changes
- Does not currently fetch keys from the remote server if the prev IDs are missing

* Linting
2020-08-05 13:41:16 +01:00
Neil Alexander 4cf45d1ce9
Don't include current state in processEventWithMissingState (#1126)
* Don't include current state in processEventWithMissingState

* Remove lookupCurrentState as not needed

Co-authored-by: Kegsay <kegan@matrix.org>
2020-06-29 14:39:21 +01:00
Kegsay 914f6cadce
Add /send restrictions and return correct error codes (#1156)
* Add /send restrictions and return correct error codes

- Max 50 PDUs / 100 EDUs
- Fail the transaction when PDUs contain bad JSON

* Update whitelist

* Unbreak test

* Linting
2020-06-23 13:15:15 +01:00
Kegsay 02565c37aa
/send auth errors are silent (#1149)
* /send auth errors are silent

* Fix test
2020-06-23 10:31:17 +01:00
Kegsay 7c36fb78a7
Fix rooms v3 url paths for good - with tests (#1130)
* Fix rooms v3 url paths for good - with tests

- Add a test rig around `federationapi` to test routing.
- Use `JSONVerifier` over `KeyRing` so we can stub things out more easily.
- Add `test.NopJSONVerifier` which verifies nothing.
- Add `base.BaseMux` which is the original `mux.Router` used to spawn public/internal routers.
- Listen on `base.BaseMux` and not the default serve mux as it cleans paths which we don't want.
- Factor out `ListenAndServe` to `test.ListenAndServe` and add flag for listening on TLS.

* Fix comments

* Linting
2020-06-15 16:57:59 +01:00
Kegsay b7187a9a35
Remove clientapi producers which aren't actually producers (#1111)
* Remove clientapi producers which aren't actually producers

They are actually just convenience wrappers around the internal APIs
for roomserver/eduserver. Move their logic to their respective `api`
packages and call them directly.

* Remove TODO

* unbreak ygg
2020-06-10 12:17:54 +01:00
Neil Alexander 76ff47c052
Use AuthChainProvider to try and speed up federated joins (#1100)
* Use MissingAuthEventHandler on performjoin to try and speed up cases where we have missing events

* Update gomatrixserverlib

* Use supplied room version

* Use AuthChainProvider

* Tweaks

* Update gomatrixserverlib

* Signature checks
2020-06-05 11:48:52 +01:00
Neil Alexander a5d822004d
Send-to-device support (#1072)
* Groundwork for send-to-device messaging

* Update sample config

* Add unstable routing for now

* Send to device consumer in sync API

* Start the send-to-device consumer

* fix indentation in dendrite-config.yaml

* Create send-to-device database tables, other tweaks

* Add some logic for send-to-device messages, add them into sync stream

* Handle incoming send-to-device messages, count them with EDU stream pos

* Undo changes to test

* pq.Array

* Fix sync

* Logging

* Fix a couple of transaction things, fix client API

* Add send-to-device test, hopefully fix bugs

* Comments

* Refactor a bit

* Fix schema

* Fix queries

* Debug logging

* Fix storing and retrieving of send-to-device messages

* Try to avoid database locks

* Update sync position

* Use latest sync position

* Jiggle about sync a bit

* Fix tests

* Break out the retrieval from the update/delete behaviour

* Comments

* nolint on getResponseWithPDUsForCompleteSync

* Try to line up sync tokens again

* Implement wildcard

* Add all send-to-device tests to whitelist, what could possibly go wrong?

* Only care about wildcard when targeted locally

* Deduplicate transactions

* Handle tokens properly, return immediately if waiting send-to-device messages

* Fix sync

* Update sytest-whitelist

* Fix copyright notice (need to do more of this)

* Comments, copyrights

* Return errors from Do, fix dendritejs

* Review comments

* Comments

* Constructor for TransactionWriter

* defletions

* Update gomatrixserverlib, sytest-blacklist
2020-06-01 17:50:19 +01:00
Neil Alexander 406b47267e
Return 500 when processing a transaction fails fatally (#1066) 2020-05-27 11:16:27 +01:00
Kegsay 24d8df664c
Fix #897 and shuffle directory around (#1054)
* Fix #897 and shuffle directory around

* Update find-lint

* goimports

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2020-05-21 14:40:13 +01:00
Neil Alexander ee140c9d6a
Reduce 500s (#1017)
* Try to avoid returning 500s on /send

* Don't return 500s from media API download requests

* Don't 500 on context errors

* Update sytest-whitelist

* Fix lint, add comments
2020-05-13 13:01:45 +01:00
Kegsay ce5dfbebf9
Implement /get_missing_events (#1022)
* WIP get_missing_events work

* More WIP get_missing_events work

* First working /get_missing_events implementation

Flakey currently due to racing between /sync and /send

* Final tweaks

* Remove log lines

* Linting

* go mod tidy

* Clamp min depth to 0

* sort events by depth because sytest makes me sad

Specifically I think it's
4172585c25/lib/SyTest/Federation/Client.pm (L265)
to blame here.
2020-05-12 16:24:28 +01:00
Kegsay 3b98535dc5
only send new events to RS; add tests for /state_ids and /event (#1011)
* only send new events to RS; add tests for /state_ids and /event

* Review comments: send in auth event order

* Ignore order of state events for this test as RespState.Events is non-deterministic
2020-05-06 18:03:25 +01:00
Kegsay 1294852270
Add tests around federationapi's txnReq (#1010)
* Add necessary stubs for testing txnReq

* Add basic tests
2020-05-06 14:27:02 +01:00
Kegsay 1db5dfe4d0
Fetch events by ID rather than use current state as this includes auth events (#1009) 2020-05-05 16:46:22 +01:00
Kegsay 31d3b0d4a5
Prefer /state_ids when missing state across federation (#1008)
* Prefer /state_ids when missing state across federation

* Linting

* Better logging
2020-05-05 15:48:37 +01:00
Neil Alexander e15f6676ac
Consolidation of roomserver APIs (#994)
* Consolidation of roomserver APIs

* Comment out alias tests for now, they are broken

* Wire AS API into roomserver again

* Roomserver didn't take asAPI param before so return to that

* Prevent roomserver asking AS API for alias info

* Rename some files

* Remove alias_test, incoherent tests and unwanted appservice integration

* Remove FS API inject on syncapi component
2020-05-01 10:48:17 +01:00
Neil Alexander 3c2e6f967b
Federation fixes and error handling (#970)
* Improve error handling in federation /send endpoint a bit

* Remove unknownRoomError, use unmarshalError when unable to get room ID

* Swap out a couple more internal server errors

* Update gomatrixserverlib

* Update gomatrixserverlib

* Update gomatrixserverlib

* Update gomatrixserverlib

* Update gomatrixserverlib

* Update gomatrixserverlib

* Return bad limit in error

* Same with domain/userid
2020-04-16 17:59:55 +01:00
Kegsay 8fbe9f4078
Implement typing over federation (#949)
Also fix a pet peeve of mine: not putting units on things!!!

Manually tested on p2p and works well, with some fudge factor delay.
2020-03-30 16:40:28 +01:00
Neil Alexander 05e1ae8745
Further room version wiring (#936)
* Room version 2 by default, other wiring updates, update gomatrixserverlib

* Fix nil pointer exception

* Fix some more nil pointer exceptions hopefully

* Update gomatrixserverlib

* Send all room versions when joining, not just stable ones

* Remove room version cquery

* Get room version when getting events from the roomserver database

* Reset default back to room version 2

* Don't generate event IDs unless needed

* Revert "Remove room version cquery"

This reverts commit a170d5873360dd059614460acc8b21ab2cda9767.

* Query room version in federation API, client API as needed

* Improvements to make_join send_join dance

* Make room server producers use headered events

* Lint tweaks

* Update gomatrixserverlib

* Versioned SendJoin

* Query room version in syncapi backfill

* Handle transaction marshalling/unmarshalling within Dendrite

* Sorta fix federation (kinda)

* whoops commit federation API too

* Use NewEventFromTrustedJSON when getting events from the database

* Update gomatrixserverlib

* Strip headers on federationapi endpoints

* Fix bug in clientapi profile room version query

* Update gomatrixserverlib

* Return more useful error if room version query doesn't find the room

* Update gomatrixserverlib

* Update gomatrixserverlib

* Maybe fix federation

* Fix formatting directive

* Update sytest whitelist and blacklist

* Temporarily disable room versions 3 and 4 until gmsl is fixed

* Fix count of EDUs in logging

* Update gomatrixserverlib

* Update gomatrixserverlib

* Update gomatrixserverlib

* Rely on EventBuilder in gmsl to generate the event IDs for us

* Some review comments fixed

* Move function out of common and into gmsl

* Comment in federationsender destinationqueue

* Update gomatrixserverlib
2020-03-27 16:28:22 +00:00
Kegsay 9f74a8798e
bugfix: Fix #908 by setting the correct state after the event (#913)
* bugfix: Fix #908 by setting the correct state after the event

Previously, this would only happen if the state already existed
previously!

* Structured logging
2020-03-16 17:51:58 +00:00
Neil Alexander acb505b717
Implement gomatrixserverlib.HeaderedEvent in roomserver query API (#912)
* Implement gomatrixserverlib.HeaderedEvent, which should allow us to store room version headers along with the event across API boundaries and consumers/producers, and intercept unmarshalling to get the event structure right

* Add federationsender to previous
2020-03-16 17:29:52 +00:00
Neil Alexander 6a1111c3d4
Try to recursively find auth events (to a point) if they are missing (#881)
* Try to recursively find auth events (to a point) if they are missing

* Remove recursion limit for now and other review fixes

* Simplify error handling for recursion

* Pass room version 1 only to MakeJoin until room version support comes later
2020-03-06 16:58:10 +00:00
Neil Alexander 59a1f4b8ed
Remove httputil.LogThenError so that the line numbers are reported properly - make error reporting slightly more useful (#879) 2020-03-02 16:20:44 +00:00
Behouba Manassé 9937c05bea
Pass cfg by reference around the codebase (#819)
* Pass cfg by reference around the codebase

* Merge branch 'master' into pass-cfg-by-ref

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2020-02-11 11:18:12 +00:00
ruben 74827428bd use go module for dependencies (#594) 2019-05-21 21:56:55 +01:00