Commit Graph

74 Commits (15dc1f4d0361da736339653ca8e6ba26ed103792)

Author SHA1 Message Date
Kegan Dougal 2197e54441 Update whitelist 2020-08-04 13:50:32 +01:00
Kegsay fb56bbf0b7
Generate stream IDs for locally uploaded device keys (#1236)
* Breaking: add stream_id to keyserver_device_keys table

* Add tests for stream ID generation

* Fix whitelist
2020-08-03 17:07:06 +01:00
Kegsay b5cb1d1534
Fix edge cases around device lists (#1234)
* Fix New users appear in /keys/changes

* Create blank device keys when logging in on a new device

* Add PerformDeviceUpdate and fix a few bugs

- Correct device deletion query on sqlite
- Return no keys on /keys/query rather than an empty key

* Unbreak sqlite properly

* Use a real DB for currentstateserver integration tests

* Race fix
2020-07-31 14:40:45 +01:00
Kegsay a7e67e65a8
Notify clients when devices are deleted (#1233)
* Recheck device lists when join/leave events come in

* Add PerformDeviceDeletion

* Notify clients when devices are deleted

* Unbreak things

* Remove debug logging
2020-07-30 18:00:56 +01:00
Kegsay a2174d3294
Implement /keys/changes (#1232)
* Implement /keys/changes

And refactor QueryKeyChanges to accept a `to` offset.

* Unbreak tests

* Sort keys when serialising log tokens
2020-07-30 14:52:21 +01:00
Kegsay 9355fb5ac8
Hook up device list updates to the sync notifier (#1231)
* WIP hooking up key changes

* Fix import cycle, get tests passing and binary compiling

* Linting and update whitelist
2020-07-30 11:15:46 +01:00
Neil Alexander acc8e80a51
User directory (#1225)
* User directory

* Fix syncapi unit test

* Make user directory only show remote users you know about from your joined rooms

* Update sytest-whitelist

* Review comments
2020-07-28 10:53:17 +01:00
Kegsay c632867135
Modify /state/{eventType}/{stateKey} to return the event at the time the user left (#1222)
* Modify /state/{eventType}/{stateKey} to return the event at the time the user left

Or live, depending on their current state. Hopefully fixes some sytests!

* Linting

* Set HasBeenInRoom

* Fix cases for world-readable history visibility

* Fix bug in finding the requested state event

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2020-07-28 10:09:10 +01:00
Kegsay c8d476a3cc
Return HTTP errors when trying to kick invalid users (#1221)
Room integrity was never compromised as GMSL does auth checks,
but we would incorrectly 200 OK the request instead of 403ing.
2020-07-27 09:20:09 +01:00
Kegsay 541a23f712
Handle inbound federation E2E key queries/claims (#1215)
* Handle inbound /keys/claim and /keys/query requests

* Add display names to device key responses

* Linting
2020-07-22 17:04:57 +01:00
Kegsay 1d72ce8b7a
Implement claiming one-time keys locally (#1210)
* Add API shape for claiming keys

* Implement claiming one-time keys locally

Fairly boring, nothing too special going on.
2020-07-21 14:47:53 +01:00
Kegsay f5e7e7513c
Implement /keys/query locally (#1204)
* Implement /keys/query locally

* Fix sqlite tests and close rows
2020-07-15 18:40:41 +01:00
Kegsay 9dd2ed7f65
Implement key uploads (#1202)
* Add storage layer for postgres/sqlite

* Return OTK counts when inserting new keys

* Hook up the key DB and make a test pass

* Convert postgres queries to be sqlite queries

* Blacklist test due to requiring rejected events

* Unbreak tests

* Update blacklist
2020-07-15 12:02:34 +01:00
Neil Alexander 72b3160776
Send-to-device messages over federation (#1198)
* Initial work to send send-to-device messages over federation

* Wire up send-to-device consumer, message formatting

* Generate random message ID

* Review comments, update sytest whitelist
2020-07-14 12:33:37 +01:00
Kegsay abf26c12f1
Add User-Interactive Authentication (#1193)
* Add User-Interactive Authentication

And use it when deleting a device. With tests.

* Make remaining sytest pass

* Linting

* 403 not 401 on wrong user/pass
2020-07-10 00:39:44 +01:00
Kegsay d9648b0615
Finish implementing redactions (#1189)
* Add a bit more logging to the fedsender

* bugfix: continue sending PDUs if ones are added whilst sending another PDU

Without this, the queue goes back to sleep on `<-oq.notifyPDUs` which won't
fire because `pendingPDUs` is already > 0. This should fix a flakey sytest.

* Break if no txn is sent

* WIP syncapi work

* More debugging

* Bump GMSL version to pull in working Event.Redact

* Remove logging

* Make redactions work on v3+

* Fix more tests
2020-07-08 17:45:39 +01:00
Kegsay ea9df46c70
Implement local redaction (#1182)
* Create redaction events and apply checks (but do not send them)

* Send redactions to the roomserver

* Linting

* Slightly better wording
2020-07-03 17:24:51 +01:00
Kegsay 4c1e6597c0
Replace publicroomsapi with a combination of clientapi/roomserver/currentstateserver (#1174)
* Use content_value instead of membership

* Fix build

* Replace publicroomsapi with a combination of clientapi/roomserver/currentstateserver

- All public rooms paths are now handled by clientapi
- Requests to (un)publish rooms are sent to the roomserver via `PerformPublish`
  which are stored in a new `published_table.go`
- Requests for public rooms are handled in clientapi by:
    * Fetch all room IDs which are published using `QueryPublishedRooms` on the roomserver.
    * Apply pagination parameters to the slice.
    * Do a `QueryBulkStateContent` request to the currentstateserver to pull out
      required state event *content* (not entire events).
    * Aggregate and return the chunk.

Mostly but not fully implemented (DB queries on currentstateserver are missing)

* Fix pq query

* Make postgres work

* Make sqlite work

* Fix tests

* Unbreak pagination tests

* Linting
2020-07-02 15:41:18 +01:00
Kegsay 1ad7219e4b
Implement /sync `limited` and read timeline limit from stored filters (#1168)
* Move filter table to syncapi where it is used

* Implement /sync `limited` and read timeline limit from stored filters

We now fully handle `room.timeline.limit` filters (in-line + stored) and
return the right value for `limited` syncs.

* Update whitelist

* Default to the default timeline limit if it's unset, also strip the extra event correctly

* Update whitelist
2020-06-26 15:34:41 +01:00
Kegsay 164057a3be
Honour event size limits and return 413 (#1167) 2020-06-26 12:51:54 +01:00
Kegsay 4897beabee
Finish implementing retiring invites (#1166)
* Pass retired invites to the syncapi with the event ID of the invite

* Implement retire invite streaming

* Update whitelist
2020-06-26 11:07:52 +01:00
Kegan Dougal c1d2382e6d Reject non-numeric ports (done in GMSL) 2020-06-25 18:05:41 +01:00
Kegsay 7a8282fccf
Handle invite v1 (#1165)
* Implement invite v1 for sytest mainly

* Bump gmsl version which falls back to invite v1 if v2 404s

* Update whitelist
2020-06-25 17:07:40 +01:00
Kegan Dougal 67f7a53f12 Add missing typing test 2020-06-25 15:06:46 +01:00
Kegsay 43cddfe00f
Return remote errors from FS.PerformJoin (#1164)
* Return remote errors from FS.PerformJoin

Follows the same pattern as PerformJoin on roomserver (no error return).

Also return the right format for incompatible room version errors.

Makes a bunch of tests pass!

* Handle network errors better when returning remote HTTP errors

* Linting

* Fix tests

* Update whitelist, pass network errors through in API=1 mode
2020-06-25 15:04:48 +01:00
Kegsay e560619f76
Refactor SendMembership - make ban test pass (#1160)
* Refactor SendMembership - make ban test pass

* Only check invite auth events for local invites
2020-06-24 18:19:54 +01:00
Kegsay a06d0921c9
Make same membership transitions 403, add sytests (#1161)
* Make same membership transitions 403, add sytests

* Update blacklist
2020-06-24 18:19:14 +01:00
Kegan Dougal ebaaf65c54 This doesn't pass 2020-06-24 13:40:50 +01:00
Kegan Dougal c72d23c8eb Update whitelist 2020-06-24 10:28:03 +01:00
Kegsay 0577bfca55
Pass join errors through internal API boundaries (#1157)
* Pass join errors through internal API boundaries

Required for certain invite sytests. We will need to think of a
better way of handling this going forwards.

* Include m.room.avatar in stripped state; handle trailing slashes when GETing state events

* Update whitelist

* Update whitelist
2020-06-24 09:59:59 +01:00
Kegsay 914f6cadce
Add /send restrictions and return correct error codes (#1156)
* Add /send restrictions and return correct error codes

- Max 50 PDUs / 100 EDUs
- Fail the transaction when PDUs contain bad JSON

* Update whitelist

* Unbreak test

* Linting
2020-06-23 13:15:15 +01:00
Kegsay 4220a374ca
Fix room checks for /state and /state_ids (#1155)
We would return a 403 first (as the server would not be allowed to
see this event) and only then return a 404 if the event is not in
the given room. We now invert those checks for /state and /state_ids
to make the tests pass.
2020-06-23 11:47:48 +01:00
Neil Alexander 7f26b0cd13
Bind build support, further Yggdrasil demo updates (#1152)
* Add gobind builds for Yggdrasil demo

* Massage client API a bit

* Fix build

* Fix gobind build

* Fix gobind client API setup

* Tweaks

* Tweaks

* Update sytest-whitelist, add comment

* Default to sending push rules on initial sync
2020-06-19 13:29:27 +01:00
Kegsay 84a7881468
Make account data sytests pass (#1147) 2020-06-17 17:01:03 +01:00
Kegan Dougal 9b408c19fb Missing sytests 2020-06-17 16:47:21 +01:00
Kegsay 8efeb8eb3b
Return the correct /joined_members response and allow ?format=event (#1146) 2020-06-17 16:21:42 +01:00
Kegsay 8e7c1eda05
Enable more sytests (#1145) 2020-06-17 15:40:37 +01:00
Neil Alexander 23bed196e6
Fallback parsing for Content-Disposition minefield (#1144) 2020-06-17 14:26:45 +01:00
Kegan Dougal c4d9b37492 add test 2020-06-17 13:54:47 +01:00
Neil Alexander 5d5aa0a31d
Media filename handling improvements (#1140)
* Derive content ID from hash+filename but preserve dedupe, improve Content-Disposition handling and ASCII handling

* Linter fix

* Some more comments

* Update sytest-whitelist
2020-06-17 11:53:26 +01:00
Neil Alexander 04c99092a4
Update whitelist for sytest media fix (#1137)
* Update sytest-whitelist, are-we-synapse-yet.list

* Update gomatrixserverlib

* Update gomatrixserverlib

* Loop avoidance

* Return UTF-8 filenames

* Replace quotes only, instead of using strconv.Quote

* Update sytest-whitelist

* Update sytest-whitelist
2020-06-16 18:31:38 +01:00
Neil Alexander 45011579eb Update sytest-whitelist 2020-06-16 14:47:30 +01:00
Neil Alexander 8a6152ca70
Enable room version 6 (#1087)
* Return bad request on CS API /send if bad JSON

* Return some more M_BAD_JSON in the right places

* nolint because damnit gocyclo all I added was a type check for an error

* Update gomatrixserverlib

* Update gomatrixserverlib

* Update sytest-whitelist

* Update gomatrixserverlib

* Update sytest-whitelist

* NotJSON -> BadJSON
2020-06-04 10:53:39 +01:00
Neil Alexander a5d822004d
Send-to-device support (#1072)
* Groundwork for send-to-device messaging

* Update sample config

* Add unstable routing for now

* Send to device consumer in sync API

* Start the send-to-device consumer

* fix indentation in dendrite-config.yaml

* Create send-to-device database tables, other tweaks

* Add some logic for send-to-device messages, add them into sync stream

* Handle incoming send-to-device messages, count them with EDU stream pos

* Undo changes to test

* pq.Array

* Fix sync

* Logging

* Fix a couple of transaction things, fix client API

* Add send-to-device test, hopefully fix bugs

* Comments

* Refactor a bit

* Fix schema

* Fix queries

* Debug logging

* Fix storing and retrieving of send-to-device messages

* Try to avoid database locks

* Update sync position

* Use latest sync position

* Jiggle about sync a bit

* Fix tests

* Break out the retrieval from the update/delete behaviour

* Comments

* nolint on getResponseWithPDUsForCompleteSync

* Try to line up sync tokens again

* Implement wildcard

* Add all send-to-device tests to whitelist, what could possibly go wrong?

* Only care about wildcard when targeted locally

* Deduplicate transactions

* Handle tokens properly, return immediately if waiting send-to-device messages

* Fix sync

* Update sytest-whitelist

* Fix copyright notice (need to do more of this)

* Comments, copyrights

* Return errors from Do, fix dendritejs

* Review comments

* Comments

* Constructor for TransactionWriter

* defletions

* Update gomatrixserverlib, sytest-blacklist
2020-06-01 17:50:19 +01:00
Neil Alexander 6d50212f29
Miscellaneous fixes (#1060)
* Add missing routing for PerformDirectoryLookupRequest

* Tweak output

* Fix some bugs in devices

* Don't default to federated room joins in response to invite

* Update sytest-whitelist

* Update comments

* Return correct room ID from PerformJoin

* Fix appservice and EDU server API setup, update sytest-whitelist

* Update sytest-whitelist
2020-05-26 14:41:16 +01:00
Kegan Dougal 91902df95d Add tests due to https://github.com/matrix-org/sytest/pull/876 2020-05-21 11:51:15 +01:00
Kegan Dougal 7d91ef0616 This now passes on sytest/develop 2020-05-20 18:31:02 +01:00
Kegsay 6091bf044f
sytest: Add remaining backfill tests (#1052)
One failed because of `null` instead of `[]` in HTTP responses.

One failed because we hadn't implemented in-line filter limits!
2020-05-20 17:30:03 +01:00
Kegsay 1414922026
sytest: Make 'Inbound federation can backfill events' pass (#1051)
* sytest: Make 'Inbound federation can backfill events' pass

This breaks 'Outbound federation can backfill events' because now
we are returning the right number of events, which the previous
test was relying on.

Previously, /messages was backfilling the membership event, causing
the test to pass. Now we are no longer backfilling the membership
event due to the change in this commit, causing the test to fail.

The test should instead be returning the membership event locally
from synacpis database, but it doesn't do it fast enough, resulting
in a no-op /sync response with a next_batch=s0_0 which will never
pick up the local membership event when it rolls in. The test
does attempt to retry, but doesn't take the new next_batch=s1_0
resulting in it missing from the /messages response.

* Linting
2020-05-20 16:04:31 +01:00
Kegsay 260e69d138
Make "Outbound federation can backfill events" pass sytest (#1049)
- Use a backfill limit of 100 regardless of what was asked.
- Special case the create event for `StateIDsBeforeEvent`
- Trim to the limit in `syncapi`
2020-05-19 18:42:55 +01:00