Commit Graph

53 Commits (125ea75b2419aa5ebf58506e3fbd03a90eb06d68)

Author SHA1 Message Date
kegsay e3df612953
Add tracing to user API (#1948)
Use the trace version in tests so we can just implement the required API functions.
2021-08-03 11:23:25 +01:00
Kegan Dougal ed4097825b Factor out StatementList to `sqlutil` and use it in `userapi`
It helps with the boilerplate.
2021-07-28 18:30:04 +01:00
Neil Alexander 9e4618000e
Alias key backup endpoints onto /unstable, fix key backup bugs (#1947)
* Default /unstable requests to stable endpoints if not overridden specifically with a custom route

* Rewrite URL

* Try something different

* Fix routing manually

* Fix selectLatestVersionSQL

* Don't return 0 if no backup version exists

* Log more useful error

* fix up replace keys check

* Don't enforce uniqueness on e2e_room_keys_versions_idx

Co-authored-by: kegsay <kegan@matrix.org>
2021-07-28 10:25:45 +01:00
kegsay 32bf14a37c
Key Backups (3/3) : Implement querying keys and various bugfixes (#1946)
* Add querying device keys

Makes a bunch of sytests pass

* Apparently only the current version supports uploading keys

* Linting
2021-07-27 19:29:32 +01:00
kegsay b3754d68fc
Key Backups (2/3) : Add E2E backup key tables (#1945)
* Add PUT key backup endpoints and glue them to PerformKeyBackup

* Add tables for storing backup keys and glue them into the user API

* Don't create tables whilst still WIPing

* writer on sqlite please

* Linting
2021-07-27 17:08:53 +01:00
kegsay a060df91e2
Use db writer on sqlite account table (#1944) 2021-07-27 12:47:50 +01:00
kegsay 32538640db
Key backups (1/2) : Add E2E session backup metadata tables (#1943)
* Initial key backup paths and userapi API

* Fix unit tests

* Add key backup table

* Glue REST API to database

* Linting

* use writer on sqlite
2021-07-27 12:47:32 +01:00
Neil Alexander f63068df3b
Only include go-sqlite3 on the relevant binaries (#1900)
* Only include go-sqlite3 on the relevant binaries

* The driver name is always sqlite3 now

* Update to matrix-org/go-sqlite3-js@e537baa
2021-07-20 11:18:14 +01:00
Bruce MacDonald d27607af78
Implement OpenID module (#599) (#1812)
* Implement OpenID module (#599)

- Unrelated: change Riot references to Element in client API routing

Signed-off-by: Bruce MacDonald <contact@bruce-macdonald.com>

* OpenID module tweaks (#599)

- specify expiry is ms rather than vague ts
- add OpenID token lifetime to configuration
- use Go naming conventions for the path params
- store plaintext token rather than hash
- remove openid table sqllite mutex

* Add default OpenID token lifetime (#599)

* Update dendrite-config.yaml

Co-authored-by: Kegsay <kegsay@gmail.com>
Co-authored-by: Kegsay <kegan@matrix.org>
2021-04-07 13:26:20 +01:00
Kegsay 850abb1dde
Make bcrypt cost configurable (#1793) 2021-03-08 13:19:02 +00:00
Will Hunt a2773922d2
Send events to appservice based on room membership (#1680)
* Check membership of room

* Use QueryStateAfterEventsResponse

* Fix complexity

* Changes that I made a long time ago

* Rename to appserviceJoinedAtEvent

* Check membership in GetMemberships

* Update QueryMembershipsForRoom

* Tweaks in client API

* Update appserviceJoinedAtEvent

* Comments

* Try QueryMembershipForUser instead

* Undo some changes to client API that shouldn't be needed

* More /event tweaks

* Refactor /event bit

* Go back to QueryMembershipsForRoom because appservices are hard

* Fix bugs in onMessage

* Add comments

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2021-03-03 16:27:44 +00:00
Neil Alexander f0139f12ca
Don't return error when account conflict is handled gracefully (#1782) 2021-03-03 14:01:34 +00:00
Neil Alexander 81312b8a78
Return the current OTK count on an empty upload request (#1774)
* Always return OTK counts

* Fix parameter ordering

* Send IDs over to keyserver internal API

* Review comments

* Fix syntax error

* Fix panic, hopefully

* Require user ID to be set

* Fix user API call
2021-03-02 11:40:20 +00:00
Neil Alexander f5cf241877
Fix user registration bug (#1777) 2021-03-02 10:43:25 +00:00
Jason Robinson 597350a67f
Ensure appservices can auth as users in their namespaces (#1672)
* Ensure appservices can auth as users in their namespaces

Currently in Dendrite appservices can only auth as a user if the user was created by said appservice. This does not align with the appservices spec which specifically says:

> The application service may specify the virtual user to act as through use of a user_id query string parameter on the request. The user specified in the query string must be covered by one of the application service’s user namespaces.

https://matrix.org/docs/spec/application_service/r0.1.2#identity-assertion

In the case that a user has been created for example via manual registration but belongs to an appservice namespace, the current functionality does not allow appservices to auth as them. This PR fixes that by replacing the appservice ID check with a check against the appservice namespace.

This also matches Synapse functionality, which I confirmed to allow appservices to auth as a user in their namespace, irregardless of how the user was registered.

* Also allow appservice itself to auth with user_id

Appservice user_id + access token check needs to work both when user_id is the appservice and when appservice has the user in their user namespace.

Signed-off-by: Jason Robinson <mail@jasonrobinson.me>
2021-01-04 10:43:23 +00:00
6543 a85051ca9d
fix imports (#1665)
* fix imports

Signed-off-by: 6543 <6543@obermui.de>

* add sqlite driver import back

Signed-off-by: 6543 <6543@obermui.de>

* rm import of userapi/storage/accounts/sqlite3/storage.go
2020-12-21 14:27:01 +00:00
Neil Alexander b5aa7ca3ab
Top-level setup package (#1605)
* Move config, setup, mscs into "setup" top-level folder

* oops, forgot the EDU server

* Add setup

* goimports
2020-12-02 17:41:00 +00:00
Neil Alexander c636be5070
Update last seen on sync requests (#1593)
* Update last seen on sync requests

* Fix MSC2836 unit tests

* Only update once per minute

* Remove debug logging

* Configurable option

* Simplify updateLastSeen/cleanLastSeen
2020-11-20 11:29:02 +00:00
David Spenler 35ea55e70b
Implemented whois endpoint (#1573)
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2020-11-17 10:07:03 +00:00
S7evinK 0974f6e2c0
Fix internal http api (#1535)
Signed-off-by: Till Faelligen <tfaelligen@gmail.com>
2020-10-19 09:38:10 +01:00
Kegsay 4a7fb9c045
Automatically upgrade databases on startup (#1529)
* Support auto-upgrading accounts DB

* Auto-upgrade device DB deltas

* Support up/downgrading from cmd/goose

* Linting

* Create tables then do migrations then prepare statements

To avoid failing due to some things not existing

* Linting
2020-10-15 18:09:41 +01:00
S7evinK 2bd0449c5b
Use default value when adding column which is not null (#1501)
Signed-off-by: Till Faelligen <tfaelligen@gmail.com>
2020-10-09 12:41:53 +01:00
S7evinK 1cd525ef0d
Extend device_devices table (#1471)
* Add last_used_ts and IP to database

* Add migrations

* Rename column
Prepare statements

* Add interface method and implement it

Signed-off-by: Till Faelligen <tfaelligen@gmail.com>

* Rename struct fields

* Add user_agent to database

* Add userAgent to registration calls

* Add missing "IF NOT EXISTS"

* use txn writer

* Add UserAgent to Device

Co-authored-by: Kegsay <kegan@matrix.org>
2020-10-09 09:17:23 +01:00
Loïck Bonniot 4e8c484618
Implement account deactivation (#1455)
* Implement account deactivation

See #610
Signed-off-by: Loïck Bonniot <git@lesterpig.com>

* Rename 'is_active' to 'is_deactivated'

Signed-off-by: Loïck Bonniot <git@lesterpig.com>

Co-authored-by: Kegsay <kegan@matrix.org>
2020-10-02 17:18:20 +01:00
Sam a6700331ce
Update all usages of tx.Stmt to sqlutil.TxStmt (#1423)
* Replace all usages of txn.Stmt with sqlutil.TxStmt

Signed-off-by: Sam Day <me@samcday.com>

* Fix sign off link in PR template.

Signed-off-by: Sam Day <me@samcday.com>

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2020-09-24 11:10:14 +01:00
Neil Alexander 5076925c18
Password changes (#1397)
* User API support for password changes

* Password changes in client API

* Update sytest-whitelist

* Remove debug logging

* Default logout_devices to true

* Fix deleting devices by local part
2020-09-04 15:16:13 +01:00
Neil Alexander 0ab5bccd11
Storage tweaks (#1373)
* Sync API tweaks

* User API tweaks
2020-09-01 11:28:35 +01:00
Kegsay 9af2f5f1f2
Remove device DB from clientapi (#1352)
* Remove device DB from clientapi

* Remove device DB from startup configuration

It's all an impl detail now in user API
2020-08-27 18:53:40 +01:00
Neil Alexander 9d53351dc2
Component-wide TransactionWriters (#1290)
* Offset updates take place using TransactionWriter

* Refactor TransactionWriter in current state server

* Refactor TransactionWriter in federation sender

* Refactor TransactionWriter in key server

* Refactor TransactionWriter in media API

* Refactor TransactionWriter in server key API

* Refactor TransactionWriter in sync API

* Refactor TransactionWriter in user API

* Fix deadlocking Sync API tests

* Un-deadlock device database

* Fix appservice API

* Rename TransactionWriters to Writers

* Move writers up a layer in sync API

* Document sqlutil.Writer interface

* Add note to Writer documentation
2020-08-21 10:42:08 +01:00
Neil Alexander b24747b305
Transaction writer changes, move roomserver writers (#1285)
* Updated TransactionWriters, moved locks in roomserver, various other tweaks

* Fix redaction deadlocks

* Fix lint issue

* Rename SQLiteTransactionWriter to ExclusiveTransactionWriter

* Fix us not sending transactions through in latest events updater
2020-08-19 15:38:27 +01:00
Kegsay 820c56c165
Fix more E2E sytests (#1265)
* WIP: Eagerly sync device lists on /user/keys/query requests

Also notify servers when a user's device display name changes. Few
caveats:
 - sytest `Device deletion propagates over federation` fails
 - `populateResponseWithDeviceKeysFromDatabase` is called from multiple
   goroutines and hence is unsafe.

* Handle deleted devices correctly over federation
2020-08-12 22:43:02 +01:00
Neil Alexander 4b09f445c9
Configuration format v1 (#1230)
* Initial pass at refactoring config (not finished)

* Don't forget current state and EDU servers

* More shifting around

* Update server key API tests

* Fix roomserver test

* Fix more tests

* Further tweaks

* Fix current state server test (sort of)

* Maybe fix appservices

* Fix client API test

* Include database connection string in database options

* Fix sync API build

* Update config test

* Fix unit tests

* Fix federation sender build

* Fix gobind build

* Set Listen address for all services in HTTP monolith mode

* Validate config, reinstate appservice derived in directory, tweaks

* Tweak federation API test

* Set MaxOpenConnections/MaxIdleConnections to previous values

* Update generate-config
2020-08-10 14:18:04 +01:00
Kegsay fdabba1851
bugfix: when a user's key changes, don't notify everyone on the server (#1253)
* bugfix: when a user's key changes, don't notify everyone on the server

Instead just notify the users you share a room with.

* Update whitelist
2020-08-10 12:38:33 +01:00
Kegsay b5cb1d1534
Fix edge cases around device lists (#1234)
* Fix New users appear in /keys/changes

* Create blank device keys when logging in on a new device

* Add PerformDeviceUpdate and fix a few bugs

- Correct device deletion query on sqlite
- Return no keys on /keys/query rather than an empty key

* Unbreak sqlite properly

* Use a real DB for currentstateserver integration tests

* Race fix
2020-07-31 14:40:45 +01:00
Kegsay a7e67e65a8
Notify clients when devices are deleted (#1233)
* Recheck device lists when join/leave events come in

* Add PerformDeviceDeletion

* Notify clients when devices are deleted

* Unbreak things

* Remove debug logging
2020-07-30 18:00:56 +01:00
Neil Alexander acc8e80a51
User directory (#1225)
* User directory

* Fix syncapi unit test

* Make user directory only show remote users you know about from your joined rooms

* Update sytest-whitelist

* Review comments
2020-07-28 10:53:17 +01:00
Kegsay 541a23f712
Handle inbound federation E2E key queries/claims (#1215)
* Handle inbound /keys/claim and /keys/query requests

* Add display names to device key responses

* Linting
2020-07-22 17:04:57 +01:00
Neil Alexander b6bc132485
Use TransactionWriter in other component SQLite (#1209)
* Use TransactionWriter on other component SQLites

* Fix sync API tests

* Fix panic in media API

* Fix a couple of transactions

* Fix wrong query, add some logging output

* Add debug logging into StoreEvent

* Adjust InsertRoomNID

* Update logging
2020-07-21 15:48:21 +01:00
Neil Alexander 84387e5249
Protect User API SQLite calls with mutexes (#1201)
* Protect SQLite calls with mutexes (replaces #1200)

* Remove locks on reads
2020-07-15 15:14:59 +01:00
Neil Alexander 8e7947926c Fix Yggdrasil gobind build, set display name at registration 2020-07-01 13:35:58 +01:00
Kegsay 6f49758b90
Remove membership table from account DB (#1172)
* Remove membership table from account DB

And make code which needs that data use the currentstate server

* Unbreak tests; use a membership enum for space
2020-06-30 13:34:59 +01:00
Kegsay 1ad7219e4b
Implement /sync `limited` and read timeline limit from stored filters (#1168)
* Move filter table to syncapi where it is used

* Implement /sync `limited` and read timeline limit from stored filters

We now fully handle `room.timeline.limit` filters (in-line + stored) and
return the right value for `limited` syncs.

* Update whitelist

* Default to the default timeline limit if it's unset, also strip the extra event correctly

* Update whitelist
2020-06-26 15:34:41 +01:00
Kegsay 46de400aa0
Hopefully fix databased is locked errors on sqlite account creation (#1162) 2020-06-24 20:46:28 +01:00
Neil Alexander 9e3d771a32 Fix comment in InputAccountDataRequest 2020-06-19 09:18:09 +01:00
Neil Alexander dc0bac85d5
Refactor account data (#1150)
* Refactor account data

* Tweak database fetching

* Tweaks

* Restore syncProducer notification

* Various tweaks, update tag behaviour

* Fix initial sync
2020-06-18 18:36:03 +01:00
Kegsay e09d24e732
Move account/device DBs to userapi (#1141) 2020-06-17 12:05:56 +01:00
Kegsay a66a3b830c
Make userapi control account creation entirely (#1139)
This makes a chokepoint with which we can finally fix
'database is locked' errors on sqlite during account creation
2020-06-17 11:22:26 +01:00
Kegsay e15a8042a1
BREAKING: Make eduserver/appservice use userapi (#1138)
* BREAKING: Make eduserver/appservice use userapi

This is a breaking change because this PR restructures how the AS API
tracks its position in Kafka streams. Previously, it used the account DB
to store partition offsets. However, this is also being used by `clientapi`
for the same purpose, which is bad (each component needs to store offsets
independently or else you might lose messages across restarts). This PR
changes this behaviour to now store partition offsets in the `appservice`
database.

This means that:
 - Upon restart, the `appservice` component will attempt to replay all
   room events from the beginning of time.
 - An additional table will be created in the appservice database, which
   in and of itself is backwards compatible.

* Return ErrorConflict
2020-06-16 17:39:56 +01:00
Kegsay 83391da0e0
Make syncapi use userapi (#1136)
* Make syncapi use userapi

* Unbreak things

* Fix tests

* Lint
2020-06-16 17:05:38 +01:00
Kegsay 1942928ee5
Make federationapi use userapi (#1135)
Removes dependencies on account DB, device DB and ASAPI.
2020-06-16 14:53:19 +01:00