Commit Graph

1506 Commits (6ae976a55a190e621835f638a137307fcff101a4)

Author SHA1 Message Date
Neil Alexander b5a8935042
Sync refactor — Part 1 (#1688)
* It's half-alive

* Wakeups largely working

* Other tweaks, typing works

* Fix bugs, add receipt stream

* Delete notifier, other tweaks

* Dedupe a bit, add a template for the invite stream

* Clean up, add templates for other streams

* Don't leak channels

* Bring forward some more PDU logic, clean up other places

* Add some more wakeups

* Use addRoomDeltaToResponse

* Log tweaks, typing fixed?

* Fix timed out syncs

* Don't reset next batch position on timeout

* Add account data stream/position

* End of day

* Fix complete sync for receipt, typing

* Streams package

* Clean up a bit

* Complete sync send-to-device

* Don't drop errors

* More lightweight notifications

* Fix typing positions

* Don't advance position on remove again unless needed

* Device list updates

* Advance account data position

* Use limit for incremental sync

* Limit fixes, amongst other things

* Remove some fmt.Println

* Tweaks

* Re-add notifier

* Fix invite position

* Fixes

* Notify account data without advancing PDU position in notifier

* Apply account data position

* Get initial position for account data

* Fix position update

* Fix complete sync positions

* Review comments @Kegsay

* Room consumer parameters
2021-01-08 16:59:06 +00:00
Jan Pechek 56a7839aed
Fixed links to Docker Hub (#1686) 2021-01-07 13:02:15 +00:00
Jason Robinson e3ce6a924f
Add MSC's section to example config (#1675)
The config section for MSC's was missing from the example config. Even though these are experimental features, it might be good to add the section into the example config for visibility.

Signed-off-by: Jason Robinson <mail@jasonrobinson.me>
2021-01-05 11:31:10 +00:00
Neil Alexander 05324b6861
Send/state tweaks (#1681)
* Check missing event count

* Don't use request context for /send
2021-01-04 13:47:48 +00:00
Jason Robinson 597350a67f
Ensure appservices can auth as users in their namespaces (#1672)
* Ensure appservices can auth as users in their namespaces

Currently in Dendrite appservices can only auth as a user if the user was created by said appservice. This does not align with the appservices spec which specifically says:

> The application service may specify the virtual user to act as through use of a user_id query string parameter on the request. The user specified in the query string must be covered by one of the application service’s user namespaces.

https://matrix.org/docs/spec/application_service/r0.1.2#identity-assertion

In the case that a user has been created for example via manual registration but belongs to an appservice namespace, the current functionality does not allow appservices to auth as them. This PR fixes that by replacing the appservice ID check with a check against the appservice namespace.

This also matches Synapse functionality, which I confirmed to allow appservices to auth as a user in their namespace, irregardless of how the user was registered.

* Also allow appservice itself to auth with user_id

Appservice user_id + access token check needs to work both when user_id is the appservice and when appservice has the user in their user namespace.

Signed-off-by: Jason Robinson <mail@jasonrobinson.me>
2021-01-04 10:43:23 +00:00
Will Hunt e7f2d770df
Update docker-compose.deps.yml to use postgres:11 (#1677) 2021-01-04 10:36:36 +00:00
6543 a85051ca9d
fix imports (#1665)
* fix imports

Signed-off-by: 6543 <6543@obermui.de>

* add sqlite driver import back

Signed-off-by: 6543 <6543@obermui.de>

* rm import of userapi/storage/accounts/sqlite3/storage.go
2020-12-21 14:27:01 +00:00
Neil Alexander 4a0461378a
Version 0.3.4 2020-12-18 14:53:31 +00:00
Neil Alexander 932ff6b166
Fix /register 2020-12-18 14:51:30 +00:00
Will Hunt e4d1f0958f
Do not check if a username is exclusive if the request is for an appservice in /register (#1652)
* Do not check if a username is exclusive if the request is for an appservice in /register

* remove useless comment

* Move statements

* fix broken test

* Also fix the senderLocalpart problem

* Check domain name is ours

* Handle accessTokenErr

* Return unauthorised instead of forbidden

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2020-12-18 14:36:59 +00:00
Neil Alexander 4fe4c180e6
Ensure we wake for our own device list updates (#1661)
* Make sure we wake up for our own key changes

* Whitelist 'Users receive device_list updates for their own devices'
2020-12-18 14:04:17 +00:00
Neil Alexander fac71edc62
Fix #1655 by re-adding the appservice alias query (#1660) 2020-12-18 13:33:28 +00:00
Neil Alexander 65ff5c9a2c
Don't send null in joined_rooms response (#1659) 2020-12-18 13:33:08 +00:00
Neil Alexander 50963b724b
More sane next batch handling, typing notification tweaks, give invites their own stream position, device list fix (#1641)
* Update sync responses

* Fix positions, add ApplyUpdates

* Fix MarshalText as non-pointer, PrevBatch is optional

* Increment by number of read receipts

* Merge branch 'master' into neilalexander/devicelist

* Tweak typing

* Include keyserver position tweak

* Fix typing next position in all cases

* Tweaks

* Fix typo

* Tweaks, restore StreamingToken.MarshalText which somehow went missing?

* Rely on positions from notifier rather than manually advancing them

* Revert "Rely on positions from notifier rather than manually advancing them"

This reverts commit 53112a62cc3bfd9989acab518e69eeb27938117a.

* Give invites their own position, fix other things

* Fix test

* Fix invites maybe

* Un-whitelist tests that look to be genuinely wrong

* Use real receipt positions

* Ensure send-to-device uses real positions too
2020-12-18 11:11:21 +00:00
Kegsay a518e2971a
Fix flakey complement (#1653) 2020-12-17 18:25:51 +00:00
Neil Alexander 9a1ef81fa2
Add CS sendevent metrics
Squashed commit of the following:

commit c38c39107b6dda0c8e6320d61da2365c47eea4e9
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Thu Dec 17 10:13:09 2020 +0000

    Update buckets

commit 5a3bcc8bd4167150374827b0e42a0dea0366beff
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Thu Dec 17 09:41:43 2020 +0000

    Update buckets

commit 78dff8165ddf596e28af04faf56466752ebc17af
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Thu Dec 17 09:37:27 2020 +0000

    Register sendEventDuration

commit 612b0f3d84f1547ff30131f7b084baf609edab52
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Thu Dec 17 09:35:59 2020 +0000

    sendevent metrics
2020-12-17 10:52:31 +00:00
Neil Alexander e1ace7e44a
Add event ID index on current state table (helps performance) (#1649) 2020-12-16 18:16:39 +00:00
Kegsay 76becac003
Add start_stream to /messages (#1648) 2020-12-16 18:10:39 +00:00
Kegsay 42e9cbf342
NOTSPEC: Make ?from= optional in /messages (#1647) 2020-12-16 17:31:03 +00:00
Neil Alexander 56b5847c74
Add prometheus metrics for destination queues, sync requests
Squashed commit of the following:

commit 7ed1c6cfe67429dbe378a763d832c150eb0f781d
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Wed Dec 16 14:53:27 2020 +0000

    Updates

commit 8442099d08760b8d086e6d58f9f30284e378a2cd
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Wed Dec 16 14:43:18 2020 +0000

    Add some sync statistics

commit ffe2a11644ed3d5297d1775a680886c574143fdb
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Wed Dec 16 14:37:00 2020 +0000

    Fix backing off display

commit 27443a93855aa60a49806ecabbf9b09f818301bd
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Wed Dec 16 14:28:43 2020 +0000

    Add some destination queue metrics
2020-12-16 15:02:39 +00:00
Neil Alexander b891c00b09
Add RoomInfo cache, remove RoomServerRoomNIDsCache (#1646)
* Add RoomInfo cache, remove RoomServerRoomNID cache, ensure caches are thread-safe

* Don't panic if the roomInfo isn't known yet

* LRU package is already threadsafe

* Use RoomInfo cache to find room version if possible in Events()

* Adding comments about RoomInfoCache safety
2020-12-16 12:15:12 +00:00
Neil Alexander 9057143033
Hit the database far less in Events to find room NIDs and room versions (#1643)
* Hit the database far less to find room NIDs for event NIDs

* Close the rows

* Fix SQLite selectRoomNIDsForEventNIDsSQL

* Give same treatment to room version lookups
2020-12-16 10:33:28 +00:00
Neil Alexander d2bcc5f746
Fix main.wasm output path (fixes #1645) 2020-12-16 09:33:48 +00:00
Neil Alexander 38318b0f16
De-map device list positions in streaming tokens (#1642)
* De-map device list positions in streaming tokens

* Fix lint error

* Tweak toOffset
2020-12-15 15:09:10 +00:00
Andrew Morgan 98ebbd01e5
Remove incorrect comment in find-lint.sh (#1619)
The referenced issue is about `go build`, not `go get`.
2020-12-15 09:45:36 +00:00
Shubham Upreti 502051421c
Added govet to golanci-lint linters (#1639) 2020-12-15 09:44:37 +00:00
Neil Alexander 2885eb0422
Don't use request context for input room event queued tasks (#1640) 2020-12-14 14:40:57 +00:00
Neil Alexander f5869daaab
Don't start more goroutines than needed on RS input, increase input worker buffer size (#1638) 2020-12-14 10:42:21 +00:00
Neil Alexander d7824ed5b1
Update INSTALL.md (#1637) 2020-12-14 09:42:27 +00:00
Neil Alexander b794873fa0
Update INSTALL.md 2020-12-12 20:55:34 +00:00
Neil Alexander ebcacd1bb5
Give receipts their own stream ID in the database (#1631)
* Give read recipts their own database sequence

* Give receipts their own stream ID

* Change migration names

* Reset sequences

* Add max receipt queries, missing stream_id table entry for SQLite
2020-12-11 14:02:17 +00:00
Neil Alexander c55361c1b8
Fix max_open_conns in Dendrite sample config 2020-12-11 10:48:24 +00:00
Neil Alexander 9c03b0a4fa
Refactor sync tokens (#1628)
* Refactor sync tokens

* Comment out broken notifier test

* Update types, sytest-whitelist

* More robust token checking

* Remove New functions for streaming tokens

* Export Logs in StreamingToken

* Fix tests
2020-12-10 18:57:10 +00:00
Neil Alexander bad81c028f
Don't recalculate event ID so often in sync (#1624)
* Don't bail so quickly in fetchMissingStateEvents

* Don't recalculate event IDs so often in sync API

* Add comments

* Fix comments

* Update to matrix-org/gomatrixserverlib@eb6a890
2020-12-09 18:07:17 +00:00
Neil Alexander 851c02659a
Version 0.3.3 2020-12-09 14:13:02 +00:00
Neil Alexander d9b3035342
Adjust latest events updater (#1623)
* Adjust forward elatest events updater

* Populate newLatest in all cases

* Re-add existingPrevs loop
2020-12-09 13:34:37 +00:00
Neil Alexander f64c8822bc
Federation sender refactor (#1621)
* Refactor federation sender, again

* Clean up better

* Missing operators

* Try to get overflowed events from database

* Fix queries

* Log less

* Comments

* nil PDUs/EDUs shouldn't happen but guard against them for safety

* Tweak logging

* Fix transaction coalescing

* Update comments

* Check nils more

* Remove channels as they add extra complexity and possibly will deadlock

* Don't hold lock while sending transaction

* Less spam about sleeping queues

* Comments

* Bug-fixing

* Don't try to rehydrate twice

* Don't queue in memory for blacklisted destinations

* Don't queue in memory for blacklisted destinations

* Fix a couple of bugs

* Check for duplicates when pulling things out of the database

* Durable transactions, some more refactoring

* Revert "Durable transactions, some more refactoring"

This reverts commit 5daf924eaaefec5e4f7c12c16ca24e898de4adbb.

* Fix deadlock
2020-12-09 10:03:22 +00:00
Neil Alexander e1d32e2ff1
Update to matrix-org/gomatrixserverlib@232bace 2020-12-07 16:21:55 +00:00
bodqhrohro 45dec20593
Update links to golangci-lint documentation (#1617)
Signed-off-by: Bohdan Horbeshko <bodqhrohro@gmail.com>
2020-12-07 11:23:01 +00:00
Neil Alexander 8f5393db5f
Update FAQ 2020-12-07 10:20:19 +00:00
Neil Alexander 5d65a879a5
Federation sender event cache (#1614)
* Cache federation sender events

* Store in the correct cache

* Update federation event cache

* Fix Unset

* Give EDUs same caching treatment as PDUs

* Make federationsender_cache_size configurable

* Default caches configuration

* Fix unit tests

* Revert "Fix unit tests"

This reverts commit 24eb5d22524f20e1024b1475debe61ae20538a5a.

* Revert "Default caches configuration"

This reverts commit 464ecd1e64b9d2983f6fd5430e9607519d543cb3.

* Revert "Make federationsender_cache_size configurable"

This reverts commit 4631f5324151e006a15d6f19008f06361b994607.
2020-12-04 14:52:10 +00:00
Kegsay b507312d4c
MSC2836 threading: part 2 (#1596)
* Update GMSL

* Add MSC2836EventRelationships to fedsender

* Call MSC2836EventRelationships in reqCtx

* auth remote servers

* Extract room ID and servers from previous events; refactor a bit

* initial cut of federated threading

* Use the right client/fed struct in the response

* Add QueryAuthChain for use with MSC2836

* Add auth chain to federated response

* Fix pointers

* under CI: more logging and enable mscs, nil fix

* Handle direction: up

* Actually send message events to the roomserver..

* Add children and children_hash to unsigned, with tests

* Add logic for exploring threads and tracking children; missing storage functions

* Implement storage functions for children

* Add fetchUnknownEvent

* Do federated hits for include_children if we have unexplored children

* Use /ev_rel rather than /event as the former includes child metadata

* Remove cross-room threading impl

* Enable MSC2836 in the p2p demo

* Namespace mscs db

* Enable msc2836 for ygg

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2020-12-04 14:11:01 +00:00
TR_SLimey c052edafdd
Fix broken image in CONTRIBUTING.md (#1612) 2020-12-04 14:10:01 +00:00
Neil Alexander 68d92fd3b4
Update sytest-whitelist/blacklist 2020-12-04 14:09:21 +00:00
Ronnie Ebrin a677a288bd
federationsender/roomserver: don't panic while federation is disabled (#1615) 2020-12-04 14:08:17 +00:00
Neil Alexander 1ce9c52442
Don't recalculate event IDs so often (#1610)
* Don't recalculate event IDs so often

* Revert invite change

* Make sure we're using the right NIDs

* Update gomatrixserverlib

* Update to NewEventFromTrustedJSONWithEventID

* Fix go.mod

* Update gomatrixserverlib to matrix-org/gomatrixserverlib#243

* Use BulkSelectEventID
2020-12-04 10:41:07 +00:00
Petter Jakub Økland 19b1d40d64
Hiawatha reverse proxy sample for monolith and fixes to polylith sample (#1609)
* Created polylith reverse proxy sample for Hiawatha

* Create monolith-sample.conf

* Added timeout to proxies and binding explanation.

* Fixed typo.

* Consistency with regards to polylith-sample.conf.
2020-12-03 13:52:00 +00:00
Neil Alexander 52905ffb82
Update PROFILING.md 2020-12-03 13:49:34 +00:00
Neil Alexander 253b05ccde
Update PROFILING.md 2020-12-03 13:48:02 +00:00
Neil Alexander 246866a131
Add PROFILING.md 2020-12-03 13:46:56 +00:00