Commit Graph

1502 Commits (30e9353fdd4cc2a3c3e5bd6de139ca0e31dae7bb)

Author SHA1 Message Date
Neil Alexander d56a5a6509
Fix well-known lookups 2021-01-28 13:25:45 +00:00
Neil Alexander e14e9a1606
Version 0.3.7 2021-01-26 13:13:48 +00:00
Neil Alexander 9f443317bc
Graceful shutdowns (#1734)
* Initial graceful stop

* Fix dendritejs

* Use process context for outbound federation requests in destination queues

* Reduce logging

* Fix log level
2021-01-26 12:56:20 +00:00
Neil Alexander 64fb6de6d4
Don't retrieve same state events over and over again (#1737) 2021-01-26 09:12:17 +00:00
Kegsay ef9d5ad4fe
Check peek state response and refactor checking send_join response (#1732) 2021-01-22 17:16:35 +00:00
Neil Alexander 6757b67a32
NewClient and NewFederationClient updates (#1730)
* Use matrix-org/gomatrixserverlib#252

* Add missing WithSkipVerify to test

* Functions instead

* Update gomatrixserverlib to matrix-org/gomatrixserverlib#252

* Fix disabling TLS validation
2021-01-22 16:09:05 +00:00
Kegsay 93942f8ab6
Gate peeking behind msc flags (#1731) 2021-01-22 16:08:47 +00:00
Matthew Hodgson 0571d395b5
Peeking over federation via MSC2444 (#1391)
* a very very WIP first cut of peeking via MSC2753.

doesn't yet compile or work.
needs to actually add the peeking block into the sync response.
checking in now before it gets any bigger, and to gather any initial feedback on the vague shape of it.

* make PeekingDeviceSet private

* add server_name param

* blind stab at adding a `peek` section to /sync

* make it build

* make it launch

* add peeking to getResponseWithPDUsForCompleteSync

* cancel any peeks when we join a room

* spell out how to runoutside of docker if you want speed

* fix SQL

* remove unnecessary txn for SelectPeeks

* fix s/join/peek/ cargocult fail

* HACK: Track goroutine IDs to determine when we write by the wrong thread

To use: set `DENDRITE_TRACE_SQL=1` then grep for `unsafe`

* Track partition offsets and only log unsafe for non-selects

* Put redactions in the writer goroutine

* Update filters on writer goroutine

* wrap peek storage in goid hack

* use exclusive writer, and MarkPeeksAsOld more efficiently

* don't log ascii in binary at sql trace...

* strip out empty roomd deltas

* re-add txn to SelectPeeks

* re-add accidentally deleted field

* reject peeks for non-worldreadable rooms

* move perform_peek

* fix package

* correctly refactor perform_peek

* WIP of implementing MSC2444

* typo

* Revert "Merge branch 'kegan/HACK-goid-sqlite-db-is-locked' into matthew/peeking"

This reverts commit 3cebd8dbfbccdf82b7930b7b6eda92095ca6ef41, reversing
changes made to ed4b3a58a7855acc43530693cc855b439edf9c7c.

* (almost) make it build

* clean up bad merge

* support SendEventWithState with optional event

* fix build & lint

* fix build & lint

* reinstate federated peeks in the roomserver (doh)

* fix sql thinko

* todo for authenticating state returned by /peek

* support returning current state from QueryStateAndAuthChain

* handle SS /peek

* reimplement SS /peek to prod the RS to tell the FS about the peek

* rename RemotePeeks as OutboundPeeks

* rename remote_peeks_table as outbound_peeks_table

* add perform_handle_remote_peek.go

* flesh out federation doc

* add inbound peeks table and hook it up

* rename ambiguous RemotePeek as InboundPeek

* rename FSAPI's PerformPeek as PerformOutboundPeek

* setup inbound peeks db correctly

* fix api.SendEventWithState with no event

* track latestevent on /peek

* go fmt

* document the peek send stream race better

* fix SendEventWithRewrite not to bail if handed a non-state event

* add fixme

* switch SS /peek to use SendEventWithRewrite

* fix comment

* use reverse topo ordering to find latest extrem

* support postgres for federated peeking

* go fmt

* back out bogus go.mod change

* Fix performOutboundPeekUsingServer

* Fix getAuthChain -> GetAuthChain

* Fix build issues

* Fix build again

* Fix getAuthChain -> GetAuthChain

* Don't repeat outbound peeks for the same room ID to the same servers

* Fix lint

* Don't omitempty to appease sytest

Co-authored-by: Kegan Dougal <kegan@matrix.org>
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2021-01-22 14:55:08 +00:00
Neil Alexander 5d8ec0ff1a
Fix build error 2021-01-22 14:18:36 +00:00
Neil Alexander 805a74892e
DNS caching (#1728)
* Allow configuring DNS cache

* Update sample configs

* Fix build errors

* Fix time resolution

* Default 5m

* In seconds

* Use WithDNScache

* Correct field name

* Update go.mod/go.sum to matrix-org/gomatrixserverlib#251
2021-01-22 14:16:59 +00:00
Neil Alexander 745ee20b90
Update sytest-whitelist after matrix-org/sytest#991 2021-01-22 12:56:20 +00:00
Neil Alexander 58f78e08af
Limit well-known response sizes (matrix-org/gomatrixserverlib#250) 2021-01-21 13:57:41 +00:00
Neil Alexander 8fe51019ad
Add sync API memberships table (#1726) 2021-01-20 20:43:20 +00:00
Kegsay c08e38df2c
MSC2946: Treat federation responses the same way as local responses (#1724)
* Start treating fed rooms/events the same as local rooms/events

* Share more code
2021-01-20 17:03:35 +00:00
Neil Alexander b70238f2d5
Basic sync filtering (#1721)
* Add some filtering (postgres only for now)

* Fix build error

* Try to use request filter

* Use default filter as a template when retrieving from the database

* Remove unused strut

* Update sytest-whitelist

* Add filtering to SelectEarlyEvents

* Fix Postgres selectEarlyEvents query

* Attempt filtering on SQLite

* Test limit, set field for limit/order in prepareWithFilters

* Remove debug logging, add comments

* Tweaks, debug logging

* Separate SQLite stream IDs

* Fix filtering in current state table

* Fix lock issues

* More tweaks

* Current state requires room ID

* Review comments
2021-01-19 18:00:42 +00:00
Kegsay 80aa9aa8b0
Implement MSC2946 over federation (#1722)
* Add fedsender dep on msc2946

* Add MSC2946Spaces to fsAPI

* Add exclude_rooms impl

* Implement fed spaces handler

* Use stripped state not room version

* Call federated spaces at the right time
2021-01-19 17:14:25 +00:00
Neil Alexander ccfcb2d280
Create bin dir 2021-01-18 15:09:54 +00:00
Neil Alexander f6151b6290
Use dendritegithub for Docker Hub images 2021-01-18 14:00:47 +00:00
Neil Alexander 67c89b3c5e
Version 0.3.6 2021-01-18 13:37:49 +00:00
Neil Alexander 244ff0dccb
Don't create so many state snapshots when updating forward extremities (#1718)
* Light-weight checking of state changes when updating forward extremities

* Only do this for non-state events, since state events will always result in state change at extremities
2021-01-18 13:21:33 +00:00
sumitks866 dd1e31bee7
Mention unstable features in /_matrix/client/versions (#1710)
* Mention unstable features in /_matrix/client/versions

* list enabled mscs

* Don't update go.mod/go.sum

* goimports

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2021-01-18 13:09:28 +00:00
Neil Alexander 534c29ab02
Log event ID on consumer errors (fixes #1714) 2021-01-18 12:58:48 +00:00
Caleb Xavier Berger efc91146f0
Use go build instead of go install (#1716)
* Use go build instead of go install

go install doesn't like to cross-compile things. (Try running build.sh with GOARCH set to something other than what it "should" be.)

With go build, it appears that cross-compilation is really, really straightforward. Simply install a compiler for your target platform and set `GOARCH` and `CC` accordingly.

* Use shell expansion instead of loop

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2021-01-18 12:44:44 +00:00
Loïck Bonniot 940577cd3c
Fix integer overflow in device_list_update.go (#1717)
Fix #1511
On 32-bits systems, int(hash.Sum32()) can be negative.
This makes the computation of array indices using modulo invalid, crashing dendrite.
Signed-off-by: Loïck Bonniot <git@lesterpig.com>
2021-01-18 12:43:15 +00:00
Neil Alexander cf82e08096
Update GHA Docker Hub builds
Squashed commit of the following:

commit 4a61aa711473deece2adf415cfd65501dbca63b2
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Mon Jan 18 12:19:24 2021 +0000

    Set back to matrixdotorg on published releases

commit 6d1ac53f2c0c9b30e1e70c0bb1559e1b8ec874a2
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Mon Jan 18 11:55:28 2021 +0000

    Rename

commit 258999f7fb7b655b3a02a06a7ea05e66fb7740fb
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Mon Jan 18 11:52:26 2021 +0000

    Refactor multi-stage builds

commit c7ab8e476939899571e7b5668860dec372b9b60f
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Mon Jan 18 11:13:19 2021 +0000

    Let's try this again
2021-01-18 12:24:23 +00:00
Neil Alexander df4386f764
Build docker branch only 2021-01-18 11:03:50 +00:00
Neil Alexander 833e01a612
Update docker-build-and-push.yml 2021-01-18 10:26:36 +00:00
TR_SLimey ff94490384
GitHub action for automatic multiarch Docker build (#1613)
* Create docker-build-and-push.yml

* Switched to using official Docker buildx action

* Added comment to docker-build-and-push.yml
In case something needs to be tweaked in the future, the link
contains some examples and explanations which would be useful

* Run only on release (and produce release tags)

As this workflow takes quite a lot of time, and [pushing to master happens frequently](https://github.com/matrix-org/dendrite/pull/1613#issuecomment-746086980), the container will now only be built when a release is created, and the builds will also be correctly tagged.

* Add latest tag, test at neilalexander/dendrite*

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2021-01-18 10:23:53 +00:00
Neil Alexander 6dadb1c06b
Fix lint error in generate-keys 2021-01-18 09:34:49 +00:00
Kegan Dougal fdd534f86a Escape the create event content key so it can be extracted correctly 2021-01-15 18:49:21 +00:00
Kegan Dougal 2113da5332 Enable MSC 2946 in CI 2021-01-15 17:56:50 +00:00
Kegsay 5366c33885
MSC2946: Use new MSC1772 format for space parents (#1713) 2021-01-15 12:30:41 +00:00
Kegsay 2626525c65
MSC2946: Allow redactions/updates for space state events (#1712) 2021-01-15 11:44:36 +00:00
rany ef24ea2678
Add madvdontneed=1 to decrease RAM consumption for systemd service (#1690)
* Add madvdontneed=1 to decrease RAM consumption for systemd service

* Update monolith-example.service

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2021-01-14 10:31:45 +00:00
Kegan Dougal e30c523089 Per request cache for now as we don't do batching correclty 2021-01-13 23:00:25 +00:00
Kegan Dougal 19392c8c6b Decode uri values correctly 2021-01-13 22:27:04 +00:00
Kegsay 3183f75aed
MSC2946: Spaces Summary (#1700)
* Add stub functions for MSC2946

* Implement core space walking algorithm

* Flesh out stub functions; add test stubs

* Implement storage bits and add sanity check test

* Implement world_readable auth with test

* Linting
2021-01-13 18:00:38 +00:00
Neil Alexander 3ac693c7a5
Add dendrite_roomserver_processroomevent_duration_millis to prometheus
Squashed commit of the following:

commit e5e2d793119733ecbcf9b85f966e018ab0318741
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Wed Jan 13 17:28:12 2021 +0000

    Add dendrite_roomserver_processroomevent_duration_millis to prometheus
2021-01-13 17:31:46 +00:00
Neil Alexander 266f9c4abd
Update to matrix-org/gomatrixserverlib#247 - cache server well-known/DNS addresses 2021-01-13 17:31:13 +00:00
Neil Alexander d8fba52e97
Simplify send-to-device messaging (#1702)
* Simplify send-to-device messaging

* Don't return error if there's no work to do

* Remove SQLite migrations for now

* Tweak Postgres migrations

* Tweaks

* Fixes

* Cleanup separately

* Fix SQLite migration
2021-01-13 17:29:46 +00:00
Neil Alexander bb9e6a1281
Sync fixes (#1709)
* omitempty some fields in sync

* Add a few more

* Don't send push rules over and over again in incremental sync

* Further tweaks
2021-01-13 14:32:49 +00:00
Neil Alexander 55cfe391f7
Update /messages pagination token behaviour (#1708)
* Tweak pagination tokens

* start should be the specified from

* Don't reverse start and end

* Tweak getStartEnd again

* Update sytest-whitelist

* NOTSPEC: Re-add iOS end of topology
2021-01-13 12:59:29 +00:00
Neil Alexander fa65c40bae
Reduce device list GetUserDevices timeout (#1704) 2021-01-12 16:13:21 +00:00
Neil Alexander ec1b017906
Version 0.3.5 2021-01-11 13:55:40 +00:00
Neil Alexander e1e34b8994
Deep-checking forward extremities (#1698) 2021-01-11 12:47:25 +00:00
Neil Alexander 0fbebecd41
Tweak ApplyUpdates (#1691) 2021-01-09 11:25:09 +00:00
Neil Alexander b5a8935042
Sync refactor — Part 1 (#1688)
* It's half-alive

* Wakeups largely working

* Other tweaks, typing works

* Fix bugs, add receipt stream

* Delete notifier, other tweaks

* Dedupe a bit, add a template for the invite stream

* Clean up, add templates for other streams

* Don't leak channels

* Bring forward some more PDU logic, clean up other places

* Add some more wakeups

* Use addRoomDeltaToResponse

* Log tweaks, typing fixed?

* Fix timed out syncs

* Don't reset next batch position on timeout

* Add account data stream/position

* End of day

* Fix complete sync for receipt, typing

* Streams package

* Clean up a bit

* Complete sync send-to-device

* Don't drop errors

* More lightweight notifications

* Fix typing positions

* Don't advance position on remove again unless needed

* Device list updates

* Advance account data position

* Use limit for incremental sync

* Limit fixes, amongst other things

* Remove some fmt.Println

* Tweaks

* Re-add notifier

* Fix invite position

* Fixes

* Notify account data without advancing PDU position in notifier

* Apply account data position

* Get initial position for account data

* Fix position update

* Fix complete sync positions

* Review comments @Kegsay

* Room consumer parameters
2021-01-08 16:59:06 +00:00
Jan Pechek 56a7839aed
Fixed links to Docker Hub (#1686) 2021-01-07 13:02:15 +00:00
Jason Robinson e3ce6a924f
Add MSC's section to example config (#1675)
The config section for MSC's was missing from the example config. Even though these are experimental features, it might be good to add the section into the example config for visibility.

Signed-off-by: Jason Robinson <mail@jasonrobinson.me>
2021-01-05 11:31:10 +00:00
Neil Alexander 05324b6861
Send/state tweaks (#1681)
* Check missing event count

* Don't use request context for /send
2021-01-04 13:47:48 +00:00