Commit Graph

921 Commits (c1d2382e6d7f459ddf911a16aac7d4e63d50838b)

Author SHA1 Message Date
Kegsay 6d832ae544
Implement backfill in the roomserver (#983)
* Initial cut for backfilling

The syncserver now asks the roomserver via QueryBackfill (which already
existed to *handle* backfill requests) which then makes federation requests
via gomatrixserverlib.RequestBackfill.

Currently, tests fail on subsequent /messages requests because we don't know
which servers are in the room, because we are unable to get state snapshots
from a backfilled event because that code doesn't exist yet.

* WIP backfill, doesn't work

* Make initial backfill pass checks

* Persist backfilled events with state snapshots

* Remove debug lines

* Linting

* Review comments
2020-04-28 11:46:47 +01:00
Neil Alexander 3a858afca2
Loopback event from invite response (#982)
* Working invite v2 support

* Fix copyright notice

* Update gomatrixserverlib

* Add fallthrough

* Add missing continue

* Update sytest-whitelist, gomatrixserverlib

* Update gomatrixserverlib to test matrix-org/gomatrixserverlib#181

* Update gomatrixserverlib
2020-04-28 10:53:07 +01:00
Neil Alexander 87f05721b0 Update gomatrixserverlib 2020-04-27 15:47:36 +01:00
Neil Alexander 3ab8ebf6b8
More invite support (#979)
* Update gomatixserverlib

* Try to build invite stripped state if not given to us

* SendInvite improvements

* Transpose invite_room_state into invite_state.events for sync API

* Remove syncapi debugging output

* Use RespInviteV2

* Update gomatrixserverlib

* Send the invite event as a normal roomserver event too, for incorporating into room (should this be done by the roomserver automatically for invite inputs?)

* Federation sender use invite_room_state, room server try to insert membership state

* Check supported room versions on the invite endpoint

* Prevent roomserver query API from trying to handle requests for stub rooms

* Adding a nolint

* Replace IsRoomStub with RoomNIDExcludingStubs, fix query API to use that instead

* Review comments
2020-04-24 16:30:25 +01:00
Kegan Dougal be558f02aa Add new sytests to list 2020-04-24 10:39:11 +01:00
Kegsay a202d88fe5
Use a single storage.Database interface (#978) 2020-04-24 10:38:58 +01:00
Neil Alexander c30b12b5a1
Fix sarama import URLs (#856)
* Fix sarama import URLs

* Update gomatrixserverlib

* Update naffka

* Update naffka

* Update in kafka-producer
2020-04-22 15:26:56 +01:00
Behouba Manassé 423c656146
Response from /send_join now use gomatrixserverlib.RespSendJoin (#796)
* response from /send_join now use gomatrixserverlib.RespSendJoin

* Update Dendrite gomatrixserverlib version

* Fix spelling

Co-authored-by: Andrew Morgan <andrew@amorgan.xyz>
Co-authored-by: Cnly <minecnly@gmail.com>
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2020-04-22 14:13:11 +01:00
Neil Alexander a466e9e9cc
LRU cache for room versions in RS query API (#976)
* Experimental LRU cache for room versions

* Don't accidentally try to type-assert nil

* Also reduce hits on query API

* Use hashicorp implementation which mutexes for us

* Define const for max cache entries

* Rename to be specifically immutable, panic if we try to mutate a cache entry

* Review comments

* Remove nil guards, give roomserver integration test a cache

* go mod tidy
2020-04-22 13:00:05 +01:00
Neil Alexander 71f9d35b7c Update gomatrixserverlib 2020-04-21 10:04:06 +01:00
Neil Alexander 9045b8e89f
Perspective key fetching, some federation room join fixes (#975)
* Update gomatrixserverlib

* Test matrix.org as perspective key server

* Base64 decode better

* Optional strict validity checking in gmsl

* Update gomatrixserverlib

* Attempt to find missing auth events over federation (this shouldn't happen but I am guessing there is a synapse bug involved where we don't get all of the auth events)

* Update gomatrixserverlib, debug logging

* Remove debugging output

* More verbose debugging

* Print outliers

* Increase timeouts for testing, observe contexts before trying to join over more servers

* Don't block on roomserver (experimental)

* Don't block on roomserver

* Update gomatrixserverlib

* Update gomatrixserverlib

* Configurable perspective key fetchers

* Output number of configured keys for perspective

* Example perspective config included

* Undo debug stack trace

* Undo debug stack trace

* Restore original HTTP listener in monolith

* Fix lint

* Review comments

* Set default HTTP server timeout to 5 minutes now, block again when joining

* Don't use HTTP address for HTTPS whoops

* Update gomatrixserverlib

* Update gomatrixserverlib

* Update gomatrixserverlib

* Actually add perspectives

* Actually add perspectives

* Update gomatrixserverlib
2020-04-20 17:42:34 +01:00
Kegsay c62844234b
Add login sytests (#974) 2020-04-20 14:19:25 +01:00
Alex Chen 9f5ece8045
Fix nil http clients provided to components in polylith mode (#973)
Signed-off-by: Alex Chen <minecnly@gmail.com>
2020-04-19 11:18:38 +01:00
Alex Chen e1a4ada0e1
Ensure room version tests can be grouped whether failed or passed (#972)
Signed-off-by: Alex Chen <minecnly@gmail.com>
2020-04-18 11:01:48 +01:00
Kegsay 0d116d9222
Allow anyone to publish rooms (#971)
* Allow anyone to publish rooms

* Formatting and unused var

* Review comments
2020-04-17 12:17:20 +01:00
Neil Alexander 3c2e6f967b
Federation fixes and error handling (#970)
* Improve error handling in federation /send endpoint a bit

* Remove unknownRoomError, use unmarshalError when unable to get room ID

* Swap out a couple more internal server errors

* Update gomatrixserverlib

* Update gomatrixserverlib

* Update gomatrixserverlib

* Update gomatrixserverlib

* Update gomatrixserverlib

* Update gomatrixserverlib

* Return bad limit in error

* Same with domain/userid
2020-04-16 17:59:55 +01:00
Neil Alexander 3110a81996
Set default room version to 4 (#957)
* Set default room version to 4

* Default to v1 when room_version key missing

Signed-off-by: Alex Chen <minecnly@gmail.com>

* Squashed commit of the following:

commit c1bca95adb
Author: Kegsay <kegan@matrix.org>
Date:   Thu Apr 16 10:06:55 2020 +0100

    Add SQL tracing via DENDRITE_TRACE_SQL (#968)

    * Add SQL tracing via DENDRITE_TRACE_SQL

    Add this to `internal/sqlutil` in preparation for #897

    * Not entirely

commit c2ea961909
Author: Kegsay <kegan@matrix.org>
Date:   Wed Apr 15 17:48:40 2020 +0100

    Add HTTP trace logging (#965)

    * Dump all requests/response server-side

    * Wrap in DENDRITE_TRACE

    * DENDRITE_TRACE_HTTP is better

    * Bugfix for response body and linting

    * False is true and true is false

    * Linting

    * How did this get missed

    * More linting

* Update gomatrixserverlib

* Remove unneeded imports

Co-authored-by: Cnly <minecnly@gmail.com>
2020-04-16 12:53:27 +01:00
Kegsay c1bca95adb
Add SQL tracing via DENDRITE_TRACE_SQL (#968)
* Add SQL tracing via DENDRITE_TRACE_SQL

Add this to `internal/sqlutil` in preparation for #897

* Not entirely
2020-04-16 10:06:55 +01:00
Kegsay c2ea961909
Add HTTP trace logging (#965)
* Dump all requests/response server-side

* Wrap in DENDRITE_TRACE

* DENDRITE_TRACE_HTTP is better

* Bugfix for response body and linting

* False is true and true is false

* Linting

* How did this get missed

* More linting
2020-04-15 17:48:40 +01:00
Neil Alexander dadb06f6ad
Use topological ordering for /messages response (#966)
* Use topological ordering for /messages response

* Update gomatrixserverlib
2020-04-15 16:10:18 +01:00
Neil Alexander 7b3edf4622
Tweaks to backfill (#964) 2020-04-15 14:39:27 +01:00
Neil Alexander 895a72b6ee
Move /room/{roomID}/state endpoints into client API (#606) (#962)
* Move /room/{roomID}/state endpoints into client API (#606)

* Update sytest-whitelist

* Blacklist tests which rely on endpoints we don't implement
2020-04-14 18:36:08 +01:00
Hilmar Gústafsson 73d2f59e30
WIP: Add libp2p-go (#956)
* Add libp2p-go

* Some tweaks, tidying up

(cherry picked from commit 1a5bb121f8121c4f68a27abbf25a9a35a1b7c63e)

* Move p2p dockerfile

(cherry picked from commit 8d3bf44ea1bf37f950034e73bcdc315afdabe79a)

* Remove containsBackwardsExtremity

* Fix some linter errors, update some libp2p packages/calls, other tidying up

* Add -port for dendrite-p2p-demo

* Use instance name as key ID

* Remove P2P demo docker stuff, no longer needed now that we have SQLite

* Remove Dockerfile-p2p too

* Remove p2p logic from dendrite-monolith-server

* Inject publicRoomsDB in publicroomsapi

Inject publicRoomsDB instead of switching on base.libP2P.
See: https://github.com/matrix-org/dendrite/pull/956/files?file-filters%5B%5D=.go#r406276914

* Fix lint warning

* Extract mDNSListener from base.go

* Extract CreateFederationClient into demo

* Create P2PDendrite from BaseDendrite

Extract logic specific to P2PDendrite from base.go

* Set base.go to upstream/master

* Move pubsub to demo cmd

* Move PostgreswithDHT to cmd

* Remove unstable features

* Add copyrights

* Move libp2pvalidator into p2pdendrite

* Rename dendrite-p2p-demo -> dendrite-demo-libp2p

* Update copyrights

* go mod tidy

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2020-04-14 16:15:59 +01:00
Kegsay 48303d06cb
Are we synapse yet? (#960)
* Add a script which tracks synapse parity based on sytests

Processes results.tap to give you a percentage check.

* Pretty

* argparse and -v output
2020-04-14 16:05:15 +01:00
Kegsay 609f034bfb
S7evinK: basicauth metrics (#961)
* Add setting to enable/disable metrics (#461)
Add basic auth to /metric handlers

Signed-off-by: Till Faelligen <tfaelligen@gmail.com>

* Add warning message if metrics are exposed without protection

* Remove redundant type conversion

Signed-off-by: Till Faelligen <tfaelligen@gmail.com>

* SetBasicAuth per test case

* Update warning message and change loglevel to warn

* Update common/config/config.go

* Update dendrite-config.yaml

Co-authored-by: Till Faelligen <tfaelligen@gmail.com>
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2020-04-14 15:54:35 +01:00
mohit kumar singh 2c43e222bd
support non-exclusive namespaces for AS (#828)
Signed-off-by: MohitKS5 <mohitkumarsingh907@gmail.com>

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
Co-authored-by: Kegsay <kegan@matrix.org>
2020-04-14 15:31:27 +01:00
Prateek Sachan 1321f8da80
Check if user has the power level to edit the room visibility (#900)
* Check if user has the power level to edit the room visibility

* fix review changes

* Add nil check for queryEventsRes.StateEvents

Co-authored-by: Alex Chen <Cnly@users.noreply.github.com>
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2020-04-14 14:07:03 +01:00
Alex Chen d359851708
Update guides for running sytest (#958)
* Update guides for running sytest

Bring the details for running sytest against postgres up to date; prefer docker over manual setup

Signed-off-by: Alex Chen <minecnly@gmail.com>

* Better flags

Signed-off-by: Alex Chen <minecnly@gmail.com>
2020-04-12 13:48:24 +01:00
Prateek Sachan 317658acea
Added checks for JSON body in accounts_data endpoint (#863)
Signed-off-by: Prateek Sachan <psachan@cs.iitr.ac.in>
2020-04-12 00:47:05 +08:00
Neil Alexander dacee648f7
Federation for v3/v4 rooms (#954)
* Update gomatrixserverlib

* Default to room version 4

* Update gomatrixserverlib

* Limit prev_events and auth_events

* Fix auth_events, prev_events

* Fix linter issues

* Update gomatrixserverlib

* Fix getState

* Update sytest-whitelist

* Squashed commit of the following:

commit 067b875063
Author: Neil Alexander <neilalexander@users.noreply.github.com>
Date:   Fri Apr 3 14:29:06 2020 +0100

    Invites v2 endpoint (#952)

    * Start converting v1 invite endpoint to v2

    * Update gomatrixserverlib

    * Early federationsender code for sending invites

    * Sending invites sorta happens now

    * Populate invite request with stripped state

    * Remodel a bit, don't reflect received invites

    * Handle invite_room_state

    * Handle room versions a bit better

    * Update gomatrixserverlib

    * Tweak order in destinationQueue.next

    * Revert check in processMessage

    * Tweak federation sender destination queue code a bit

    * Add comments

commit 955244c092
Author: Ben B <benne@klimlive.de>
Date:   Fri Apr 3 12:40:50 2020 +0200

    use custom http client instead of the http DefaultClient (#823)

    This commit replaces the default client from the http lib with a custom one.
    The previously used default client doesn't come with a timeout. This could cause
    unwanted locks.
    That solution chosen here creates a http client in the base component dendrite
    with a constant timeout of 30 seconds. If it should be necessary to overwrite
    this, we could include the timeout in the dendrite configuration.
    Here it would be a good idea to extend the type "Address" by a timeout and
    create an http client for each service.

    Closes #820

    Signed-off-by: Benedikt Bongartz <benne@klimlive.de>

    Co-authored-by: Kegsay <kegan@matrix.org>

* Update sytest-whitelist, sytest-blacklist

* Update go.mod/go.sum

* Add some error wrapping for debug

* Add a NOTSPEC to common/events.go

* Perform state resolution at send_join

* Set default room version to v2 again

* Tweak GetCapabilities

* Add comments to ResolveConflictsAdhoc

* Update sytest-blacklist

* go mod tidy

* Update sytest-whitelist, sytest-blacklist

* Update versions

* Updates from review comments

* Update sytest-blacklist, sytest-whitelist

* Check room versions compatible at make_join, add some comments, update gomatrixserverlib, other tweaks

* Set default room version back to v2

* Update gomatrixserverlib, sytest-whitelist
2020-04-09 15:46:06 +01:00
Neil Alexander 067b875063
Invites v2 endpoint (#952)
* Start converting v1 invite endpoint to v2

* Update gomatrixserverlib

* Early federationsender code for sending invites

* Sending invites sorta happens now

* Populate invite request with stripped state

* Remodel a bit, don't reflect received invites

* Handle invite_room_state

* Handle room versions a bit better

* Update gomatrixserverlib

* Tweak order in destinationQueue.next

* Revert check in processMessage

* Tweak federation sender destination queue code a bit

* Add comments
2020-04-03 14:29:06 +01:00
Ben B 955244c092
use custom http client instead of the http DefaultClient (#823)
This commit replaces the default client from the http lib with a custom one.
The previously used default client doesn't come with a timeout. This could cause
unwanted locks.
That solution chosen here creates a http client in the base component dendrite
with a constant timeout of 30 seconds. If it should be necessary to overwrite
this, we could include the timeout in the dendrite configuration.
Here it would be a good idea to extend the type "Address" by a timeout and
create an http client for each service.

Closes #820

Signed-off-by: Benedikt Bongartz <benne@klimlive.de>

Co-authored-by: Kegsay <kegan@matrix.org>
2020-04-03 11:40:50 +01:00
Alex Chen 2c8950221e
Fix: show-expected-fail-tests.sh ignores # in whitelist and strips quotes in test names in output (#943)
Signed-off-by: Alex Chen <minecnly@gmail.com>
2020-03-31 15:53:09 +01:00
Kegsay 8fbe9f4078
Implement typing over federation (#949)
Also fix a pet peeve of mine: not putting units on things!!!

Manually tested on p2p and works well, with some fudge factor delay.
2020-03-30 16:40:28 +01:00
Kegsay 11a8059bba
Rename the typing server to EDU server (#948)
* Blunt move and sed rename

* Rename common/ refs to typing

* Rename internal stuff in eduserver

* Rename docs and scripts

* Rename constants/filenames, goimports everything to re-order imports
2020-03-30 15:02:20 +01:00
Kegan Dougal f72b759426 Update to latest go-sqite-js 2020-03-30 12:47:19 +01:00
Neil Alexander 664f31ec98 Ensure state res results are unique 2020-03-30 09:51:45 +01:00
Neil Alexander 05e1ae8745
Further room version wiring (#936)
* Room version 2 by default, other wiring updates, update gomatrixserverlib

* Fix nil pointer exception

* Fix some more nil pointer exceptions hopefully

* Update gomatrixserverlib

* Send all room versions when joining, not just stable ones

* Remove room version cquery

* Get room version when getting events from the roomserver database

* Reset default back to room version 2

* Don't generate event IDs unless needed

* Revert "Remove room version cquery"

This reverts commit a170d5873360dd059614460acc8b21ab2cda9767.

* Query room version in federation API, client API as needed

* Improvements to make_join send_join dance

* Make room server producers use headered events

* Lint tweaks

* Update gomatrixserverlib

* Versioned SendJoin

* Query room version in syncapi backfill

* Handle transaction marshalling/unmarshalling within Dendrite

* Sorta fix federation (kinda)

* whoops commit federation API too

* Use NewEventFromTrustedJSON when getting events from the database

* Update gomatrixserverlib

* Strip headers on federationapi endpoints

* Fix bug in clientapi profile room version query

* Update gomatrixserverlib

* Return more useful error if room version query doesn't find the room

* Update gomatrixserverlib

* Update gomatrixserverlib

* Maybe fix federation

* Fix formatting directive

* Update sytest whitelist and blacklist

* Temporarily disable room versions 3 and 4 until gmsl is fixed

* Fix count of EDUs in logging

* Update gomatrixserverlib

* Update gomatrixserverlib

* Update gomatrixserverlib

* Rely on EventBuilder in gmsl to generate the event IDs for us

* Some review comments fixed

* Move function out of common and into gmsl

* Comment in federationsender destinationqueue

* Update gomatrixserverlib
2020-03-27 16:28:22 +00:00
Suvid Sahay 314da91f1d
Fixed Didn't call Close on UserStreamListener log messages when syncing (#940)
Signed-off-by: Suvid Sahay <sahay.suvid61@gmail.com>

Removing the extra imports
2020-03-26 09:57:18 +00:00
Neil Alexander 0b732d6f45
Use HeaderedEvents in appservice component (#939)
* App service HeaderedEvents

* Fix database queries

* Fix lint error
2020-03-24 15:46:17 +00:00
Kegan Dougal 951b5d5e68 Missing whitelist 2020-03-24 13:39:50 +00:00
Kegsay 6bac7e5efd
Implement backfill over federation (#938)
* Implement history visibility checks for /backfill

Required for p2p to show history correctly.

* Add sytest

* Logging

* Fix two backfill bugs which prevented backfill from working correctly

- When receiving backfill requests, do not send the event that was in the original request.
- When storing backfill results, correctly update the backwards extremity for the room.

* hack: make backfill work multiple times

* add sqlite impl and remove logging

* Linting
2020-03-24 12:20:10 +00:00
Alex Chen 5a1a1ded1b
Fix dendrite config file location in docker guide (#934) 2020-03-23 11:16:17 +00:00
Prateek Sachan c5b7a17272
Added special case for m.room.member event in appservice component (#874)
* Added special case for m.room.member event in appservice component

* fix review changes

Co-authored-by: Kegsay <kegan@matrix.org>
2020-03-20 21:40:24 +00:00
Neil Alexander f2030286de
Room server changes for room versions (#930)
* Rearrange state package a bit, add some code to look up the right state resolution algorithm

* Remove shared

* Add GetRoomVersionForRoomNID

* Try to use room version to get correct state resolution algorithm

* Fix room joins over federation

* nolint resolveConflictsV2 because all attempts to break it up so far just result in it being awfully less obvious how it works

* Rename Prepare to NewStateResolution

* Update comments

* Re-add missing tests
2020-03-19 18:33:04 +00:00
Neil Alexander 944d454cb0 Fix SQLite problems after sync API changes 2020-03-19 13:15:35 +00:00
Neil Alexander ad5849d222
HeaderedEvents in sync API (#922)
* Use HeaderedEvent in syncapi

* Update notifier test

* Fix persisting headered event

* Clean up unused API function

* Fix overshadowed err from linter

* Write headered JSON to invites table too

* Rename event_json to headered_event_json in syncapi database schemae

* Fix invites_table queries

* Update QueryRoomVersionCapabilitiesResponse comment

* Fix syncapi SQLite
2020-03-19 12:07:01 +00:00
Kegsay bfbf96eec9
p2p: Implement published rooms (#923)
* Create and glue ExternalPublicRoomsProvider into the public rooms component

This is how we will link p2p stuff to dendrite proper.

* Use gmsl structs rather than our own

* Implement federated public rooms

- Make thirdparty endpoint r0 so riot-web loads the public room list

* Typo

* Missing callsites
2020-03-19 11:04:08 +00:00
Prateek Sachan dc06c69887
Implement /joined_rooms (#911)
* Implemented /joined_rooms

* Removed account endpoint added by mistake

* trigger ci
2020-03-19 10:25:36 +00:00
Abhinav Krishna C K ec38783192
maintenance: Fix matrix-org#896 use %w format verb to wrap errors (#916)
* maintenance: Fix matrix-org#896 use %w format verb to wrap errors

* In Go version 1.13 a new formatting verb introduced for fmt.Errorf
  %w https://blog.golang.org/go1.13-errors

* update %s to %w to wrap errors.

* Update all instances of error type to use %w

Signed-off-by: Abhinav Krishna C K <me@abhy.me>

Co-authored-by: Kegsay <kegan@matrix.org>
2020-03-18 12:48:51 +00:00