* Initial work on persistent queues
* Update index for event ID and server name
* Put things into database (postgres for now)
* Duplicate postgres code into sqlite for now just to stop build errors, will fix SQLite soon
* Fix table name
* Fix index
* Fix table name
* Use RETURNING because LastInsertID is not supported by postgres
* Use functions
* Marshal headered event
* Don't error on now rows
* Don't block if there are PDUs waiting
* Try to tidy up JSON
* Debug logging
* Fix query, use transactions in postgres
* Clean up
* Rehydrate more opportunistically
* Fix SQLite
* remove unused types
* Review comments
* Shuffle things around a bit
* Clean up transaction properly
* Don't send empty transactions
* Reduce unnecessary retries
* Count PDUs to make more resilient
* Don't stop when there is work to be done
* Try to limit wakeups
* well this is tedious
* Fix race in incomplete transactions
* Thread safety on transaction ID/count
* Remove membership table from account DB
And make code which needs that data use the currentstate server
* Unbreak tests; use a membership enum for space
* BREAKING: Make eduserver/appservice use userapi
This is a breaking change because this PR restructures how the AS API
tracks its position in Kafka streams. Previously, it used the account DB
to store partition offsets. However, this is also being used by `clientapi`
for the same purpose, which is bad (each component needs to store offsets
independently or else you might lose messages across restarts). This PR
changes this behaviour to now store partition offsets in the `appservice`
database.
This means that:
- Upon restart, the `appservice` component will attempt to replay all
room events from the beginning of time.
- An additional table will be created in the appservice database, which
in and of itself is backwards compatible.
* Return ErrorConflict
* Make userapi responsible for checking access tokens
There's still plenty of dependencies on account/device DBs, but this
is a start. This is a breaking change as it adds a required config
value `listen.user_api`.
* Cleanup
* Review comments and test fix
* More key tweaks
* Start testing stuff
* Move responsibility for generating local keys into server key API, don't register prom in caches unless needed, start tests
* Don't store our own keys in the database
* Don't store our own keys in the database
* Don't run tests for now
* Tweak caching behaviour, update tests
* Update comments, add fixes from forward-merge
* Debug logging
* Debug logging
* Perform final comparison against original set of requests
* oops
* Fetcher timeouts
* Fetcher timeouts
* missing func
* Tweaks
* Update gomatrixserverlib
* Fix Federation API test
* Break up FetchKeys
* Add comments to caching
* Add URL check in test
* Partially revert "Move responsibility for generating local keys into server key API, don't register prom in caches unless needed, start tests"
This reverts commit d7eb54c5b30b2f6a9d6514b643e32e6ad2b602f3.
* Fix federation API test
* Fix internal cache stuff again
* Fix server key API test
* Update comments
* Update comments from review
* Fix lint
* Fix rooms v3 url paths for good - with tests
- Add a test rig around `federationapi` to test routing.
- Use `JSONVerifier` over `KeyRing` so we can stub things out more easily.
- Add `test.NopJSONVerifier` which verifies nothing.
- Add `base.BaseMux` which is the original `mux.Router` used to spawn public/internal routers.
- Listen on `base.BaseMux` and not the default serve mux as it cleans paths which we don't want.
- Factor out `ListenAndServe` to `test.ListenAndServe` and add flag for listening on TLS.
* Fix comments
* Linting
This is a wrapper around whatever impl we have which then logs
the function name/request/response/error.
Also tweak when we log on kafka streams: only log on the producer
side not the consumer side: we've never had issues with comms and
having 1 message rather than N would be nice.
* Remove clientapi producers which aren't actually producers
They are actually just convenience wrappers around the internal APIs
for roomserver/eduserver. Move their logic to their respective `api`
packages and call them directly.
* Remove TODO
* unbreak ygg
Previously we had 3 monoliths:
- dendrite-monolith-server
- dendrite-demo-libp2p
- dendritejs
which all had their own of setting up public routes. Factor this
out into a new `setup.Monolith` struct which gets all dependencies
set as fields. This is different to `basecomponent.Base` which
doesn't provide any way to set configured deps (e.g public rooms db)
Part of a larger process to clean up how we initialise Dendrite.
* Split out adding HTTP routes from making internal APIs for clarity
* Split out more components
* Split out more things
* Finish converting
* internal mux for internal routes
* Return newly created error if user already exists (#1002)
Signed-off-by: Till Faelligen <tfaelligen@gmail.com>
* Rename variable
* Remove check for account and use returned error
* Return ErrUserExists
Signed-off-by: Till Faelligen <tfaelligen@gmail.com>
* State that CreateAccount will return err ErrUserExists if the user exists
Signed-off-by: Till Faelligen <tfaelligen@gmail.com>
* Also check sqlite for constraint error
* Revert "Also check sqlite for constraint error"
This reverts commit 7d310514
* Check for sqlite3 constraint error
* Add documentation to CreateAccount
* Move ErrUserExists to accounts package
* Revert "Move ErrUserExists to accounts package"
Import Cycle..
This reverts commit be3d4cda
Co-authored-by: Kegsay <kegan@matrix.org>
* Add a way to force federationsender to retry sending transactions
And use it in P2P mode when we pick up new nodes.
* Linting
* Use atomic bool to stop us blocking on the channel
* Groundwork for send-to-device messaging
* Update sample config
* Add unstable routing for now
* Send to device consumer in sync API
* Start the send-to-device consumer
* fix indentation in dendrite-config.yaml
* Create send-to-device database tables, other tweaks
* Add some logic for send-to-device messages, add them into sync stream
* Handle incoming send-to-device messages, count them with EDU stream pos
* Undo changes to test
* pq.Array
* Fix sync
* Logging
* Fix a couple of transaction things, fix client API
* Add send-to-device test, hopefully fix bugs
* Comments
* Refactor a bit
* Fix schema
* Fix queries
* Debug logging
* Fix storing and retrieving of send-to-device messages
* Try to avoid database locks
* Update sync position
* Use latest sync position
* Jiggle about sync a bit
* Fix tests
* Break out the retrieval from the update/delete behaviour
* Comments
* nolint on getResponseWithPDUsForCompleteSync
* Try to line up sync tokens again
* Implement wildcard
* Add all send-to-device tests to whitelist, what could possibly go wrong?
* Only care about wildcard when targeted locally
* Deduplicate transactions
* Handle tokens properly, return immediately if waiting send-to-device messages
* Fix sync
* Update sytest-whitelist
* Fix copyright notice (need to do more of this)
* Comments, copyrights
* Return errors from Do, fix dendritejs
* Review comments
* Comments
* Constructor for TransactionWriter
* defletions
* Update gomatrixserverlib, sytest-blacklist
- We don't want a serverKeyAPI as fetching keys doesn't need a DB.
- De-dupe rooms so we don't see them multiple times, but shuffle the
alias we join via so we don't all flood a single server.
* Server key API (works for monolith but not for polylith yet)
* Re-enable caching on server key API component
* Groundwork for HTTP APIs for server key API
* Hopefully implement HTTP for server key API
* Simplify public key request marshalling from map keys
* Update gomatrixserverlib
* go mod tidy
* Common -> internal
* remove keyring.go
* Update Docker Hub for server key API
* YAML is funny about indentation
* Wire in new server key API into hybrid monolith mode
* Create maps
* Route server key API endpoints on internal API mux
* Fix server key API URLs
* Add fetcher behaviour into server key API implementation
* Return error if we failed to fetch some keys
* Return results anyway
* Move things about a bit
* Remove unused code
* Fix comments, don't use federation sender URL in polylith mode
* Add server_key_api to sample config
* Review comments
* HTTP API to cache keys that have been requested
* Overwrite server_key_api listen in monolith hybrid mode
* Separate muxes for public and internal APIs
* Update client-api-proxy and federation-api-proxy so they don't add /api to the path
* Tidy up
* Consistent HTTP setup
* Set up prefixes properly
* Limit database connections (#564)
- Add new options to the config file database:
max_open_conns: 100
max_idle_conns: 2
conn_max_lifetime: -1
- Implement connection parameter setup on the *DB (database/sql) in internal/sqlutil/trace.go:Open()
- Propagate the values in the form of DbProperties interface via all the
Open() and NewDatabase() functions
Signed-off-by: Tomas Jirka <tomas.jirka@email.cz>
* Fix wasm builds
* Remove file accidentally added from working tree
Co-authored-by: Tomas Jirka <tomas.jirka@email.cz>
* Consolidation of roomserver APIs
* Comment out alias tests for now, they are broken
* Wire AS API into roomserver again
* Roomserver didn't take asAPI param before so return to that
* Prevent roomserver asking AS API for alias info
* Rename some files
* Remove alias_test, incoherent tests and unwanted appservice integration
* Remove FS API inject on syncapi component
* Define an input API for the federationsender
* Wiring for rooomserver input API and federation sender input API
* Whoops, commit common too
* Merge input API into query API
* Rename FederationSenderQueryAPI to FederationSenderInternalAPI
* Fix dendritejs
* Rename Input to Perform
* Fix a couple of inputs -> performs
* Remove needless storage interface, add comments
* Initial cut for backfilling
The syncserver now asks the roomserver via QueryBackfill (which already
existed to *handle* backfill requests) which then makes federation requests
via gomatrixserverlib.RequestBackfill.
Currently, tests fail on subsequent /messages requests because we don't know
which servers are in the room, because we are unable to get state snapshots
from a backfilled event because that code doesn't exist yet.
* WIP backfill, doesn't work
* Make initial backfill pass checks
* Persist backfilled events with state snapshots
* Remove debug lines
* Linting
* Review comments
* Experimental LRU cache for room versions
* Don't accidentally try to type-assert nil
* Also reduce hits on query API
* Use hashicorp implementation which mutexes for us
* Define const for max cache entries
* Rename to be specifically immutable, panic if we try to mutate a cache entry
* Review comments
* Remove nil guards, give roomserver integration test a cache
* go mod tidy
* Update gomatrixserverlib
* Test matrix.org as perspective key server
* Base64 decode better
* Optional strict validity checking in gmsl
* Update gomatrixserverlib
* Attempt to find missing auth events over federation (this shouldn't happen but I am guessing there is a synapse bug involved where we don't get all of the auth events)
* Update gomatrixserverlib, debug logging
* Remove debugging output
* More verbose debugging
* Print outliers
* Increase timeouts for testing, observe contexts before trying to join over more servers
* Don't block on roomserver (experimental)
* Don't block on roomserver
* Update gomatrixserverlib
* Update gomatrixserverlib
* Configurable perspective key fetchers
* Output number of configured keys for perspective
* Example perspective config included
* Undo debug stack trace
* Undo debug stack trace
* Restore original HTTP listener in monolith
* Fix lint
* Review comments
* Set default HTTP server timeout to 5 minutes now, block again when joining
* Don't use HTTP address for HTTPS whoops
* Update gomatrixserverlib
* Update gomatrixserverlib
* Update gomatrixserverlib
* Actually add perspectives
* Actually add perspectives
* Update gomatrixserverlib
* Add libp2p-go
* Some tweaks, tidying up
(cherry picked from commit 1a5bb121f8121c4f68a27abbf25a9a35a1b7c63e)
* Move p2p dockerfile
(cherry picked from commit 8d3bf44ea1bf37f950034e73bcdc315afdabe79a)
* Remove containsBackwardsExtremity
* Fix some linter errors, update some libp2p packages/calls, other tidying up
* Add -port for dendrite-p2p-demo
* Use instance name as key ID
* Remove P2P demo docker stuff, no longer needed now that we have SQLite
* Remove Dockerfile-p2p too
* Remove p2p logic from dendrite-monolith-server
* Inject publicRoomsDB in publicroomsapi
Inject publicRoomsDB instead of switching on base.libP2P.
See: https://github.com/matrix-org/dendrite/pull/956/files?file-filters%5B%5D=.go#r406276914
* Fix lint warning
* Extract mDNSListener from base.go
* Extract CreateFederationClient into demo
* Create P2PDendrite from BaseDendrite
Extract logic specific to P2PDendrite from base.go
* Set base.go to upstream/master
* Move pubsub to demo cmd
* Move PostgreswithDHT to cmd
* Remove unstable features
* Add copyrights
* Move libp2pvalidator into p2pdendrite
* Rename dendrite-p2p-demo -> dendrite-demo-libp2p
* Update copyrights
* go mod tidy
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
* Add setting to enable/disable metrics (#461)
Add basic auth to /metric handlers
Signed-off-by: Till Faelligen <tfaelligen@gmail.com>
* Add warning message if metrics are exposed without protection
* Remove redundant type conversion
Signed-off-by: Till Faelligen <tfaelligen@gmail.com>
* SetBasicAuth per test case
* Update warning message and change loglevel to warn
* Update common/config/config.go
* Update dendrite-config.yaml
Co-authored-by: Till Faelligen <tfaelligen@gmail.com>
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
This commit replaces the default client from the http lib with a custom one.
The previously used default client doesn't come with a timeout. This could cause
unwanted locks.
That solution chosen here creates a http client in the base component dendrite
with a constant timeout of 30 seconds. If it should be necessary to overwrite
this, we could include the timeout in the dendrite configuration.
Here it would be a good idea to extend the type "Address" by a timeout and
create an http client for each service.
Closes#820
Signed-off-by: Benedikt Bongartz <benne@klimlive.de>
Co-authored-by: Kegsay <kegan@matrix.org>
* Room version 2 by default, other wiring updates, update gomatrixserverlib
* Fix nil pointer exception
* Fix some more nil pointer exceptions hopefully
* Update gomatrixserverlib
* Send all room versions when joining, not just stable ones
* Remove room version cquery
* Get room version when getting events from the roomserver database
* Reset default back to room version 2
* Don't generate event IDs unless needed
* Revert "Remove room version cquery"
This reverts commit a170d5873360dd059614460acc8b21ab2cda9767.
* Query room version in federation API, client API as needed
* Improvements to make_join send_join dance
* Make room server producers use headered events
* Lint tweaks
* Update gomatrixserverlib
* Versioned SendJoin
* Query room version in syncapi backfill
* Handle transaction marshalling/unmarshalling within Dendrite
* Sorta fix federation (kinda)
* whoops commit federation API too
* Use NewEventFromTrustedJSON when getting events from the database
* Update gomatrixserverlib
* Strip headers on federationapi endpoints
* Fix bug in clientapi profile room version query
* Update gomatrixserverlib
* Return more useful error if room version query doesn't find the room
* Update gomatrixserverlib
* Update gomatrixserverlib
* Maybe fix federation
* Fix formatting directive
* Update sytest whitelist and blacklist
* Temporarily disable room versions 3 and 4 until gmsl is fixed
* Fix count of EDUs in logging
* Update gomatrixserverlib
* Update gomatrixserverlib
* Update gomatrixserverlib
* Rely on EventBuilder in gmsl to generate the event IDs for us
* Some review comments fixed
* Move function out of common and into gmsl
* Comment in federationsender destinationqueue
* Update gomatrixserverlib
* Create and glue ExternalPublicRoomsProvider into the public rooms component
This is how we will link p2p stuff to dendrite proper.
* Use gmsl structs rather than our own
* Implement federated public rooms
- Make thirdparty endpoint r0 so riot-web loads the public room list
* Typo
* Missing callsites
* maintenance: Fix matrix-org#896 use %w format verb to wrap errors
* In Go version 1.13 a new formatting verb introduced for fmt.Errorf
%w https://blog.golang.org/go1.13-errors
* update %s to %w to wrap errors.
* Update all instances of error type to use %w
Signed-off-by: Abhinav Krishna C K <me@abhy.me>
Co-authored-by: Kegsay <kegan@matrix.org>
* p2p: Use JSServer for comms rather than GoJsConn
This has several benefits:
- it fixes a bug whereby you could not transmit >4k bytes to/from JS/Go land.
- it more clearly exposes the interface point between Go and JS: a single
global function call.
- it presents a nicer API shape than the previous `net.Conn`.
- it doesn't needlessly 'stream' data which is already sitting in-memory.
This is currently only active for local CS API traffic, another PR will
add Federation P2P support.
* Typo
* Pass in the private key seed to NewP2PLocalNode to use the same key for p2p
* Extract the ed25519 key from the server name and use it for event checks
This allows HSes which no longer exist to still have events verified for
them. This means that new members can join rooms that has missing servers.
* Remove no-op ipfs uri which we don't use anymore
* Use a fork of pq which supports userCurrent on wasm
* Use sqlite3_js driver when running in JS
* Add cmd/dendritejs to pull in sqlite3_js driver for wasm only
* Update to latest go-sqlite-js version
* Replace prometheus with a stub. sigh
* Hard-code a config and don't use opentracing
* Latest go-sqlite3-js version
* Generate a key for now
* Listen for fetch traffic rather than HTTP
* Latest hacks for js
* libp2p support
* More libp2p
* Fork gjson to allow us to enforce auth checks as before
Previously, all events would come down redacted because the hash
checks would fail. They would fail because sjson.DeleteBytes didn't
remove keys not used for hashing. This didn't work because of a build
tag which included a file which no-oped the index returned.
See https://github.com/tidwall/gjson/issues/157
When it's resolved, let's go back to mainline.
* Use gjson@1.6.0 as it fixes https://github.com/tidwall/gjson/issues/157
* Use latest gomatrixserverlib for sig checks
* Fix a bug which could cause exclude_from_sync to not be set
Caused when sending events over federation.
* Use query variadic to make lookups actually work!
* Latest gomatrixserverlib
* Add notes on getting p2p up and running
Partly so I don't forget myself!
* refactor: Move p2p specific stuff to cmd/dendritejs
This is important or else the normal build of dendrite will fail
because the p2p libraries depend on syscall/js which doesn't work
on normal builds.
Also, clean up main.go to read a bit better.
* Update ho-http-js-libp2p to return errors from RoundTrip
* Add an LRU cache around the key DB
We actually need this for P2P because otherwise we can *segfault*
with things like: "runtime: unexpected return pc for runtime.handleEvent"
where the event is a `syscall/js` event, caused by spamming sql.js
caused by "Checking event signatures for 14 events of room state" which
hammers the key DB repeatedly in quick succession.
Using a cache fixes this, though the underlying cause is probably a bug
in the version of Go I'm on (1.13.7)
* breaking: Add Tracing.Enabled to toggle whether we do opentracing
Defaults to false, which is why this is a breaking change. We need
this flag because WASM builds cannot do opentracing.
* Start adding conditional builds for wasm to handle lib/pq
The general idea here is to have the wasm build have a `NewXXXDatabase`
that doesn't import any postgres package and hence we never import
`lib/pq`, which doesn't work under WASM (undefined `userCurrent`).
* Remove lib/pq for wasm for syncapi
* Add conditional building to remaining storage APIs
* Update build script to set env vars correctly for dendritejs
* sqlite bug fixes
* Docs
* Add a no-op main for dendritejs when not building under wasm
* Use the real prometheus, even for WASM
Instead, the dendrite-sw.js must mock out `process.pid` and
`fs.stat` - which must invoke the callback with an error (e.g `EINVAL`)
in order for it to work:
```
global.process = {
pid: 1,
};
global.fs.stat = function(path, cb) {
cb({
code: "EINVAL",
});
}
```
* Linting
* Move current work into single branch
* Initial massaging of clientapi etc (not working yet)
* Interfaces for accounts/devices databases
* Duplicate postgres package for sqlite3 (no changes made to it yet)
* Some keydb, accountdb, devicedb, common partition fixes, some more syncapi tweaking
* Fix accounts DB, device DB
* Update naffka dependency for SQLite
* Naffka SQLite
* Update naffka to latest master
* SQLite support for federationsender
* Mostly not-bad support for SQLite in syncapi (although there are problems where lots of events get classed incorrectly as backward extremities, probably because of IN/ANY clauses that are badly supported)
* Update Dockerfile -> Go 1.13.7, add build-base (as gcc and friends are needed for SQLite)
* Implement GET endpoints for account_data in clientapi
* Nuke filtering for now...
* Revert "Implement GET endpoints for account_data in clientapi"
This reverts commit 4d80dff4583d278620d9b3ed437e9fcd8d4674ee.
* Implement GET endpoints for account_data in clientapi (#861)
* Implement GET endpoints for account_data in clientapi
* Fix accountDB parameter
* Remove fmt.Println
* Fix insertAccountData SQLite query
* Fix accountDB storage interfaces
* Add empty push rules into account data on account creation (#862)
* Put SaveAccountData into the right function this time
* Not sure if roomserver is better or worse now
* sqlite work
* Allow empty last sent ID for the first event
* sqlite: room creation works
* Support sending messages
* Nuke fmt.println
* Move QueryVariadic etc into common, other device fixes
* Fix some linter issues
* Fix bugs
* Fix some linting errors
* Fix errcheck lint errors
* Make naffka use postgres as fallback, fix couple of compile errors
* What on earth happened to the /rooms/{roomID}/send/{eventType} routing
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
* Wire up publicroomsapi to roomserver events
* Remove parameter that was incorrectly brought over from p2p work
* nolint containsBackwardExtremity for now
* Merge forward
* Tidy up a bit
* TODO: What to do with NextBatch here?
* Replace SyncPosition with PaginationToken throughout syncapi
* Fix PaginationTokens
* Fix lint errors
* Add a couple of missing functions into the syncapi external storage interface
* Some updates based on review comments from @babolivier
* Some updates based on review comments from @babolivier
* argh whitespacing
* Fix opentracing span
* Remove dead code
* Don't overshadow err (fix lint issue)
* Handle extremities after inserting event into topology
* Try insert event topology as ON CONFLICT DO NOTHING
* Prevent OOB error in addRoomDeltaToResponse
* Thwarted by gocyclo again
* Fix NewPaginationTokenFromString, define unit test for it
* Update pagination token test
* Update sytest-whitelist
* Hopefully fix some of the sync batch tokens
* Remove extraneous sync position func
* Revert to topology tokens in addRoomDeltaToResponse etc
* Fix typo
* Remove prevPDUPos as dead now that backwardTopologyPos is used instead
* Fix selectEventsWithEventIDsSQL
* Update sytest-blacklist
* Update sytest-whitelist
This PR adds a block in the dendrite config for the services to bind to. The microservices should bind to the addresses in the bind block, and will be contacted at the address in the listen block.
This fixes an issue with the microservices and kubernetes services.
Neil Alexander
Kegsay
Ashley Nelson
S7evinK
Hilmar Gústafsson
Ben B
Abhinav Krishna C K
aditsachde
Alex Chen
Andrew Morgan
ruben