* Add configuration for max_message_bytes for sarama
* Log all errors when sending multiple messages
Signed-off-by: Till Faelligen <tfaelligen@gmail.com>
* Add missing config
* - Better comments on what MaxMessageBytes is used for
- Also sets the size the consumer may use
* Rename serverkeyapi to signingkeyserver
We use "api" for public facing stuff and "server" for internal stuff.
As the server key API is internal only, we call it 'signing key server',
which also clarifies the type of key (as opposed to TLS keys, E2E keys, etc)
* Convert docker/scripts to use signing-key-server
* Rename missed bits
* Initial pass at refactoring config (not finished)
* Don't forget current state and EDU servers
* More shifting around
* Update server key API tests
* Fix roomserver test
* Fix more tests
* Further tweaks
* Fix current state server test (sort of)
* Maybe fix appservices
* Fix client API test
* Include database connection string in database options
* Fix sync API build
* Update config test
* Fix unit tests
* Fix federation sender build
* Fix gobind build
* Set Listen address for all services in HTTP monolith mode
* Validate config, reinstate appservice derived in directory, tweaks
* Tweak federation API test
* Set MaxOpenConnections/MaxIdleConnections to previous values
* Update generate-config
* Add storage layer for postgres/sqlite
* Return OTK counts when inserting new keys
* Hook up the key DB and make a test pass
* Convert postgres queries to be sqlite queries
* Blacklist test due to requiring rejected events
* Unbreak tests
* Update blacklist
* Add a new component: currentstateserver
- Add a skeleton for it, with databases and a single query method.
- Add integration tests for it.
- Add listen/address fields in the config (breaking as this will force people to specify this to validate)
Not currently hooked up to anything yet.
* Unbreak config tests
* Add current_state to sample config
* comments
* Make userapi responsible for checking access tokens
There's still plenty of dependencies on account/device DBs, but this
is a start. This is a breaking change as it adds a required config
value `listen.user_api`.
* Cleanup
* Review comments and test fix
* Groundwork for send-to-device messaging
* Update sample config
* Add unstable routing for now
* Send to device consumer in sync API
* Start the send-to-device consumer
* fix indentation in dendrite-config.yaml
* Create send-to-device database tables, other tweaks
* Add some logic for send-to-device messages, add them into sync stream
* Handle incoming send-to-device messages, count them with EDU stream pos
* Undo changes to test
* pq.Array
* Fix sync
* Logging
* Fix a couple of transaction things, fix client API
* Add send-to-device test, hopefully fix bugs
* Comments
* Refactor a bit
* Fix schema
* Fix queries
* Debug logging
* Fix storing and retrieving of send-to-device messages
* Try to avoid database locks
* Update sync position
* Use latest sync position
* Jiggle about sync a bit
* Fix tests
* Break out the retrieval from the update/delete behaviour
* Comments
* nolint on getResponseWithPDUsForCompleteSync
* Try to line up sync tokens again
* Implement wildcard
* Add all send-to-device tests to whitelist, what could possibly go wrong?
* Only care about wildcard when targeted locally
* Deduplicate transactions
* Handle tokens properly, return immediately if waiting send-to-device messages
* Fix sync
* Update sytest-whitelist
* Fix copyright notice (need to do more of this)
* Comments, copyrights
* Return errors from Do, fix dendritejs
* Review comments
* Comments
* Constructor for TransactionWriter
* defletions
* Update gomatrixserverlib, sytest-blacklist
* Server key API (works for monolith but not for polylith yet)
* Re-enable caching on server key API component
* Groundwork for HTTP APIs for server key API
* Hopefully implement HTTP for server key API
* Simplify public key request marshalling from map keys
* Update gomatrixserverlib
* go mod tidy
* Common -> internal
* remove keyring.go
* Update Docker Hub for server key API
* YAML is funny about indentation
* Wire in new server key API into hybrid monolith mode
* Create maps
* Route server key API endpoints on internal API mux
* Fix server key API URLs
* Add fetcher behaviour into server key API implementation
* Return error if we failed to fetch some keys
* Return results anyway
* Move things about a bit
* Remove unused code
* Fix comments, don't use federation sender URL in polylith mode
* Add server_key_api to sample config
* Review comments
* HTTP API to cache keys that have been requested
* Overwrite server_key_api listen in monolith hybrid mode
* Limit database connections (#564)
- Add new options to the config file database:
max_open_conns: 100
max_idle_conns: 2
conn_max_lifetime: -1
- Implement connection parameter setup on the *DB (database/sql) in internal/sqlutil/trace.go:Open()
- Propagate the values in the form of DbProperties interface via all the
Open() and NewDatabase() functions
Signed-off-by: Tomas Jirka <tomas.jirka@email.cz>
* Fix wasm builds
* Remove file accidentally added from working tree
Co-authored-by: Tomas Jirka <tomas.jirka@email.cz>
* Update gomatrixserverlib
* Test matrix.org as perspective key server
* Base64 decode better
* Optional strict validity checking in gmsl
* Update gomatrixserverlib
* Attempt to find missing auth events over federation (this shouldn't happen but I am guessing there is a synapse bug involved where we don't get all of the auth events)
* Update gomatrixserverlib, debug logging
* Remove debugging output
* More verbose debugging
* Print outliers
* Increase timeouts for testing, observe contexts before trying to join over more servers
* Don't block on roomserver (experimental)
* Don't block on roomserver
* Update gomatrixserverlib
* Update gomatrixserverlib
* Configurable perspective key fetchers
* Output number of configured keys for perspective
* Example perspective config included
* Undo debug stack trace
* Undo debug stack trace
* Restore original HTTP listener in monolith
* Fix lint
* Review comments
* Set default HTTP server timeout to 5 minutes now, block again when joining
* Don't use HTTP address for HTTPS whoops
* Update gomatrixserverlib
* Update gomatrixserverlib
* Update gomatrixserverlib
* Actually add perspectives
* Actually add perspectives
* Update gomatrixserverlib
* Add setting to enable/disable metrics (#461)
Add basic auth to /metric handlers
Signed-off-by: Till Faelligen <tfaelligen@gmail.com>
* Add warning message if metrics are exposed without protection
* Remove redundant type conversion
Signed-off-by: Till Faelligen <tfaelligen@gmail.com>
* SetBasicAuth per test case
* Update warning message and change loglevel to warn
* Update common/config/config.go
* Update dendrite-config.yaml
Co-authored-by: Till Faelligen <tfaelligen@gmail.com>
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
* update gomatrixserverlib
* Make removeUser public
* Implement api.TypingServerInputAPI
* Integrate the typing server component, create kafka topic
* Add typing server cmd for multiprocess dendrite
* Prevent sql scanning into nil value in accounts_table
Signed-off-by: Andrew Morgan <andrewm@matrix.org>
* Remove uneccessary logging, null checking
* Don't forget to set the localpart
* Simplify error checking
* Store And Send Application Service Events
* Modify INSTALL.md and dendrite-config.yaml for the new appservice database
* Correct all instances of casing on 'application service' to align with
spec
* Store incoming events that an app service is interested in in the
database to be later read by transaction workers.
* Retrieve these events from transaction workers, one per AS.
* Minimal transaction ID data is stored as well to recover after
server failure.
* Send events to AS and exponentially backoff on failure.
Signed-off-by: Andrew Morgan <andrewm@matrix.org>
* Finish my own sentences.
* Fix up database interaction
* Change to event-based AS sending
* Reduce cyclomatic complexity
* Appease the errcheck gods
* Delete by int ID instead of string.
This was causing some events to not be deleted, as < an eventID doesn't
really make much sense.
* Check if there are more events to send before sleeping
* Send same transaction if last send attempt failed
* Don't backoff on non-200s, tight send loop, 1 event query
* Remove tight send loop. Fix events not being deleted
* Additionally order by event id, track main.go
* Return the last txnID, which our events are using
* Remove old main.go file
* Prevent duplicate events from being sent...
* Strip event content if it doesn't contain anything
Signed-off-by: Andrew Morgan <andrewm@matrix.org>
* Update gomatrixserverlib and use Unsigned AS event prop
* Fixes
* Fix sync server comment
* Remove unnecessary printlns
* Use logrus Fields
* Worker state methods
* Remove sillyness
* Fix up event filtering
* Handle transaction event limit in loop
* Switch to using a sequence for transaction IDs
* Don't verify self-signed AS certificates
* Fix logging
* Use gmsl.Event instead of AS-only event in transactions
Also clear up the logic on lookupStateEvents a little bit.
* Change invalid_txn_id to global (for efficiency)
* Use a bool for EventsReady instead of an int
* Add Application Service component
The component subscribes to the Roomserver kafka stream, filtering
events to be eventually sent off to relevant app services, as well as
handling incoming HTTP traffic from app services on the
`/matrix/app/r0/*` route.
Signed-off-by: Andrew Morgan <andrewm@matrix.org>
* Make linting happy
Signed-off-by: Andrew Morgan <andrewm@matrix.org>
* Update the install instructions
To include some of the newer components.
* update INSTALL diagram
client_data goes to sync-api-server, not room-server.
* Add config for trusted ID servers
* Add new error
* Implement check for trusted ID server
* Complete unfinished comment
* Make comment more explicit in the config file
* Use go standard errors in membership.go
* Use standard errors instead of JSON responses in threepid
* Doc errors
* Remove unused parameter
* Move events contents to common
* Basic database structure
* Complete database update
* Support visibility update and retrieval
* Add HTTP methods for visibility update and retrieval
* Add the database for the new component
* Add a listener for the new component
* Fix attribute update statements
* Create public rooms component
* Fix failing test
* Add roomserver consumer
* Fix a bug in aliases creation
* Add a check on type
* Implement public rooms directory
* Use auth API for visibility update
* Support filtering
* Add component to monolith
* Various fixes
* Fix computation of next public rooms batch
* Retrieve state events from the roomserver query API + avoid dupes on join
* Split update of string or boolean attribute in two separate functions
* Use event type to detect duplicate joins
* Improve the joined members counter computation
* Use event.RoomID()
* dependency injection for the kafka consumers/producers
* Optionally use naffka in the monolithic server
* remember to call setupKafka()
* tweak imports
* fix integration tests
* Add use_naffka to the example config
* Update comment on the listen APIs
* Clean roomserver consumer
* Make account data sync incremental
* Use a different name for the sync AD table
* Improved error logging
* Created missing topic in tests
* Add client API topic to tests
* Add client API topic to common
* Move data batch retrieval
* Add database index for data retrieval
* Fix typo in table name
* Fix indentation
* Profile retrieval
* Saving avatar (without propagating it)
* Saving display name (without propagating it)
* Getters for display name and avatar URL
* Doc'd
* Introduced new Kafka topic and producer
* Updated config with new kafka topic
* Switched to samara producer and now sending messages
* Doc'd
* Put kafka update after the database insert
* Doc'd profileUpdate structure
* Removed unused parameter
* Moved user updates producer to clientapi/producers
S7evinK
Neil Alexander
Kegsay
Will Hunt
Aaron Raimist
Anant Prakash
Andrew Morgan
Tristan Claverie
Erik Johnston
Michael Telatynski
Mark Haines
Richard van der Hoff
Brendan Abolivier