* Update GMSL
* Add MSC2836EventRelationships to fedsender
* Call MSC2836EventRelationships in reqCtx
* auth remote servers
* Extract room ID and servers from previous events; refactor a bit
* initial cut of federated threading
* Use the right client/fed struct in the response
* Add QueryAuthChain for use with MSC2836
* Add auth chain to federated response
* Fix pointers
* under CI: more logging and enable mscs, nil fix
* Handle direction: up
* Actually send message events to the roomserver..
* Add children and children_hash to unsigned, with tests
* Add logic for exploring threads and tracking children; missing storage functions
* Implement storage functions for children
* Add fetchUnknownEvent
* Do federated hits for include_children if we have unexplored children
* Use /ev_rel rather than /event as the former includes child metadata
* Remove cross-room threading impl
* Enable MSC2836 in the p2p demo
* Namespace mscs db
* Enable msc2836 for ygg
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
* Don't recalculate event IDs so often
* Revert invite change
* Make sure we're using the right NIDs
* Update gomatrixserverlib
* Update to NewEventFromTrustedJSONWithEventID
* Fix go.mod
* Update gomatrixserverlib to matrix-org/gomatrixserverlib#243
* Use BulkSelectEventID
* Add mscs/hooks package, begin work for msc2836
* Flesh out hooks and add SQL schema
* Begin implementing core msc2836 logic
* Add test harness
* Linting
* Implement visibility checks; stub out APIs for tests
* Flesh out testing
* Flesh out walkThread a bit
* Persist the origin_server_ts as well
* Edges table instead of relationships
* Add nodes table for event metadata
* LEFT JOIN to extract origin_server_ts for children
* Add graph walking structs
* Implement walking algorithm
* Add more graph walking tests
* Add auto_join for local rooms
* Fix create table syntax on postgres
* Add relationship_room_id|servers to the unsigned section of events
* Persist the parent room_id/servers in edge metadata
Other events cannot assert the true room_id/servers for the
parent event, only make claims to them, hence why this is
edge metadata.
* guts to pass through room_id/servers
* Refactor msc2836 to allow handling from federation
* Add JoinedVia to PerformJoin responses
* Fix tests; review comments
The previous implementation was only checking if room history was
"shared", which it wasn't for rooms where a user was invited, or world
readable rooms.
This implementation leverages the IsServerAllowed method, which already
implements the complete verification algorithm.
Signed-off-by: `Mayeul Cantan <oss+matrix@mayeul.net>`
Co-authored-by: Kegsay <kegan@matrix.org>
* Add basic storage methods
* Add internal api handler
* Add check for forgotten room
* Add /rooms/{roomID}/forget endpoint
* Add missing rsAPI method
* Remove unused parameters
* Add passing tests
Signed-off-by: Till Faelligen <tfaelligen@gmail.com>
* Add missing file
* Add postgres migration
* Add sqlite migration
* Use Forgetter to forget room
* Remove empty line
* Update HTTP status codes
It looks like the spec calls for these to be 400, rather than 403: https://matrix.org/docs/spec/client_server/r0.6.1#post-matrix-client-r0-rooms-roomid-forget
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
* Add configuration for max_message_bytes for sarama
* Log all errors when sending multiple messages
Signed-off-by: Till Faelligen <tfaelligen@gmail.com>
* Add missing config
* - Better comments on what MaxMessageBytes is used for
- Also sets the size the consumer may use
* Set RewritesState once
* Check if any new state provided
* Obey rewritesState
* Don't nuke everything the sync API knows when purging state
* Fix panic from duplicate insert
* Consistency
* Use HasState
* Remove nolint
* Clean up joined rooms on state rewrite
* Add resolve-state helper
* Tweaks
* Refactor forward extremities, again
* Tweaks
* Minor optimisation
* Make path a bit clearer
* Only process state/membership if forward extremities have changed
* Usage comments in resolve-state
* Fix sqite locking bugs present on sytest
Comments do the explaining.
* Fix deadlock in sqlite mode
Caused by starting a writer whilst within a writer
* Only complain about invalid state deltas for non-overwrite events
* Do not re-process outlier unnecessarily
* Add KindOld
* Don't process latest events/memberships for old events
* Allow federationsender to ignore duplicate key entries when LatestEventIDs is duplicated by RS output events
* Signal to downstream components if an event has become a forward extremity
* Don't exclude from sync
* Soft-fail checks on KindNew
* Don't run the latest events updater at all for KindOld
* Don't make federation sender change after all
* Kind in federation sender join
* Don't send isForwardExtremity
* Fix syncapi
* Update comments
* Fix SendEventWithState
* Update sytest-whitelist
* Generate old output events
* Sync API consumes old room events
* Update comments
* Start Kafka connection for each component that needs one
* Fix roomserver unit tests
* Rename to naffkaInstance (@Kegsay review comment)
* Fix import cycle
* Capture errors
* Don't request only state key tuples needed for auth (we end up discarding room state this way)
* QueryStateAfterEvent returns all state when no tuples supplied
* Resolve state
* Comments
* Adjust backfill to send backward extremity with state before other backfilled events, include prev_events with no state amongst missing events
* Not finished refactor
* Fix test
* Remove isInboundTxn
* Remove debug logging
We expect to have missing events as we walk back in the DAG over federation
as we didn't always create the room. When checking if the server is allowed
to see those events, just give up and stop rather than fail the request.
* Rename serverkeyapi to signingkeyserver
We use "api" for public facing stuff and "server" for internal stuff.
As the server key API is internal only, we call it 'signing key server',
which also clarifies the type of key (as opposed to TLS keys, E2E keys, etc)
* Convert docker/scripts to use signing-key-server
* Rename missed bits
* Deep forward extremity calculation
* Use updater txn
* Update error
* Update error
* Create previous event references in StoreEvent
* Use latest events updater to row-lock prev events
* Fix unexpected fallthrough
* Fix deadlock
* Don't roll back
* Update comments in calculateLatest
* Don't include events that we can't find references for in the forward extremities
* Add another passing test
* Don't send rewrite events
* Remove final traces of rewrite events
* Remove test that is no longer needed
* Revert "Remove test that is no longer needed"
This reverts commit 9a45babff690480acd656a52f2c2950a5f7e9ada.
* Update test to use KindOutlier
* Resolve state after event against current room state when determining latest state changes
* Update sytest-whitelist
* Update sytest-whitelist, blacklist
* Try to ask other servers in the room for missing events if the origin won't provide them
* Logging
* More logging
* Implement QueryMissingAuthPrevEvents
* Try to get missing auth events badly
* Use processEvent
* Logging
* Update QueryMissingAuthPrevEvents
* Try to find missing auth events
* Patchy fix for test
* Logging tweaks
* Send auth events as outliers
* Update check in QueryMissingAuthPrevEvents
* Error responses
* More return codes
* Don't return error on reject/soft-fail since it was ultimately handled
* More tweaks
* More error tweaks
* Sanity-check room version on RS event input
* Update gomatrixserverlib
* Reject make_join when no room members are left
* Revert some changes from wrong branch
* Distinguish between room not existing and room being abandoned on this server
* nolint
* Replace all usages of txn.Stmt with sqlutil.TxStmt
Signed-off-by: Sam Day <me@samcday.com>
* Fix sign off link in PR template.
Signed-off-by: Sam Day <me@samcday.com>
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
* WIP Event rejection
* Still send back errors for rejected events
Instead, discard them at the federationapi /send layer rather than
re-implementing checks at the clientapi/PerformJoin layer.
* Implement rejected events
Critically, rejected events CAN cause state resolution to happen
as it can merge forks in the DAG. This is fine, _provided_ we
do not add the rejected event when performing state resolution,
which is what this PR does. It also fixes the error handling
when NotAllowed happens, as we were checking too early and needlessly
handling NotAllowed in more than one place.
* Update test to match reality
* Modify InputRoomEvents to no longer return an error
Errors do not serialise across HTTP boundaries in polylith mode,
so instead set fields on the InputRoomEventsResponse. Add `Err()`
function to make the API shape basically the same.
* Remove redundant returns; linting
* Update blacklist
* SendEventWithState events as new
* Use cumulative state IDs for final event
* Error wrapping in calculateAndSetState
* Handle overwriting same event type and state key
* Hacky way to spot historical events
* Don't exclude from sync
* Don't generate output events when rewriting forward extremities
* Update output event check
* Historical output events
* Define output room event type
* Notify key changes on state
* Don't send our membership event twice
* Deduplicate state entries
* Tweaks
* Remove unnecessary nolint
* Fix current state upsert in sync API
* Send auth events as outliers, state events as rewrite
* Sync API don't consume state events
* Process events actually
* Improve outlier check
* Fix local room check
* Remove extra room check, it seems to break the whole damn world
* Fix federated join check
* Fix nil pointer exception
* Better comments on DeduplicateStateEntries
* Reflow forced federated joins
* Don't force federated join for possibly even local invites
* Comment SendEventWithState better
* Rewrite room state in sync API storage
* Add TODO
* Clean up all room data when receiving create event
* Don't generate output events for rewrites, but instead notify that state is rewritten on the final new event
* Rename to PurgeRoom
* Exclude backfilled messages from /sync
* Split out rewriting state from updating state from state res
Co-authored-by: Kegan Dougal <kegan@matrix.org>
Neil Alexander
Kegsay
Mayeul Cantan
S7evinK
Sam
Matthew Hodgson