Commit Graph

181 Commits (30e9353fdd4cc2a3c3e5bd6de139ca0e31dae7bb)

Author SHA1 Message Date
Neil Alexander 30e9353fdd
Ensure user IDs match the spec (matrix-org/gomatrixserverlib#261) 2021-06-15 15:18:56 +01:00
Neil Alexander 5ec7d3a3ec
Update go.mod/go.sum 2021-06-14 15:04:05 +01:00
Neil Alexander 051cd63f1e
Update go.mod/go.sum 2021-06-14 13:26:02 +01:00
Neil Alexander bd9dec8e06
Pinecone demo updates 2021-06-14 13:13:07 +01:00
Neil Alexander b0aa101dcd
Update go.mod/go.sum for matrix-org/pinecone (Build 79) 2021-06-02 12:23:01 +01:00
Neil Alexander 3797d818c0
Update gomatrixserverlib to matrix-org/gomatrixserverlib#259 2021-05-25 12:01:07 +01:00
Neil Alexander 3f84cd6cd0
Update go.mod/go.sum 2021-05-25 09:52:55 +01:00
Neil Alexander 30f021700a
Update go.mod/go.sum 2021-05-25 09:49:02 +01:00
Neil Alexander a7f2845a6a
Demo tweaks 2021-05-24 13:12:05 +01:00
Neil Alexander d2d2164025
Update pinecone demo 2021-05-24 11:17:32 +01:00
Neil Alexander 2948ffd782
Update go.mod/go.sum 2021-05-18 16:20:04 +01:00
Neil Alexander 08a4370c94
Update go.mod/go.sum 2021-05-18 15:58:50 +01:00
Neil Alexander 140cae81cc
Update dendrite-demo-pinecone 2021-05-18 15:47:15 +01:00
Neil Alexander 1d89c4a3cd
Update go.mod/go.sum 2021-05-10 17:04:06 +01:00
Neil Alexander c76f820c86
Update go.mod/go.sum 2021-05-10 17:03:02 +01:00
Neil Alexander a49d06138e
Updates to dendrite-demo-pinecone 2021-05-10 16:59:03 +01:00
Neil Alexander 603bf590f0
Fix public room directory in Pinecone demo 2021-05-07 12:17:14 +01:00
Neil Alexander aa672068ab
Update pinecone in go.mod/go.sum 2021-05-07 11:49:44 +01:00
Neil Alexander 1002e87b60
Pinecone P2P demo (#1856)
* Pinecone demo

* Enable multicast, fix HTTP routing

* Fix multicast import

* Fix build

* Update Pinecone demo

* Fix the keys

* Tweaks

* Pinecone room directory support (early)

* Fix gobind-pinecone

* Add pinecone listener

* Fix public key value

* Use AuthenticatedConnect for dial

* Fix gobind-pinecone

* Stop panics

* Give fsAPI to keyserver

* Pinecone demo fixes

* Update gobind build scripts

* Account creation

* Tweaks

* Setup tweaks

* API tweaks

* API tweaks

* API tweaks

* Port mutex

* Re-enable multicast

* Add ReadCopy

* Update quic-go, fixes

* Shutdowns fixed for iOS

* Update build script

* Add WebSocket support

* Bug fixes

* Netconn context

* Fix WebSocket connectivity

* Fixes to gobind API

* Strip frameworks

* Configurability updates

* Update go.mod

* Update go.mod/go.sum

* Update go.mod/go.sum

* Update go.mod/go.sum

* Try to stay connected tto static peer

* Update gobind-pinecone

* Update go.mod/go.sum

* Test uTP+TLS

* Use HTTP/2

* Don't use HTTP/2

* Update go.mod/go.sum

* Attempt to reconnect to the static peer if it drops

* Stay connected to static peers more stickily

* Retry room directory lookups if they fail

* NewQUIC -> NewSessions

* Storage updates

* Don't return immediately when there's nothing to sync

* Updates

* Try to reconnect to static peer more

* Update go.mod/go.sum

* Require Go 1.14

* Update go.mod/go.sum

* Update go.mod/go.sum
2021-05-06 12:00:42 +01:00
Neil Alexander 5ce1fe80de
State storage refactor (#1839)
* Hash-deduplicated state storage (and migrations) for PostgreSQL and SQLite

* Refactor droomserver database setup for migrations

* Fix conflict statements

* Update migration names

* Set a boundary for old to new block/snapshot IDs so we don't rewrite them more than once accidentally

* Create sequence if not exists

* Fix boundary queries

* Fix boundary queries

* Use Query

* Break out queries a bit

* More sequence tweaks

* Query parameters are not playing the game

* Injection escaping may not work for CREATE SEQUENCE after all

* Fix snapshot sequence name

* Use boundaried IDs in SQLite too

* Use IFNULL for SQLite

* Use COALESCE in PostgreSQL

* Review comments @Kegsay
2021-04-26 13:25:57 +01:00
Kegsay af41f6d454
Add Sentry support (#1803)
* Add Sentry support

* Use HTTP Sentry properly maybe

* Capture panics

* Log fed Sentry stuff correctly

* British english linter
2021-03-24 10:25:24 +00:00
Neil Alexander 25dc99f1b0
Upgrade dependencies (#1779) 2021-03-03 11:08:41 +00:00
Kegsay 586cc5be5e
Use new path for MSC2946 (#1778)
* Use new path for MSC2946

* Update GMSL
2021-03-02 16:41:28 +00:00
Neil Alexander 3069079e37
Revert quic-go dependency update 2021-02-18 10:18:14 +00:00
Neil Alexander db637515a5
Update libp2p dependencies 2021-02-18 10:14:24 +00:00
Neil Alexander 11830fd4af
Fix transitive dependency for P2P demo 2021-02-18 09:45:55 +00:00
Neil Alexander 5d74a1757f
Don't query for servers so often in /send (#1766)
* Look up servers less often, don't hit API for missing auth events unless there are actually missing auth events

* Remove ResolveConflictsAdhoc (since it is already in GMSL), other tweaks

* Update gomatrixserverlib to matrix-org/gomatrixserverlib#254

* Fix resolve-state

* Initialise t.servers on first use
2021-02-16 17:12:17 +00:00
Neil Alexander 369d3939fd
Drop state events we can't auth instead of failing altogether 2021-01-29 16:33:59 +00:00
Neil Alexander d56a5a6509
Fix well-known lookups 2021-01-28 13:25:45 +00:00
Neil Alexander 6757b67a32
NewClient and NewFederationClient updates (#1730)
* Use matrix-org/gomatrixserverlib#252

* Add missing WithSkipVerify to test

* Functions instead

* Update gomatrixserverlib to matrix-org/gomatrixserverlib#252

* Fix disabling TLS validation
2021-01-22 16:09:05 +00:00
Matthew Hodgson 0571d395b5
Peeking over federation via MSC2444 (#1391)
* a very very WIP first cut of peeking via MSC2753.

doesn't yet compile or work.
needs to actually add the peeking block into the sync response.
checking in now before it gets any bigger, and to gather any initial feedback on the vague shape of it.

* make PeekingDeviceSet private

* add server_name param

* blind stab at adding a `peek` section to /sync

* make it build

* make it launch

* add peeking to getResponseWithPDUsForCompleteSync

* cancel any peeks when we join a room

* spell out how to runoutside of docker if you want speed

* fix SQL

* remove unnecessary txn for SelectPeeks

* fix s/join/peek/ cargocult fail

* HACK: Track goroutine IDs to determine when we write by the wrong thread

To use: set `DENDRITE_TRACE_SQL=1` then grep for `unsafe`

* Track partition offsets and only log unsafe for non-selects

* Put redactions in the writer goroutine

* Update filters on writer goroutine

* wrap peek storage in goid hack

* use exclusive writer, and MarkPeeksAsOld more efficiently

* don't log ascii in binary at sql trace...

* strip out empty roomd deltas

* re-add txn to SelectPeeks

* re-add accidentally deleted field

* reject peeks for non-worldreadable rooms

* move perform_peek

* fix package

* correctly refactor perform_peek

* WIP of implementing MSC2444

* typo

* Revert "Merge branch 'kegan/HACK-goid-sqlite-db-is-locked' into matthew/peeking"

This reverts commit 3cebd8dbfbccdf82b7930b7b6eda92095ca6ef41, reversing
changes made to ed4b3a58a7855acc43530693cc855b439edf9c7c.

* (almost) make it build

* clean up bad merge

* support SendEventWithState with optional event

* fix build & lint

* fix build & lint

* reinstate federated peeks in the roomserver (doh)

* fix sql thinko

* todo for authenticating state returned by /peek

* support returning current state from QueryStateAndAuthChain

* handle SS /peek

* reimplement SS /peek to prod the RS to tell the FS about the peek

* rename RemotePeeks as OutboundPeeks

* rename remote_peeks_table as outbound_peeks_table

* add perform_handle_remote_peek.go

* flesh out federation doc

* add inbound peeks table and hook it up

* rename ambiguous RemotePeek as InboundPeek

* rename FSAPI's PerformPeek as PerformOutboundPeek

* setup inbound peeks db correctly

* fix api.SendEventWithState with no event

* track latestevent on /peek

* go fmt

* document the peek send stream race better

* fix SendEventWithRewrite not to bail if handed a non-state event

* add fixme

* switch SS /peek to use SendEventWithRewrite

* fix comment

* use reverse topo ordering to find latest extrem

* support postgres for federated peeking

* go fmt

* back out bogus go.mod change

* Fix performOutboundPeekUsingServer

* Fix getAuthChain -> GetAuthChain

* Fix build issues

* Fix build again

* Fix getAuthChain -> GetAuthChain

* Don't repeat outbound peeks for the same room ID to the same servers

* Fix lint

* Don't omitempty to appease sytest

Co-authored-by: Kegan Dougal <kegan@matrix.org>
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2021-01-22 14:55:08 +00:00
Neil Alexander 805a74892e
DNS caching (#1728)
* Allow configuring DNS cache

* Update sample configs

* Fix build errors

* Fix time resolution

* Default 5m

* In seconds

* Use WithDNScache

* Correct field name

* Update go.mod/go.sum to matrix-org/gomatrixserverlib#251
2021-01-22 14:16:59 +00:00
Neil Alexander 58f78e08af
Limit well-known response sizes (matrix-org/gomatrixserverlib#250) 2021-01-21 13:57:41 +00:00
Kegsay 80aa9aa8b0
Implement MSC2946 over federation (#1722)
* Add fedsender dep on msc2946

* Add MSC2946Spaces to fsAPI

* Add exclude_rooms impl

* Implement fed spaces handler

* Use stripped state not room version

* Call federated spaces at the right time
2021-01-19 17:14:25 +00:00
Neil Alexander 266f9c4abd
Update to matrix-org/gomatrixserverlib#247 - cache server well-known/DNS addresses 2021-01-13 17:31:13 +00:00
Neil Alexander bad81c028f
Don't recalculate event ID so often in sync (#1624)
* Don't bail so quickly in fetchMissingStateEvents

* Don't recalculate event IDs so often in sync API

* Add comments

* Fix comments

* Update to matrix-org/gomatrixserverlib@eb6a890
2020-12-09 18:07:17 +00:00
Neil Alexander e1d32e2ff1
Update to matrix-org/gomatrixserverlib@232bace 2020-12-07 16:21:55 +00:00
Kegsay b507312d4c
MSC2836 threading: part 2 (#1596)
* Update GMSL

* Add MSC2836EventRelationships to fedsender

* Call MSC2836EventRelationships in reqCtx

* auth remote servers

* Extract room ID and servers from previous events; refactor a bit

* initial cut of federated threading

* Use the right client/fed struct in the response

* Add QueryAuthChain for use with MSC2836

* Add auth chain to federated response

* Fix pointers

* under CI: more logging and enable mscs, nil fix

* Handle direction: up

* Actually send message events to the roomserver..

* Add children and children_hash to unsigned, with tests

* Add logic for exploring threads and tracking children; missing storage functions

* Implement storage functions for children

* Add fetchUnknownEvent

* Do federated hits for include_children if we have unexplored children

* Use /ev_rel rather than /event as the former includes child metadata

* Remove cross-room threading impl

* Enable MSC2836 in the p2p demo

* Namespace mscs db

* Enable msc2836 for ygg

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2020-12-04 14:11:01 +00:00
Neil Alexander 1ce9c52442
Don't recalculate event IDs so often (#1610)
* Don't recalculate event IDs so often

* Revert invite change

* Make sure we're using the right NIDs

* Update gomatrixserverlib

* Update to NewEventFromTrustedJSONWithEventID

* Fix go.mod

* Update gomatrixserverlib to matrix-org/gomatrixserverlib#243

* Use BulkSelectEventID
2020-12-04 10:41:07 +00:00
Neil Alexander 3ef6187e96
Revert "Update quic-go"
This reverts commit 93c3f1a7b3.
2020-12-02 17:03:36 +00:00
Neil Alexander 93c3f1a7b3
Update quic-go 2020-12-02 16:39:41 +00:00
Neil Alexander 417c7d3569
Send client events to appservices (#1603)
* Send client events to appservices

* FormatSync instead of FormatAll
2020-12-02 15:14:12 +00:00
Neil Alexander 20a01bceb2
Pass pointers to events — reloaded (#1583)
* Pass events as pointers

* Fix lint errors

* Update gomatrixserverlib

* Update gomatrixserverlib

* Update to matrix-org/gomatrixserverlib#240
2020-11-16 15:44:53 +00:00
Neil Alexander 2ce2112ddb
Update gjson for safe integer ranges 2020-11-05 09:42:28 +00:00
Neil Alexander 7612f64e3c
Update gomatrixserverlib to matrix-org/gomatrixserverlib#220 2020-10-20 17:23:04 +01:00
Neil Alexander 24e38c4135
Internal HTTP APIs over H2C (#1541)
* H2C on internal HTTP because SCIENCE

* Update comments
2020-10-20 17:13:12 +01:00
Neil Alexander 7ca89ef511
Update gomatrixserverlib 2020-10-20 17:10:37 +01:00
Neil Alexander 3e5d38e284
Improve state resolution v2 performance (matrix-org/gomatrixserverlib#237) 2020-10-15 16:20:19 +01:00
Neil Alexander fe5d1400bf
Update federation timeouts (#1504)
* Update to matrix-org/gomatrixserverlib#234

* Update gomatrixserverlib

* Update federation timeouts

* Fix dendritejs

* Increase /send context time in destination queue
2020-10-09 17:08:32 +01:00
Neil Alexander ee79d662e7
Update to matrix-org/gomatrixserverlib#233 2020-10-06 15:37:31 +01:00