Commit Graph

195 Commits (034f5ae3f1d9c01352a71f4717971f727bae57ee)

Author SHA1 Message Date
J. Ryan Stinnett 034f5ae3f1
Add startup testing for Wasm Pinecone build (#1910)
* Only include go-sqlite3 on the relevant binaries

* The driver name is always sqlite3 now

* Update to matrix-org/go-sqlite3-js@e537baa

* Add initial Wasm test harness

* Upgrade go-sqlite3-js

This fixes an error about semicolons in single statements.

* Add browser-like WebSocket API for testing

* Upgrade go-sqlite3-js

This upgrade includes printing panic messages next to stacks.

* Run for all PRs targeting any branch

* Use manual Node caching

* Temporarily run for all pushes

* Use npm ci instead of install

* Use HTTPS auth for repo packages

* Match path style from build.sh

* update utp

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2021-07-20 12:14:58 +01:00
Neil Alexander f63068df3b
Only include go-sqlite3 on the relevant binaries (#1900)
* Only include go-sqlite3 on the relevant binaries

* The driver name is always sqlite3 now

* Update to matrix-org/go-sqlite3-js@e537baa
2021-07-20 11:18:14 +01:00
Neil Alexander 628e4cc709
Update to matrix-org/gomatrixserverlib#269 2021-07-19 15:09:01 +01:00
Neil Alexander 1827dd7c09
Update dendrite-demo-yggdrasil to Yggdrasil 0.4 (#1921)
* Update Yggdrasil P2P demo for Yggdrasil v0.4

* Build fixes
2021-07-15 12:16:44 +01:00
kegsay e8de96b78c
sytests: fix failing PL tests by updating GMSL (#1920) 2021-07-14 15:48:14 +01:00
Kegan Dougal 93cf1e7590 Actually bump GMSL 2021-07-12 18:58:25 +01:00
kegsay 1ed732cc78
Implement /_synapse/admin/v1/register (#1911)
* Implement /_synapse/admin/v1/register

This is implemented identically to Synapse, so scripts which work
with Synapse should work with Dendrite.

```
    Test 27 POST /_synapse/admin/v1/register with shared secret... OK
    Test 28 POST /_synapse/admin/v1/register admin with shared secret... OK
    Test 29 POST /_synapse/admin/v1/register with shared secret downcases capitals... OK
    Test 30 POST /_synapse/admin/v1/register with shared secret disallows symbols... OK
```

Sytest however has `implementation_specific => "synapse"` which stops these
tests from running.

* Add missing muxes to gobind

* Linting
2021-07-09 16:52:31 +01:00
kegsay d72d634391
Add dendrite-upgrade-test (#1901)
* Add WIP binary for testing dendrite version upgrades

* WIP dendrite upgrade work

* Finish dendrite upgrade checks

* go mod tidy

* Review comments; print container logs on failure

* Linting
2021-07-07 12:06:17 +01:00
Neil Alexander 355ce7ed30
Update to matrix-org/gomatrixserverlib#266 (+ missing file) 2021-07-02 16:30:37 +01:00
Neil Alexander 6bb994bcfa
Update to matrix-org/gomatrixserverlib#265 for faster power level content parsing 2021-07-02 16:13:56 +01:00
Neil Alexander bd7b0b1787
Update go.mod/go.sum 2021-06-25 15:13:21 +01:00
Neil Alexander 097a5833c8
dendritejs-pinecone 2021-06-25 14:20:36 +01:00
Neil Alexander 6ae976a55a
dendrite-demo-pinecone: Accept any origin 2021-06-22 15:39:29 +01:00
Neil Alexander 3a77bca28b
Revert "Ensure user IDs match the spec (matrix-org/gomatrixserverlib#261)"
This reverts commit 30e9353fdd.
2021-06-16 09:07:41 +01:00
Neil Alexander 30e9353fdd
Ensure user IDs match the spec (matrix-org/gomatrixserverlib#261) 2021-06-15 15:18:56 +01:00
Neil Alexander 5ec7d3a3ec
Update go.mod/go.sum 2021-06-14 15:04:05 +01:00
Neil Alexander 051cd63f1e
Update go.mod/go.sum 2021-06-14 13:26:02 +01:00
Neil Alexander bd9dec8e06
Pinecone demo updates 2021-06-14 13:13:07 +01:00
Neil Alexander b0aa101dcd
Update go.mod/go.sum for matrix-org/pinecone (Build 79) 2021-06-02 12:23:01 +01:00
Neil Alexander 3797d818c0
Update gomatrixserverlib to matrix-org/gomatrixserverlib#259 2021-05-25 12:01:07 +01:00
Neil Alexander 3f84cd6cd0
Update go.mod/go.sum 2021-05-25 09:52:55 +01:00
Neil Alexander 30f021700a
Update go.mod/go.sum 2021-05-25 09:49:02 +01:00
Neil Alexander a7f2845a6a
Demo tweaks 2021-05-24 13:12:05 +01:00
Neil Alexander d2d2164025
Update pinecone demo 2021-05-24 11:17:32 +01:00
Neil Alexander 2948ffd782
Update go.mod/go.sum 2021-05-18 16:20:04 +01:00
Neil Alexander 08a4370c94
Update go.mod/go.sum 2021-05-18 15:58:50 +01:00
Neil Alexander 140cae81cc
Update dendrite-demo-pinecone 2021-05-18 15:47:15 +01:00
Neil Alexander 1d89c4a3cd
Update go.mod/go.sum 2021-05-10 17:04:06 +01:00
Neil Alexander c76f820c86
Update go.mod/go.sum 2021-05-10 17:03:02 +01:00
Neil Alexander a49d06138e
Updates to dendrite-demo-pinecone 2021-05-10 16:59:03 +01:00
Neil Alexander 603bf590f0
Fix public room directory in Pinecone demo 2021-05-07 12:17:14 +01:00
Neil Alexander aa672068ab
Update pinecone in go.mod/go.sum 2021-05-07 11:49:44 +01:00
Neil Alexander 1002e87b60
Pinecone P2P demo (#1856)
* Pinecone demo

* Enable multicast, fix HTTP routing

* Fix multicast import

* Fix build

* Update Pinecone demo

* Fix the keys

* Tweaks

* Pinecone room directory support (early)

* Fix gobind-pinecone

* Add pinecone listener

* Fix public key value

* Use AuthenticatedConnect for dial

* Fix gobind-pinecone

* Stop panics

* Give fsAPI to keyserver

* Pinecone demo fixes

* Update gobind build scripts

* Account creation

* Tweaks

* Setup tweaks

* API tweaks

* API tweaks

* API tweaks

* Port mutex

* Re-enable multicast

* Add ReadCopy

* Update quic-go, fixes

* Shutdowns fixed for iOS

* Update build script

* Add WebSocket support

* Bug fixes

* Netconn context

* Fix WebSocket connectivity

* Fixes to gobind API

* Strip frameworks

* Configurability updates

* Update go.mod

* Update go.mod/go.sum

* Update go.mod/go.sum

* Update go.mod/go.sum

* Try to stay connected tto static peer

* Update gobind-pinecone

* Update go.mod/go.sum

* Test uTP+TLS

* Use HTTP/2

* Don't use HTTP/2

* Update go.mod/go.sum

* Attempt to reconnect to the static peer if it drops

* Stay connected to static peers more stickily

* Retry room directory lookups if they fail

* NewQUIC -> NewSessions

* Storage updates

* Don't return immediately when there's nothing to sync

* Updates

* Try to reconnect to static peer more

* Update go.mod/go.sum

* Require Go 1.14

* Update go.mod/go.sum

* Update go.mod/go.sum
2021-05-06 12:00:42 +01:00
Neil Alexander 5ce1fe80de
State storage refactor (#1839)
* Hash-deduplicated state storage (and migrations) for PostgreSQL and SQLite

* Refactor droomserver database setup for migrations

* Fix conflict statements

* Update migration names

* Set a boundary for old to new block/snapshot IDs so we don't rewrite them more than once accidentally

* Create sequence if not exists

* Fix boundary queries

* Fix boundary queries

* Use Query

* Break out queries a bit

* More sequence tweaks

* Query parameters are not playing the game

* Injection escaping may not work for CREATE SEQUENCE after all

* Fix snapshot sequence name

* Use boundaried IDs in SQLite too

* Use IFNULL for SQLite

* Use COALESCE in PostgreSQL

* Review comments @Kegsay
2021-04-26 13:25:57 +01:00
Kegsay af41f6d454
Add Sentry support (#1803)
* Add Sentry support

* Use HTTP Sentry properly maybe

* Capture panics

* Log fed Sentry stuff correctly

* British english linter
2021-03-24 10:25:24 +00:00
Neil Alexander 25dc99f1b0
Upgrade dependencies (#1779) 2021-03-03 11:08:41 +00:00
Kegsay 586cc5be5e
Use new path for MSC2946 (#1778)
* Use new path for MSC2946

* Update GMSL
2021-03-02 16:41:28 +00:00
Neil Alexander 3069079e37
Revert quic-go dependency update 2021-02-18 10:18:14 +00:00
Neil Alexander db637515a5
Update libp2p dependencies 2021-02-18 10:14:24 +00:00
Neil Alexander 11830fd4af
Fix transitive dependency for P2P demo 2021-02-18 09:45:55 +00:00
Neil Alexander 5d74a1757f
Don't query for servers so often in /send (#1766)
* Look up servers less often, don't hit API for missing auth events unless there are actually missing auth events

* Remove ResolveConflictsAdhoc (since it is already in GMSL), other tweaks

* Update gomatrixserverlib to matrix-org/gomatrixserverlib#254

* Fix resolve-state

* Initialise t.servers on first use
2021-02-16 17:12:17 +00:00
Neil Alexander 369d3939fd
Drop state events we can't auth instead of failing altogether 2021-01-29 16:33:59 +00:00
Neil Alexander d56a5a6509
Fix well-known lookups 2021-01-28 13:25:45 +00:00
Neil Alexander 6757b67a32
NewClient and NewFederationClient updates (#1730)
* Use matrix-org/gomatrixserverlib#252

* Add missing WithSkipVerify to test

* Functions instead

* Update gomatrixserverlib to matrix-org/gomatrixserverlib#252

* Fix disabling TLS validation
2021-01-22 16:09:05 +00:00
Matthew Hodgson 0571d395b5
Peeking over federation via MSC2444 (#1391)
* a very very WIP first cut of peeking via MSC2753.

doesn't yet compile or work.
needs to actually add the peeking block into the sync response.
checking in now before it gets any bigger, and to gather any initial feedback on the vague shape of it.

* make PeekingDeviceSet private

* add server_name param

* blind stab at adding a `peek` section to /sync

* make it build

* make it launch

* add peeking to getResponseWithPDUsForCompleteSync

* cancel any peeks when we join a room

* spell out how to runoutside of docker if you want speed

* fix SQL

* remove unnecessary txn for SelectPeeks

* fix s/join/peek/ cargocult fail

* HACK: Track goroutine IDs to determine when we write by the wrong thread

To use: set `DENDRITE_TRACE_SQL=1` then grep for `unsafe`

* Track partition offsets and only log unsafe for non-selects

* Put redactions in the writer goroutine

* Update filters on writer goroutine

* wrap peek storage in goid hack

* use exclusive writer, and MarkPeeksAsOld more efficiently

* don't log ascii in binary at sql trace...

* strip out empty roomd deltas

* re-add txn to SelectPeeks

* re-add accidentally deleted field

* reject peeks for non-worldreadable rooms

* move perform_peek

* fix package

* correctly refactor perform_peek

* WIP of implementing MSC2444

* typo

* Revert "Merge branch 'kegan/HACK-goid-sqlite-db-is-locked' into matthew/peeking"

This reverts commit 3cebd8dbfbccdf82b7930b7b6eda92095ca6ef41, reversing
changes made to ed4b3a58a7855acc43530693cc855b439edf9c7c.

* (almost) make it build

* clean up bad merge

* support SendEventWithState with optional event

* fix build & lint

* fix build & lint

* reinstate federated peeks in the roomserver (doh)

* fix sql thinko

* todo for authenticating state returned by /peek

* support returning current state from QueryStateAndAuthChain

* handle SS /peek

* reimplement SS /peek to prod the RS to tell the FS about the peek

* rename RemotePeeks as OutboundPeeks

* rename remote_peeks_table as outbound_peeks_table

* add perform_handle_remote_peek.go

* flesh out federation doc

* add inbound peeks table and hook it up

* rename ambiguous RemotePeek as InboundPeek

* rename FSAPI's PerformPeek as PerformOutboundPeek

* setup inbound peeks db correctly

* fix api.SendEventWithState with no event

* track latestevent on /peek

* go fmt

* document the peek send stream race better

* fix SendEventWithRewrite not to bail if handed a non-state event

* add fixme

* switch SS /peek to use SendEventWithRewrite

* fix comment

* use reverse topo ordering to find latest extrem

* support postgres for federated peeking

* go fmt

* back out bogus go.mod change

* Fix performOutboundPeekUsingServer

* Fix getAuthChain -> GetAuthChain

* Fix build issues

* Fix build again

* Fix getAuthChain -> GetAuthChain

* Don't repeat outbound peeks for the same room ID to the same servers

* Fix lint

* Don't omitempty to appease sytest

Co-authored-by: Kegan Dougal <kegan@matrix.org>
Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2021-01-22 14:55:08 +00:00
Neil Alexander 805a74892e
DNS caching (#1728)
* Allow configuring DNS cache

* Update sample configs

* Fix build errors

* Fix time resolution

* Default 5m

* In seconds

* Use WithDNScache

* Correct field name

* Update go.mod/go.sum to matrix-org/gomatrixserverlib#251
2021-01-22 14:16:59 +00:00
Neil Alexander 58f78e08af
Limit well-known response sizes (matrix-org/gomatrixserverlib#250) 2021-01-21 13:57:41 +00:00
Kegsay 80aa9aa8b0
Implement MSC2946 over federation (#1722)
* Add fedsender dep on msc2946

* Add MSC2946Spaces to fsAPI

* Add exclude_rooms impl

* Implement fed spaces handler

* Use stripped state not room version

* Call federated spaces at the right time
2021-01-19 17:14:25 +00:00
Neil Alexander 266f9c4abd
Update to matrix-org/gomatrixserverlib#247 - cache server well-known/DNS addresses 2021-01-13 17:31:13 +00:00
Neil Alexander bad81c028f
Don't recalculate event ID so often in sync (#1624)
* Don't bail so quickly in fetchMissingStateEvents

* Don't recalculate event IDs so often in sync API

* Add comments

* Fix comments

* Update to matrix-org/gomatrixserverlib@eb6a890
2020-12-09 18:07:17 +00:00