Commit graph

188 commits

Author SHA1 Message Date
kegsay
bb6e4487dd
dendrite-upgrade-test: support --from HEAD-N, cleanup on startup (#1903) 2021-07-07 14:20:07 +01:00
kegsay
d72d634391
Add dendrite-upgrade-test (#1901)
* Add WIP binary for testing dendrite version upgrades

* WIP dendrite upgrade work

* Finish dendrite upgrade checks

* go mod tidy

* Review comments; print container logs on failure

* Linting
2021-07-07 12:06:17 +01:00
Neil Alexander
b7a2d369c0
Change how servers are selected for missing auth/prev events (#1892)
* Change how servers are selected for missing auth/prev events

* Shuffle order

* Move ServersInRoomProvider into api package
2021-06-30 12:05:58 +01:00
Neil Alexander
a6f7e83596
Fix bugs in P2P demos 2021-06-25 15:59:59 +01:00
Neil Alexander
097a5833c8
dendritejs-pinecone 2021-06-25 14:20:36 +01:00
Neil Alexander
6ae976a55a
dendrite-demo-pinecone: Accept any origin 2021-06-22 15:39:29 +01:00
S7evinK
183f43d38d
Add parameters to specify password (#1868)
* Add parameters to specify password

Signed-off-by: Till Faelligen <tfaelligen@gmail.com>

* Fix typo

* Add testdata

* Use go1.13 compatible way to read files
2021-06-14 14:08:29 +01:00
Neil Alexander
bd9dec8e06
Pinecone demo updates 2021-06-14 13:13:07 +01:00
Neil Alexander
a7f2845a6a
Demo tweaks 2021-05-24 13:12:05 +01:00
Neil Alexander
79c5485c8d
Allow clearing federation blacklist at startup for P2P demos 2021-05-24 11:43:24 +01:00
Neil Alexander
d2d2164025
Update pinecone demo 2021-05-24 11:17:32 +01:00
Neil Alexander
140cae81cc
Update dendrite-demo-pinecone 2021-05-18 15:47:15 +01:00
Neil Alexander
a49d06138e
Updates to dendrite-demo-pinecone 2021-05-10 16:59:03 +01:00
Neil Alexander
81d60d5448
Update room directory in Pinecone demo some more 2021-05-07 12:48:30 +01:00
Neil Alexander
603bf590f0
Fix public room directory in Pinecone demo 2021-05-07 12:17:14 +01:00
Neil Alexander
1002e87b60
Pinecone P2P demo (#1856)
* Pinecone demo

* Enable multicast, fix HTTP routing

* Fix multicast import

* Fix build

* Update Pinecone demo

* Fix the keys

* Tweaks

* Pinecone room directory support (early)

* Fix gobind-pinecone

* Add pinecone listener

* Fix public key value

* Use AuthenticatedConnect for dial

* Fix gobind-pinecone

* Stop panics

* Give fsAPI to keyserver

* Pinecone demo fixes

* Update gobind build scripts

* Account creation

* Tweaks

* Setup tweaks

* API tweaks

* API tweaks

* API tweaks

* Port mutex

* Re-enable multicast

* Add ReadCopy

* Update quic-go, fixes

* Shutdowns fixed for iOS

* Update build script

* Add WebSocket support

* Bug fixes

* Netconn context

* Fix WebSocket connectivity

* Fixes to gobind API

* Strip frameworks

* Configurability updates

* Update go.mod

* Update go.mod/go.sum

* Update go.mod/go.sum

* Update go.mod/go.sum

* Try to stay connected tto static peer

* Update gobind-pinecone

* Update go.mod/go.sum

* Test uTP+TLS

* Use HTTP/2

* Don't use HTTP/2

* Update go.mod/go.sum

* Attempt to reconnect to the static peer if it drops

* Stay connected to static peers more stickily

* Retry room directory lookups if they fail

* NewQUIC -> NewSessions

* Storage updates

* Don't return immediately when there's nothing to sync

* Updates

* Try to reconnect to static peer more

* Update go.mod/go.sum

* Require Go 1.14

* Update go.mod/go.sum

* Update go.mod/go.sum
2021-05-06 12:00:42 +01:00
Bruce MacDonald
d27607af78
Implement OpenID module (#599) (#1812)
* Implement OpenID module (#599)

- Unrelated: change Riot references to Element in client API routing

Signed-off-by: Bruce MacDonald <contact@bruce-macdonald.com>

* OpenID module tweaks (#599)

- specify expiry is ms rather than vague ts
- add OpenID token lifetime to configuration
- use Go naming conventions for the path params
- store plaintext token rather than hash
- remove openid table sqllite mutex

* Add default OpenID token lifetime (#599)

* Update dendrite-config.yaml

Co-authored-by: Kegsay <kegsay@gmail.com>
Co-authored-by: Kegsay <kegan@matrix.org>
2021-04-07 13:26:20 +01:00
Kegsay
850abb1dde
Make bcrypt cost configurable (#1793) 2021-03-08 13:19:02 +00:00
Neil Alexander
1ad96e2e2d
Tweak AS registration check and AS component HTTP clients (#1785)
* Tweak AS registration check

* Check appservice usernames using correct function

* Update sytest-whitelist

* Use gomatrixserverlib.Client since that allows us to disable TLS validation using the config

* Add appservice-specific client and ability to control TLS validation for appservices only

* Set timeout on appservice client

* Review comments

* Remove dead code

* Enforce LoginTypeApplicationService after all

* Check correct auth type field
2021-03-05 10:40:27 +00:00
Neil Alexander
d15836e260
Increase gocyclo complexity to 25 (and remove all but 2 golint directives related to it) (#1783) 2021-03-03 14:35:57 +00:00
Neil Alexander
5d74a1757f
Don't query for servers so often in /send (#1766)
* Look up servers less often, don't hit API for missing auth events unless there are actually missing auth events

* Remove ResolveConflictsAdhoc (since it is already in GMSL), other tweaks

* Update gomatrixserverlib to matrix-org/gomatrixserverlib#254

* Fix resolve-state

* Initialise t.servers on first use
2021-02-16 17:12:17 +00:00
Neil Alexander
9f443317bc
Graceful shutdowns (#1734)
* Initial graceful stop

* Fix dendritejs

* Use process context for outbound federation requests in destination queues

* Reduce logging

* Fix log level
2021-01-26 12:56:20 +00:00
Neil Alexander
6757b67a32
NewClient and NewFederationClient updates (#1730)
* Use matrix-org/gomatrixserverlib#252

* Add missing WithSkipVerify to test

* Functions instead

* Update gomatrixserverlib to matrix-org/gomatrixserverlib#252

* Fix disabling TLS validation
2021-01-22 16:09:05 +00:00
Kegsay
93942f8ab6
Gate peeking behind msc flags (#1731) 2021-01-22 16:08:47 +00:00
Neil Alexander
6dadb1c06b
Fix lint error in generate-keys 2021-01-18 09:34:49 +00:00
Kegan Dougal
2113da5332 Enable MSC 2946 in CI 2021-01-15 17:56:50 +00:00
Neil Alexander
fac71edc62
Fix #1655 by re-adding the appservice alias query (#1660) 2020-12-18 13:33:28 +00:00
Kegsay
a518e2971a
Fix flakey complement (#1653) 2020-12-17 18:25:51 +00:00
Kegsay
b507312d4c
MSC2836 threading: part 2 (#1596)
* Update GMSL

* Add MSC2836EventRelationships to fedsender

* Call MSC2836EventRelationships in reqCtx

* auth remote servers

* Extract room ID and servers from previous events; refactor a bit

* initial cut of federated threading

* Use the right client/fed struct in the response

* Add QueryAuthChain for use with MSC2836

* Add auth chain to federated response

* Fix pointers

* under CI: more logging and enable mscs, nil fix

* Handle direction: up

* Actually send message events to the roomserver..

* Add children and children_hash to unsigned, with tests

* Add logic for exploring threads and tracking children; missing storage functions

* Implement storage functions for children

* Add fetchUnknownEvent

* Do federated hits for include_children if we have unexplored children

* Use /ev_rel rather than /event as the former includes child metadata

* Remove cross-room threading impl

* Enable MSC2836 in the p2p demo

* Namespace mscs db

* Enable msc2836 for ygg

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2020-12-04 14:11:01 +00:00
Neil Alexander
ec7a0e42ae
Simplify create-account (#1608) 2020-12-03 10:55:17 +00:00
Neil Alexander
b5aa7ca3ab
Top-level setup package (#1605)
* Move config, setup, mscs into "setup" top-level folder

* oops, forgot the EDU server

* Add setup

* goimports
2020-12-02 17:41:00 +00:00
Neil Alexander
c16abb089c
Give fsAPI to keyserver in polylith/gobind 2020-12-01 10:52:02 +00:00
Kegsay
6353b0b7e4
MSC2836: Threading - part one (#1589)
* Add mscs/hooks package, begin work for msc2836

* Flesh out hooks and add SQL schema

* Begin implementing core msc2836 logic

* Add test harness

* Linting

* Implement visibility checks; stub out APIs for tests

* Flesh out testing

* Flesh out walkThread a bit

* Persist the origin_server_ts as well

* Edges table instead of relationships

* Add nodes table for event metadata

* LEFT JOIN to extract origin_server_ts for children

* Add graph walking structs

* Implement walking algorithm

* Add more graph walking tests

* Add auto_join for local rooms

* Fix create table syntax on postgres

* Add relationship_room_id|servers to the unsigned section of events

* Persist the parent room_id/servers in edge metadata

Other events cannot assert the true room_id/servers for the
parent event, only make claims to them, hence why this is
edge metadata.

* guts to pass through room_id/servers

* Refactor msc2836 to allow handling from federation

* Add JoinedVia to PerformJoin responses

* Fix tests; review comments
2020-11-19 11:34:59 +00:00
Neil Alexander
20a01bceb2
Pass pointers to events — reloaded (#1583)
* Pass events as pointers

* Fix lint errors

* Update gomatrixserverlib

* Update gomatrixserverlib

* Update to matrix-org/gomatrixserverlib#240
2020-11-16 15:44:53 +00:00
Neil Alexander
534f9a9eb6
Refactor forward extremities (#1556)
* Add resolve-state helper

* Tweaks

* Refactor forward extremities, again

* Tweaks

* Minor optimisation

* Make path a bit clearer

* Only process state/membership if forward extremities have changed

* Usage comments in resolve-state
2020-10-21 15:37:07 +01:00
Neil Alexander
39c7a8915c
Multi-personality polylith binary (#1552)
* Initial work oon multipersonality binary

* Remove old binaries

* Monolith and polylith binaries

* Better logging

* dendrite-poly-multi

* Fix path

* Copyright notices etc

* Tweaks

* Update Docker, INSTALL.md

* Take first argument if flags package doesn't find any args

* Postgres 9.6 or later, fix some more Docker stuff

* Don't create unnecessary e2ekey DB

* Run go mod tidy
2020-10-20 16:11:24 +01:00
Pika
53a745f333
fix create-account (#1546) 2020-10-20 11:47:37 +01:00
Kegsay
4a7fb9c045
Automatically upgrade databases on startup (#1529)
* Support auto-upgrading accounts DB

* Auto-upgrade device DB deltas

* Support up/downgrading from cmd/goose

* Linting

* Create tables then do migrations then prepare statements

To avoid failing due to some things not existing

* Linting
2020-10-15 18:09:41 +01:00
Neil Alexander
49abe359e6
Start Kafka connections for each component that needs them (#1527)
* Start Kafka connection for each component that needs one

* Fix roomserver unit tests

* Rename to naffkaInstance (@Kegsay review comment)

* Fix import cycle
2020-10-15 13:27:13 +01:00
Neil Alexander
fe5d1400bf
Update federation timeouts (#1504)
* Update to matrix-org/gomatrixserverlib#234

* Update gomatrixserverlib

* Update federation timeouts

* Fix dendritejs

* Increase /send context time in destination queue
2020-10-09 17:08:32 +01:00
S7evinK
1cd525ef0d
Extend device_devices table (#1471)
* Add last_used_ts and IP to database

* Add migrations

* Rename column
Prepare statements

* Add interface method and implement it

Signed-off-by: Till Faelligen <tfaelligen@gmail.com>

* Rename struct fields

* Add user_agent to database

* Add userAgent to registration calls

* Add missing "IF NOT EXISTS"

* use txn writer

* Add UserAgent to Device

Co-authored-by: Kegsay <kegan@matrix.org>
2020-10-09 09:17:23 +01:00
Neil Alexander
8bca7a83a9
Update monolith -api behaviour (#1484)
* Update monolith -api mode listeners

* Fix check

* Fix another check

* Update HTTP API addr behaviour

* Redefine NoExternalListener

* NoListener
2020-10-07 16:59:22 +01:00
Kegsay
bf7e85848b
Rename serverkeyapi to signingkeyserver (#1492)
* Rename serverkeyapi to signingkeyserver

We use "api" for public facing stuff and "server" for internal stuff.
As the server key API is internal only, we call it 'signing key server',
which also clarifies the type of key (as opposed to TLS keys, E2E keys, etc)

* Convert docker/scripts to use signing-key-server

* Rename missed bits
2020-10-07 16:23:18 +01:00
Neil Alexander
4feff8e8d9
Don't give up if we fail to fetch a key (#1483)
* Don't give up if we fail to fetch a key

* Fix logging line

* furl nolint
2020-10-06 17:59:08 +01:00
Neil Alexander
d69eba10e5
Add furl (#1482)
* Add furl

* Add POST support
2020-10-06 14:51:32 +01:00
Neil Alexander
8fb74fe99a
Yggdrasil demo tweaks 2020-10-06 10:37:52 +01:00
S7evinK
2bfab5f58b
Add example goose sqlite3 migration (#1474)
Signed-off-by: Till Faelligen <tfaelligen@gmail.com>
2020-10-05 11:28:10 +01:00
Neil Alexander
85c43e1bef
Don't drop federation retries for Yggdrasil demos 2020-10-02 15:14:01 +01:00
Neil Alexander
b1d5360335
Update HTTP clients (#1457)
* Update gomatrixserverlib

* Use separate HTTP client for API calls, set User-Agent for outbound HTTP requests
2020-10-01 11:55:17 +01:00
Kegsay
18231f25b4
Implement rejected events (#1426)
* WIP Event rejection

* Still send back errors for rejected events

Instead, discard them at the federationapi /send layer rather than
re-implementing checks at the clientapi/PerformJoin layer.

* Implement rejected events

Critically, rejected events CAN cause state resolution to happen
as it can merge forks in the DAG. This is fine, _provided_ we
do not add the rejected event when performing state resolution,
which is what this PR does. It also fixes the error handling
when NotAllowed happens, as we were checking too early and needlessly
handling NotAllowed in more than one place.

* Update test to match reality

* Modify InputRoomEvents to no longer return an error

Errors do not serialise across HTTP boundaries in polylith mode,
so instead set fields on the InputRoomEventsResponse. Add `Err()`
function to make the API shape basically the same.

* Remove redundant returns; linting

* Update blacklist
2020-09-16 13:00:52 +01:00