Commit Graph

1489 Commits (8b22c4270da28bb6e968f0804da8e2dc4a865651)

Author SHA1 Message Date
S7evinK 8b22c4270d
Use LimitReader to prevent DoS risk (#1843)
* Use LimitReader to prevent DoS risk

Signed-off-by: Till Faelligen <tfaelligen@gmail.com>

* Check if bytesWritten is equal to the maxFileSize
Add tests

Signed-off-by: Till Faelligen <tfaelligen@gmail.com>

* Use oldschool defer to cleanup after the tests

* Let LimitReader read MaxFileSizeBytes + 1

Co-authored-by: Kegsay <kegan@matrix.org>
2021-06-07 09:17:20 +01:00
bodqhrohro c488d3db75
Fix SIGSEGV in IsInterestedInRoomID (#1846)
* Avoid crash on non-compiled room regex

Roughly fixes #1845 (actual compiling still needed)

Signed-off-by: Bohdan Horbeshko <bodqhrohro@gmail.com>

* Compile regexes for all namespaces

Deadheres the regex compiling from building larger regexes for possibly
exclusive namespaces only. A complete fix for #1845, so regexes for
rooms namespaces and other non-whitelisted namespaces can be used
more safely.

Signed-off-by: Bohdan Horbeshko <bodqhrohro@gmail.com>

* Appservice config: handle regexp parsing errors

Signed-off-by: diamondburned <datutbrus@gmail.com>
Signed-off-by: Bohdan Horbeshko <bodqhrohro@gmail.com>

Co-authored-by: Kegsay <kegan@matrix.org>
2021-06-07 09:13:40 +01:00
bodqhrohro f18001ecbb
Split the select+update query for txn_id counter (#1855)
The update part wasn't executed actually for SQLite, so it is moved to
a separate statement.

Fixes #1852.

Signed-off-by: Bohdan Horbeshko <bodqhrohro@gmail.com>
2021-06-07 08:51:19 +01:00
Neil Alexander b0aa101dcd
Update go.mod/go.sum for matrix-org/pinecone (Build 79) 2021-06-02 12:23:01 +01:00
Neil Alexander 3797d818c0
Update gomatrixserverlib to matrix-org/gomatrixserverlib#259 2021-05-25 12:01:07 +01:00
Neil Alexander 3f84cd6cd0
Update go.mod/go.sum 2021-05-25 09:52:55 +01:00
Neil Alexander 30f021700a
Update go.mod/go.sum 2021-05-25 09:49:02 +01:00
Neil Alexander a7f2845a6a
Demo tweaks 2021-05-24 13:12:05 +01:00
Neil Alexander 79c5485c8d
Allow clearing federation blacklist at startup for P2P demos 2021-05-24 11:43:24 +01:00
Neil Alexander d2d2164025
Update pinecone demo 2021-05-24 11:17:32 +01:00
Neil Alexander 2948ffd782
Update go.mod/go.sum 2021-05-18 16:20:04 +01:00
Neil Alexander 08a4370c94
Update go.mod/go.sum 2021-05-18 15:58:50 +01:00
Neil Alexander e9e0309089
Merge branch 'master' of github.com:matrix-org/dendrite 2021-05-18 15:47:23 +01:00
Neil Alexander 140cae81cc
Update dendrite-demo-pinecone 2021-05-18 15:47:15 +01:00
Michael Telatynski 4691adc8f8
Update MSC2946 implementation for stable spaces (#1859)
Now that MSC1772 passed FCP its identifiers have stabilised
This outright drops support for experimental spaces but that's what you get for being on the bleeding edge
2021-05-18 15:17:37 +01:00
Neil Alexander 1d89c4a3cd
Update go.mod/go.sum 2021-05-10 17:04:06 +01:00
Neil Alexander c76f820c86
Update go.mod/go.sum 2021-05-10 17:03:02 +01:00
Neil Alexander a49d06138e
Updates to dendrite-demo-pinecone 2021-05-10 16:59:03 +01:00
Neil Alexander 81d60d5448
Update room directory in Pinecone demo some more 2021-05-07 12:48:30 +01:00
Neil Alexander 603bf590f0
Fix public room directory in Pinecone demo 2021-05-07 12:17:14 +01:00
Neil Alexander aa672068ab
Update pinecone in go.mod/go.sum 2021-05-07 11:49:44 +01:00
Neil Alexander 1002e87b60
Pinecone P2P demo (#1856)
* Pinecone demo

* Enable multicast, fix HTTP routing

* Fix multicast import

* Fix build

* Update Pinecone demo

* Fix the keys

* Tweaks

* Pinecone room directory support (early)

* Fix gobind-pinecone

* Add pinecone listener

* Fix public key value

* Use AuthenticatedConnect for dial

* Fix gobind-pinecone

* Stop panics

* Give fsAPI to keyserver

* Pinecone demo fixes

* Update gobind build scripts

* Account creation

* Tweaks

* Setup tweaks

* API tweaks

* API tweaks

* API tweaks

* Port mutex

* Re-enable multicast

* Add ReadCopy

* Update quic-go, fixes

* Shutdowns fixed for iOS

* Update build script

* Add WebSocket support

* Bug fixes

* Netconn context

* Fix WebSocket connectivity

* Fixes to gobind API

* Strip frameworks

* Configurability updates

* Update go.mod

* Update go.mod/go.sum

* Update go.mod/go.sum

* Update go.mod/go.sum

* Try to stay connected tto static peer

* Update gobind-pinecone

* Update go.mod/go.sum

* Test uTP+TLS

* Use HTTP/2

* Don't use HTTP/2

* Update go.mod/go.sum

* Attempt to reconnect to the static peer if it drops

* Stay connected to static peers more stickily

* Retry room directory lookups if they fail

* NewQUIC -> NewSessions

* Storage updates

* Don't return immediately when there's nothing to sync

* Updates

* Try to reconnect to static peer more

* Update go.mod/go.sum

* Require Go 1.14

* Update go.mod/go.sum

* Update go.mod/go.sum
2021-05-06 12:00:42 +01:00
Neil Alexander 464b908bd0
Don't return immediately when there's nothing to sync 2021-04-26 16:33:42 +01:00
Neil Alexander c67d8da3eb
Fix bug in SQLite migration 2021-04-26 13:45:47 +01:00
Neil Alexander 5ce1fe80de
State storage refactor (#1839)
* Hash-deduplicated state storage (and migrations) for PostgreSQL and SQLite

* Refactor droomserver database setup for migrations

* Fix conflict statements

* Update migration names

* Set a boundary for old to new block/snapshot IDs so we don't rewrite them more than once accidentally

* Create sequence if not exists

* Fix boundary queries

* Fix boundary queries

* Use Query

* Break out queries a bit

* More sequence tweaks

* Query parameters are not playing the game

* Injection escaping may not work for CREATE SEQUENCE after all

* Fix snapshot sequence name

* Use boundaried IDs in SQLite too

* Use IFNULL for SQLite

* Use COALESCE in PostgreSQL

* Review comments @Kegsay
2021-04-26 13:25:57 +01:00
Fero d6e9b7b307
Remove the 'Content-Type' request header requirement (#1834) 2021-04-19 16:29:51 +01:00
Neil Alexander a9faa1bc44
Fix registration error when disabled 2021-04-15 09:58:26 +01:00
Kegsay 656d11ec90
fedsender: tolerate dupe membership events (#1824)
* fedsender: tolerate dupe membership events

Previously if the fedsender got a duplicate membership event it would cause
the entire process to crash. Now it doesn't. This masks an issue with the
roomserver where it can emit duplicate membership events.

* Update joined_hosts_table.go
2021-04-14 11:11:54 +01:00
Fero 653e30619c
Remove the Content-Length requirement on upload request (#1831)
* Remove the Content-Length requirement

* Make sure that the file size does not exceed max content size

* Address review comment - universally check if temp file size exceeds max file size
2021-04-14 10:53:24 +01:00
Neil Alexander 080ae6a829
Move room mutex in federation API (#1830)
* Move room mutex in federation API to surround resolveStatesAndCheck

* Guard processEventWithMissingState instead

* Revert "Guard processEventWithMissingState instead"

This reverts commit 0ce88036aa79b0ce97e967afb70fe932a547d5f0.
2021-04-13 11:13:07 +01:00
Kegsay e08942fb00
Remove legacy register endpoint (#1822)
* Remove legacy register endpoint

We only support `/r0` CS API paths, not `/v1`.

* Finish removing
2021-04-09 10:21:35 +01:00
Kegsay b769d5a25e
Optimise memory usage when calling /g_m_e (#1819)
* Optimise memory usage when calling /g_m_e

* cache more events

* refactor handling of device list update pokes

* Sigh
2021-04-08 13:50:39 +01:00
Tim McCormack 5ade348d14
Document need for max-body-size change in reverse proxy as well (#1816)
Just changing the Media API's `max_file_size_bytes` isn't enough if
Dendrite is running behind a proxy; document the need for a proxy config
change in the place the admin is most likely to notice it.

Signed-off-by: Tim McCormack <cortex@brainonfire.net>
2021-04-08 12:08:38 +01:00
Bruce MacDonald d27607af78
Implement OpenID module (#599) (#1812)
* Implement OpenID module (#599)

- Unrelated: change Riot references to Element in client API routing

Signed-off-by: Bruce MacDonald <contact@bruce-macdonald.com>

* OpenID module tweaks (#599)

- specify expiry is ms rather than vague ts
- add OpenID token lifetime to configuration
- use Go naming conventions for the path params
- store plaintext token rather than hash
- remove openid table sqllite mutex

* Add default OpenID token lifetime (#599)

* Update dendrite-config.yaml

Co-authored-by: Kegsay <kegsay@gmail.com>
Co-authored-by: Kegsay <kegan@matrix.org>
2021-04-07 13:26:20 +01:00
Kegsay f8d3a762c4
Add a per-room mutex to federationapi when processing transactions (#1810)
* Add a per-room mutex to federationapi when processing transactions

This has numerous benefits:
 - Prevents us doing lots of state resolutions in busy rooms. Previously, room forks would always result
   in a state resolution being performed immediately, without checking if we were already doing this in
   a different transaction. Now they will queue up, resulting in fewer calls to `/state_ids`, `/g_m_e`, etc.
 - Prevents memory usage from growing too large as a result and potentially OOMing.

And costs:
 - High traffic rooms will be slightly slower due to head-of-line blocking from other servers,
   though this has always been an issue as roomserver has a per-room mutex already.

* Fix unit tests

* Correct mutex lock ordering
2021-03-30 10:01:32 +01:00
Eric Eastwood 0ee1c56ffd
Use log directory that we will be able to write to (#1799)
Fix https://github.com/matrix-org/dendrite/issues/1644
2021-03-30 09:53:02 +01:00
Kegsay af41f6d454
Add Sentry support (#1803)
* Add Sentry support

* Use HTTP Sentry properly maybe

* Capture panics

* Log fed Sentry stuff correctly

* British english linter
2021-03-24 10:25:24 +00:00
Kegsay 802f1c96f8
Add more metrics (#1802)
* Add more metrics

* Linting
2021-03-23 15:22:00 +00:00
Kegsay a1b7e4ef3f
log less for failed key querys, add counters for incoming pdus/edus (#1801)
* log less for failed key querys, add counters for incoming pdus/edus

* use labels

* Blacklist flakey test

* Fix metrics
2021-03-23 11:33:36 +00:00
Neil Alexander 01267a34b9
Fix nil pointer crash in QueryMembershipsForRoom 2021-03-17 13:58:04 +00:00
Kegsay 3c419be6af
roomserver: don't make_join with ourselves if clients ask us to (#1797)
* roomserver: don't make_join with ourselves if clients ask us to

* delete properly
2021-03-08 18:16:28 +00:00
Kegsay 77fb981da5
device lists: backoff for longer if the wrong error type is returned (#1796) 2021-03-08 17:45:20 +00:00
Kegan Dougal e865a1507a Make sure the component max open conns doesn't exceed 100 2021-03-08 14:50:37 +00:00
Neil Alexander 5912429d53
Return a more useful error on /register spec compliance violation (#1792) 2021-03-08 13:57:15 +00:00
Neil Alexander 5acf30cd3c
Update sytest-whitelist 2021-03-08 13:32:21 +00:00
Kegsay 850abb1dde
Make bcrypt cost configurable (#1793) 2021-03-08 13:19:02 +00:00
Neil Alexander c3ad2cca49
Fix database default connection limits for CI (#1794) 2021-03-08 13:18:29 +00:00
Neil Alexander 6aa262ead8
Use default transport for AS traffic (#1789)
* Use default transport for AS traffic

* Update gmsl and use default client

* Remove replace

* Fix go.sum

* Update gomatrixserverlib

* Go back to appservices managing their own HTTP clients because argh

* Add missing context
2021-03-05 16:40:32 +00:00
Will Hunt fe021d3742
Treat the sender_localpart as an exclusive namespace of one user (#1790) 2021-03-05 14:57:42 +00:00
Neil Alexander 1ad96e2e2d
Tweak AS registration check and AS component HTTP clients (#1785)
* Tweak AS registration check

* Check appservice usernames using correct function

* Update sytest-whitelist

* Use gomatrixserverlib.Client since that allows us to disable TLS validation using the config

* Add appservice-specific client and ability to control TLS validation for appservices only

* Set timeout on appservice client

* Review comments

* Remove dead code

* Enforce LoginTypeApplicationService after all

* Check correct auth type field
2021-03-05 10:40:27 +00:00