Commit Graph

1326 Commits (ebcacd1bb56d6e37ff743c0430bc91e24d440199)

Author SHA1 Message Date
Neil Alexander 3034d8e805
Update sytest lists to blacklist flaky 'Forgotten room messages cannot be paginated' test 2020-11-16 10:51:55 +00:00
Mayeul Cantan af41fcadc4
Fix Dendrite not backfilling on world_readable rooms (#1575)
The previous implementation was only checking if room history was
"shared", which it wasn't for rooms where a user was invited, or world
readable rooms.
This implementation leverages the IsServerAllowed method, which already
implements the complete verification algorithm.

Signed-off-by: `Mayeul Cantan <oss+matrix@mayeul.net>`

Co-authored-by: Kegsay <kegan@matrix.org>
2020-11-16 10:47:16 +00:00
Felix Hanley 41c0b5acfb
Update build script to run on any POSIX (#1577)
Signed-off-by: Felix Hanley <felix@userspace.com.au>
2020-11-16 10:09:53 +00:00
S7evinK cb41c992f7
Add -version flag (#1581)
Signed-off-by: Till Faelligen <tfaelligen@gmail.com>
2020-11-16 09:54:58 +00:00
Neil Alexander b7e288084c
Update README.md 2020-11-15 12:39:11 +00:00
Neil Alexander 8ce740d949
Fix username regex to allow = character (#1578) 2020-11-12 10:36:54 +00:00
S7evinK bcb89ada5e
Implement read receipts (#1528)
* fix conversion from int to string yields a string of one rune, not a string of digits

* Add receipts table to syncapi

* Use StreamingToken as the since value

* Add required method to testEDUProducer

* Make receipt json creation "easier" to read

* Add receipts api to the eduserver

* Add receipts endpoint

* Add eduserver kafka consumer

* Add missing kafka config

* Add passing tests to whitelist

Signed-off-by: Till Faelligen <tfaelligen@gmail.com>

* Fix copy & paste error

* Fix column count error

* Make outbound federation receipts pass

* Make "Inbound federation rejects receipts from wrong remote" pass

* Don't use errors package

* - Add TODO for batching requests
- Rename variable

* Return a better error message

* - Use OutputReceiptEvent instead of InputReceiptEvent as result
- Don't use the errors package for errors
- Defer CloseAndLogIfError to close rows
- Fix Copyright

* Better creation/usage of JoinResponse

* Query all joined rooms instead of just one

* Update gomatrixserverlib

* Add sqlite3 migration

* Add postgres migration

* Ensure required sequence exists before running migrations

* Clarification on comment

* - Fix a bug when creating client receipts
- Use concrete types instead of interface{}

* Remove dead code
Use key for timestamp

* Fix postgres query...

* Remove single purpose struct

* Use key/value directly

* Only apply receipts on initial sync or if edu positions differ,
otherwise we'll be sending the same receipts over and over again.

* Actually update the id, so it is correctly send in syncs

* Set receipt on request to /read_markers

* Fix issue with receipts getting overwritten

* Use fmt.Errorf instead of pkg/errors

* Revert "Add postgres migration"

This reverts commit 722fe5a04628882b787d096942459961db159b06.

* Revert "Add sqlite3 migration"

This reverts commit d113b03f6495a4b8f8bcf158a3d00b510b4240cc.

* Fix selectRoomReceipts query

* Make golangci-lint happy

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2020-11-09 18:46:11 +00:00
S7evinK eccd0d2c1b
Implement forgetting about rooms (#1572)
* Add basic storage methods

* Add internal api handler

* Add check for forgotten room

* Add /rooms/{roomID}/forget endpoint

* Add missing rsAPI method

* Remove unused parameters

* Add passing tests

Signed-off-by: Till Faelligen <tfaelligen@gmail.com>

* Add missing file

* Add postgres migration

* Add sqlite migration

* Use Forgetter to forget room

* Remove empty line

* Update HTTP status codes

It looks like the spec calls for these to be 400, rather than 403: https://matrix.org/docs/spec/client_server/r0.6.1#post-matrix-client-r0-rooms-roomid-forget

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2020-11-05 10:19:23 +00:00
Neil Alexander 2ce2112ddb
Update gjson for safe integer ranges 2020-11-05 09:42:28 +00:00
Petter Jakub Økland fe339c8b98
Created polylith reverse proxy sample for Hiawatha (#1571) 2020-10-28 16:37:56 +00:00
S7evinK d5675feb96
Add possibilty to configure MaxMessageBytes for sarama (#1563)
* Add configuration for max_message_bytes for sarama

* Log all errors when sending multiple messages

Signed-off-by: Till Faelligen <tfaelligen@gmail.com>

* Add missing config

* - Better comments on what MaxMessageBytes is used for
- Also sets the size the consumer may use
2020-10-27 14:11:37 +00:00
Neil Alexander c5888bb64c
Update Docker sample configs 2020-10-27 13:58:58 +00:00
Neil Alexander 0af35bec1a
Update documentation (#1569)
* Update dendrite-config.yaml

* Update README.md

* Update INSTALL.md

* Update INSTALL.md
2020-10-26 22:09:13 +00:00
Neil Alexander 59428cdde3
Version 0.2.1 (#1558)
* Version 0.2.1

* Credit where credit due

* Update CHANGES.md

* Include PRs
2020-10-22 12:38:12 +01:00
Adit Sachde b66b2eab32
Document dendrite server key format (#1520)
Co-authored-by: Kegsay <kegan@matrix.org>
2020-10-22 12:21:31 +01:00
Neil Alexander f32320a235
Exclude old state events from sync when consumed from roomserver (#1548)
* Exclude old room events from sync when consumed from roomserver

* Fix comment

* Experimental hack
2020-10-22 11:50:48 +01:00
Neil Alexander 3afc623098
Fix RewritesState bug (#1557)
* Set RewritesState once

* Check if any new state provided

* Obey rewritesState

* Don't nuke everything the sync API knows when purging state

* Fix panic from duplicate insert

* Consistency

* Use HasState

* Remove nolint

* Clean up joined rooms on state rewrite
2020-10-22 10:39:16 +01:00
Neil Alexander 04dc019e5e
Don't set empty state snapshots 2020-10-21 16:21:36 +01:00
Neil Alexander 534f9a9eb6
Refactor forward extremities (#1556)
* Add resolve-state helper

* Tweaks

* Refactor forward extremities, again

* Tweaks

* Minor optimisation

* Make path a bit clearer

* Only process state/membership if forward extremities have changed

* Usage comments in resolve-state
2020-10-21 15:37:07 +01:00
Vladislav Povalyaev e4f3f38f35
Update development percentages (#1553) 2020-10-20 20:49:03 +01:00
Kegsay 6b8791b868
Always call overridden form of GetLatestEventsForUpdate (#1554)
This ensures we don't make txns on sqlite still, which can cause
'database is locked' errors.
2020-10-20 19:32:33 +01:00
Neil Alexander 7612f64e3c
Update gomatrixserverlib to matrix-org/gomatrixserverlib#220 2020-10-20 17:23:04 +01:00
Neil Alexander 24e38c4135
Internal HTTP APIs over H2C (#1541)
* H2C on internal HTTP because SCIENCE

* Update comments
2020-10-20 17:13:12 +01:00
Neil Alexander 7ca89ef511
Update gomatrixserverlib 2020-10-20 17:10:37 +01:00
Neil Alexander 6a16d46fba
Version 0.2.0 (#1551)
* v0.2.0-rc2

* Update CHANGES.md (also with some markdown lint suggestions)

* Update version number

* Update CHANGES.md

* Update CHANGES.md

* Update CHANGES.md

* Add known issue

* Update CHANGES.md
2020-10-20 16:46:53 +01:00
Neil Alexander 39c7a8915c
Multi-personality polylith binary (#1552)
* Initial work oon multipersonality binary

* Remove old binaries

* Monolith and polylith binaries

* Better logging

* dendrite-poly-multi

* Fix path

* Copyright notices etc

* Tweaks

* Update Docker, INSTALL.md

* Take first argument if flags package doesn't find any args

* Postgres 9.6 or later, fix some more Docker stuff

* Don't create unnecessary e2ekey DB

* Run go mod tidy
2020-10-20 16:11:24 +01:00
Neil Alexander 6c3c621de0
Remove invalid state delta check (#1550) 2020-10-20 12:36:16 +01:00
Kegan Dougal 837c295c26 Linting 2020-10-20 12:29:53 +01:00
Pika 53a745f333
fix create-account (#1546) 2020-10-20 11:47:37 +01:00
Kegsay eb86e2b336
Fix sqlite locking bugs present on sytest (#1543)
* Fix sqite locking bugs present on sytest

Comments do the explaining.

* Fix deadlock in sqlite mode

Caused by starting a writer whilst within a writer

* Only complain about invalid state deltas for non-overwrite events

* Do not re-process outlier unnecessarily
2020-10-20 11:42:54 +01:00
Neil Alexander 92982a402f
Update Docker (#1542)
* Separate Docker images, rather than tags

* Allow specifying tag to build/push/pull

* Include goose in Docker builds
2020-10-20 11:34:22 +01:00
Neil Alexander a71360d099
Update INSTALL.md (#1549) 2020-10-20 11:22:46 +01:00
Neil Alexander c7bf122a26
Update sytest lists 2020-10-19 15:38:42 +01:00
Neil Alexander 5d80ff11a0
Update sytest-whitelist 2020-10-19 15:17:28 +01:00
Devon Johnson 45abdcaeb9
Send state after event, not current
Signed-off-by: Devon Johnson <djohnson1865@gmail.com>
2020-10-19 15:12:37 +01:00
Neil Alexander 6e63df1d9a
KindOld (#1531)
* Add KindOld

* Don't process latest events/memberships for old events

* Allow federationsender to ignore duplicate key entries when LatestEventIDs is duplicated by RS output events

* Signal to downstream components if an event has become a forward extremity

* Don't exclude from sync

* Soft-fail checks on KindNew

* Don't run the latest events updater at all for KindOld

* Don't make federation sender change after all

* Kind in federation sender join

* Don't send isForwardExtremity

* Fix syncapi

* Update comments

* Fix SendEventWithState

* Update sytest-whitelist

* Generate old output events

* Sync API consumes old room events

* Update comments
2020-10-19 14:59:13 +01:00
S7evinK 0974f6e2c0
Fix internal http api (#1535)
Signed-off-by: Till Faelligen <tfaelligen@gmail.com>
2020-10-19 09:38:10 +01:00
Neil Alexander 640e8c50ec
Take write lock for rate limit map (#1532)
* Take write lock for rate limit map

* Fix potential race condition
2020-10-16 15:44:39 +01:00
Kegsay 4a7fb9c045
Automatically upgrade databases on startup (#1529)
* Support auto-upgrading accounts DB

* Auto-upgrade device DB deltas

* Support up/downgrading from cmd/goose

* Linting

* Create tables then do migrations then prepare statements

To avoid failing due to some things not existing

* Linting
2020-10-15 18:09:41 +01:00
Neil Alexander 3e5d38e284
Improve state resolution v2 performance (matrix-org/gomatrixserverlib#237) 2020-10-15 16:20:19 +01:00
Neil Alexander e154c45b51
Better logging around db.StoreEvent 2020-10-15 14:14:17 +01:00
Neil Alexander 49abe359e6
Start Kafka connections for each component that needs them (#1527)
* Start Kafka connection for each component that needs one

* Fix roomserver unit tests

* Rename to naffkaInstance (@Kegsay review comment)

* Fix import cycle
2020-10-15 13:27:13 +01:00
Neil Alexander 10f1beb0de
Don't re-run state resolution on a single trusted state snapshot (#1526)
* Don't re-run state resolution on a single trusted state snapshot

* Lint

* Check if backward extremity is create event before checking missing state
2020-10-15 12:08:49 +01:00
Kegsay e3c2b081c7
txn nil guard when rolling back LatestEventsUpdater (#1524)
* txn nil guard when rolling back LatestEventsUpdater

* Spell lint correctly
2020-10-14 17:05:09 +01:00
Neil Alexander 6f12b8f85c
Ignore typing events where sender doesn't match origin (#1523)
* Ignore typing notifications where the sender doesn't match the origin

* Update sytest-whitelist

* Fix formatting directives
2020-10-14 16:49:25 +01:00
Neil Alexander e3a3908654
Update sytest-whitelist (closes #1315) 2020-10-14 16:07:15 +01:00
Neil Alexander 8d9ecb3996
Ignore duplicate redaction entries (#1522) 2020-10-14 15:24:43 +01:00
Neil Alexander 286dd408ae
Better semver compliance for version reporting 2020-10-14 12:51:06 +01:00
Neil Alexander 7a1fd123de
Improved state handling in /send (#1521)
* Capture errors

* Don't request only state key tuples needed for auth (we end up discarding room state this way)

* QueryStateAfterEvent returns all state when no tuples supplied

* Resolve state

* Comments
2020-10-14 12:39:37 +01:00
Neil Alexander 20aec70ead
Send cumulative state when creating room (#1519)
* Send state with new room events

* lookupEvent sends outliers

* Revert "lookupEvent sends outliers"

This reverts commit 3e1655644105a542b806e28d6d2536fbd23ecc83.
2020-10-14 09:57:27 +01:00