Commit Graph

33 Commits (main)

Author SHA1 Message Date
S7evinK 5094bc89bf
Set MaxFileSizeBytes <= 0 to unlimited (#1876)
* Revert "Set MaxFileSizeBytes <= 0 to "unlimited" (#1875)"

This reverts commit 9ed0440421.

* Actually allow unlimited upload

Signed-off-by: Till Faelligen <tfaelligen@gmail.com>

Co-authored-by: kegsay <kegan@matrix.org>
2021-07-19 17:58:51 +01:00
database64128 79181f2348
✂️ Media API: Handle unlimited file size (#1881) 2021-06-22 10:01:44 +01:00
S7evinK 9ed0440421
Set MaxFileSizeBytes <= 0 to "unlimited" (#1875)
* Set MaxFileSizeBytes < 0 to "unlimited"

Signed-off-by: Till Faelligen <tfaelligen@gmail.com>

* int64 overflows later in mediaapi/routing/upload.go[doUpload]

* Prevent int overflow when uploading
2021-06-14 14:12:03 +01:00
S7evinK 8b22c4270d
Use LimitReader to prevent DoS risk (#1843)
* Use LimitReader to prevent DoS risk

Signed-off-by: Till Faelligen <tfaelligen@gmail.com>

* Check if bytesWritten is equal to the maxFileSize
Add tests

Signed-off-by: Till Faelligen <tfaelligen@gmail.com>

* Use oldschool defer to cleanup after the tests

* Let LimitReader read MaxFileSizeBytes + 1

Co-authored-by: Kegsay <kegan@matrix.org>
2021-06-07 09:17:20 +01:00
Fero d6e9b7b307
Remove the 'Content-Type' request header requirement (#1834) 2021-04-19 16:29:51 +01:00
Fero 653e30619c
Remove the Content-Length requirement on upload request (#1831)
* Remove the Content-Length requirement

* Make sure that the file size does not exceed max content size

* Address review comment - universally check if temp file size exceeds max file size
2021-04-14 10:53:24 +01:00
darkgallium da797c7998
Retrieve remote file size without the Content-Size header (#1537)
* Get downloaded file size regardless of the Content-Size header

Signed-off-by: Florian Le Minoux <darkgallium+git@deuxfleurs.fr>

* fixing lint issues

* Don't exhaust memory for large files, don't limit more than necessary

* Don't use errors.Wrap in download.go

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2021-02-17 13:54:53 +00:00
Neil Alexander b5aa7ca3ab
Top-level setup package (#1605)
* Move config, setup, mscs into "setup" top-level folder

* oops, forgot the EDU server

* Add setup

* goimports
2020-12-02 17:41:00 +00:00
Kegsay 3802efe301
Port of #945 (#1343)
- Make sure we always cleanup the temp directory on error.
- Complain about it having an error prone API shape.
2020-08-26 15:38:34 +01:00
Neil Alexander 287700baee
Randomly generated media IDs (#1336)
* Filenames based on hashes but media IDs randomly generated and per-upload metadata

* Lint

* Comment generateMediaID
2020-08-25 15:08:37 +01:00
Neil Alexander 9677a95afc
API setup refactoring (#1266)
* Start HTTP endpoint refactoring

* Update SetupAndServeHTTP

* Fix builds

* Don't set up external listener if no address configured

* TLS HTTP setup

* Break apart client/federation/key/media muxes

* Tweaks

* Fix P2P demos

* Fix media API routing

* Review comments @Kegsay

* Update sample config

* Fix gobind build

* Fix External -> Public in federation API test
2020-08-13 12:16:37 +01:00
Neil Alexander 4b09f445c9
Configuration format v1 (#1230)
* Initial pass at refactoring config (not finished)

* Don't forget current state and EDU servers

* More shifting around

* Update server key API tests

* Fix roomserver test

* Fix more tests

* Further tweaks

* Fix current state server test (sort of)

* Maybe fix appservices

* Fix client API test

* Include database connection string in database options

* Fix sync API build

* Update config test

* Fix unit tests

* Fix federation sender build

* Fix gobind build

* Set Listen address for all services in HTTP monolith mode

* Validate config, reinstate appservice derived in directory, tweaks

* Tweak federation API test

* Set MaxOpenConnections/MaxIdleConnections to previous values

* Update generate-config
2020-08-10 14:18:04 +01:00
Neil Alexander 6bee6e77d9
Yet another v1 vs r0 media endpoint (#1190) 2020-07-08 16:39:50 +01:00
Neil Alexander 23bed196e6
Fallback parsing for Content-Disposition minefield (#1144) 2020-06-17 14:26:45 +01:00
Neil Alexander 5d5aa0a31d
Media filename handling improvements (#1140)
* Derive content ID from hash+filename but preserve dedupe, improve Content-Disposition handling and ASCII handling

* Linter fix

* Some more comments

* Update sytest-whitelist
2020-06-17 11:53:26 +01:00
Neil Alexander 04c99092a4
Update whitelist for sytest media fix (#1137)
* Update sytest-whitelist, are-we-synapse-yet.list

* Update gomatrixserverlib

* Update gomatrixserverlib

* Loop avoidance

* Return UTF-8 filenames

* Replace quotes only, instead of using strconv.Quote

* Update sytest-whitelist

* Update sytest-whitelist
2020-06-16 18:31:38 +01:00
Neil Alexander fc0e74ae0f
Fix media API for demos and possibly Synapse (#1134)
* Fix media API for demos and possibly Synapse

* User API

* goimports
2020-06-16 14:29:11 +01:00
Kegsay 9c77022513
Make userapi responsible for checking access tokens (#1133)
* Make userapi responsible for checking access tokens

There's still plenty of dependencies on account/device DBs, but this
is a start. This is a breaking change as it adds a required config
value `listen.user_api`.

* Cleanup

* Review comments and test fix
2020-06-16 14:10:55 +01:00
Kegsay ecd7accbad
Rehuffle where things are in the internal package (#1122)
renamed:    internal/eventcontent.go -> internal/eventutil/eventcontent.go
	renamed:    internal/events.go -> internal/eventutil/events.go
	renamed:    internal/types.go -> internal/eventutil/types.go
	renamed:    internal/http/http.go -> internal/httputil/http.go
	renamed:    internal/httpapi.go -> internal/httputil/httpapi.go
	renamed:    internal/httpapi_test.go -> internal/httputil/httpapi_test.go
	renamed:    internal/httpapis/paths.go -> internal/httputil/paths.go
	renamed:    internal/routing.go -> internal/httputil/routing.go
	renamed:    internal/basecomponent/base.go -> internal/setup/base.go
	renamed:    internal/basecomponent/flags.go -> internal/setup/flags.go
	renamed:    internal/partition_offset_table.go -> internal/sqlutil/partition_offset_table.go
	renamed:    internal/postgres.go -> internal/sqlutil/postgres.go
	renamed:    internal/postgres_wasm.go -> internal/sqlutil/postgres_wasm.go
	renamed:    internal/sql.go -> internal/sqlutil/sql.go
2020-06-12 14:55:57 +01:00
Neil Alexander fe82e1f725
Separate muxes for public and internal APIs (#1056)
* Separate muxes for public and internal APIs

* Update client-api-proxy and federation-api-proxy so they don't add /api to the path

* Tidy up

* Consistent HTTP setup

* Set up prefixes properly
2020-05-22 11:43:17 +01:00
Kegsay 24d8df664c
Fix #897 and shuffle directory around (#1054)
* Fix #897 and shuffle directory around

* Update find-lint

* goimports

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2020-05-21 14:40:13 +01:00
Sid Karunaratne 31e6a7f193
Enforce `mediaIDRegex` to be only valid `mediaIDCharacters` (#1020)
Error messages indicate that:
> mediaId must be a non-empty string using only characters in `mediaIDCharacters`

However the regex used only required that some characters in the filename match
the restriction, not that the entire filename does. This commit ensures that
the filename must entirely fullfill the `mediaIDCharacters` restriction

Signed-off-by: Sid Karunaratne <sid@karunaratne.net>

Co-authored-by: Kegsay <kegan@matrix.org>
2020-05-13 12:04:54 +01:00
Neil Alexander 4fd97df2c5 Don't return 500s from media API download requests 2020-05-11 14:01:07 +01:00
Kegsay b6ea1bc67a
Support sqlite in addition to postgres (#869)
* Move current work into single branch

* Initial massaging of clientapi etc (not working yet)

* Interfaces for accounts/devices databases

* Duplicate postgres package for sqlite3 (no changes made to it yet)

* Some keydb, accountdb, devicedb, common partition fixes, some more syncapi tweaking

* Fix accounts DB, device DB

* Update naffka dependency for SQLite

* Naffka SQLite

* Update naffka to latest master

* SQLite support for federationsender

* Mostly not-bad support for SQLite in syncapi (although there are problems where lots of events get classed incorrectly as backward extremities, probably because of IN/ANY clauses that are badly supported)

* Update Dockerfile -> Go 1.13.7, add build-base (as gcc and friends are needed for SQLite)

* Implement GET endpoints for account_data in clientapi

* Nuke filtering for now...

* Revert "Implement GET endpoints for account_data in clientapi"

This reverts commit 4d80dff4583d278620d9b3ed437e9fcd8d4674ee.

* Implement GET endpoints for account_data in clientapi (#861)

* Implement GET endpoints for account_data in clientapi

* Fix accountDB parameter

* Remove fmt.Println

* Fix insertAccountData SQLite query

* Fix accountDB storage interfaces

* Add empty push rules into account data on account creation (#862)

* Put SaveAccountData into the right function this time

* Not sure if roomserver is better or worse now

* sqlite work

* Allow empty last sent ID for the first event

* sqlite: room creation works

* Support sending messages

* Nuke fmt.println

* Move QueryVariadic etc into common, other device fixes

* Fix some linter issues

* Fix bugs

* Fix some linting errors

* Fix errcheck lint errors

* Make naffka use postgres as fallback, fix couple of compile errors

* What on earth happened to the /rooms/{roomID}/send/{eventType} routing

Co-authored-by: Neil Alexander <neilalexander@users.noreply.github.com>
2020-02-13 17:27:33 +00:00
Neil Alexander c28577ea25
Implement storage interfaces (#841)
* Implement interfaces for federationsender storage

* Implement interfaces for mediaapi storage

* Implement interfaces for publicroomsapi storage

* Implement interfaces for roomserver storage

* Implement interfaces for syncapi storage

* Implement interfaces for keydb storage

* common.PartitionStorer in publicroomsapi interface

* Update copyright notices
2020-01-03 14:07:05 +00:00
Andrew Morgan f392ce8ed7 URL decode args 2019-12-18 15:29:27 +00:00
Andrew Morgan 9b5d6c9745 Refactor InstrumentHandlerCounter definition 2019-12-18 15:10:53 +00:00
Andrew Morgan de5d463dd2 goimports 2019-12-17 16:54:21 +00:00
Andrew Morgan 8fb2c9c33c Replace deprecated prometheus.InstrumentHandler and unsafe time.Ticker 2019-12-17 16:47:45 +00:00
Andrew Morgan 504d23f468 Remove unnecessary http method checks (#747)
Closes #523

There were a lot of unnecessary checks for HTTP methods of requests. gorilla/mux makes sure that these methods will only be called if certain HTTP methods are used, thus there's no reason to have these extra checks.
2019-07-18 15:40:10 +08:00
Behouba Manassé 69f8d5a77e Content-Disposition HTTP header in mediaapi's responses added (#685)
This should fix #628
I also Noticed that GET /_matrix/media/r0/download/{serverName}/{mediaId}/{fileName} is not yet implemented, but it should work for both.

Signed-off-by: Kouamé Behouba Manassé behouba@gmail.com
2019-07-08 14:06:17 +01:00
Andrew Morgan 33a1392541
Encode URLs properly (#728)
We were escaping the URL before performing any pattern matching on it.
This meant that if you sent data that URLdecoded to a "/", it would count as
a "/" in the URL, potentially causing a 404. This was causing some flaky tests
with some randomly-generated query parameters.

Now, we keep URLs encoded while doing the pattern matching, and only afterwards
do we URL decode each query parameter individually before passing them to their
respective handler functions.

github.com/gorilla/mux was also updated to v1.7.3 to fix a bug with URL encoding and subrouters.
2019-07-03 16:38:50 +01:00
ruben 74827428bd use go module for dependencies (#594) 2019-05-21 21:56:55 +01:00