Commit graph

207 commits

Author SHA1 Message Date
Mark Haines
b184a48897 Fetching missing state from the roomserver. (#135)
* Fetching missing state from the roomserver.

Whenever the syncserver receives an event from the room server that adds
state that isn't in the syncserver's local database it should fetch
those state events from the roomserver.

* Fix append

* Put comment back

* Comments

* s/addsStateEvents/lookupStateEvents/

* Fix spelling

* Include the stream position that a state event was added at in the current state tables

* Fix comment

* Review comments
2017-06-07 16:35:41 +01:00
Mark Haines
515cce1a45 Add support for receiving room events over federation. (#130)
* Add API for querying events by ID.

* Fix tense

* Start implementing federation ingress

* More stuff

* Hook up federation event receiving

* Handle the case where we are missing state

* Fix docstring and comments

* Fix infinite loop when printing unknownRoomError
2017-06-07 14:32:53 +01:00
Robert Swain
2d202cec07 mediaapi: Add thumbnail support (#132)
* vendor: Add bimg image processing library

bimg is MIT licensed. It depends on the C library libvips which is LGPL
v2.1+ licensed. libvips must be installed separately.

* mediaapi: Add YAML config file support

* mediaapi: Add thumbnail support

* mediaapi: Add missing thumbnail files

* travis: Add ppa and install libvips-dev

* travis: Another ppa and install libvips-dev attempt

* travis: Add sudo: required for sudo apt* usage

* mediaapi/thumbnailer: Make comparison code more readable

* mediaapi: Simplify logging of thumbnail properties

* mediaapi/thumbnailer: Rename metrics to fitness

Metrics is used in the context of monitoring with Prometheus so renaming
to avoid confusion.

* mediaapi/thumbnailer: Use math.Inf() for max aspect and size

* mediaapi/thumbnailer: Limit number of parallel generators

Fall back to selecting from already-/pre-generated thumbnails or serving
the original.

* mediaapi/thumbnailer: Split bimg code into separate file

* vendor: Add github.com/nfnt/resize pure go image scaler

* mediaapi/thumbnailer: Add nfnt/resize thumbnailer

* travis: Don't install libvips-dev via ppa

* mediaapi: Add notes to README about resizers

* mediaapi: Elaborate on scaling libs in README
2017-06-07 01:12:49 +02:00
Robert Swain
def49400bc cmd/client-api-proxy: Add proxy for media API (#133) 2017-06-05 15:55:04 +02:00
Mark Haines
721c2df484 Clean up syncapi storage to match the coding style set by roomserver storage (#131) 2017-06-05 10:37:04 +01:00
Mark Haines
ef7b934d51 Add Query API for querying events by ID (#129) 2017-06-02 14:32:36 +01:00
Mark Haines
1b6e06aa59 Revert "Add API for querying events by ID. (#127)" (#128)
This reverts commit ce82158abb.
2017-06-02 14:24:13 +01:00
Mark Haines
ce82158abb Add API for querying events by ID. (#127)
* Add API for querying events by ID.

* Fix tense

* Start implementing federation ingress

* More stuff

* Hook up federation event receiving

* Fix comments

* Comment on the order of the arrays
2017-06-02 11:19:34 +01:00
Robert Swain
7244fc15b3 media/writers/download: Make use of CreateMediaDownloadRequest from lib 2017-06-01 17:57:05 +02:00
Robert Swain
a398cdd193 mediaapi/writers/download: Use mime.ParseMediaType to parse Content-Disposition 2017-06-01 16:04:41 +02:00
Robert Swain
2d822c57f9 mediaapi/writers/download: Use DNS address as not all HSes have SRV 2017-06-01 15:30:36 +02:00
Robert Swain
cae309fa26 mediaapi/writers/download: Handle panic to ensure waking of goroutines
If the active request were to panic, we need to ensure all the waiting
goroutines get woken up.
2017-06-01 14:54:59 +02:00
Robert Swain
ce21b32d38 mediaapi/writers/download: Wrap broadcast call in closure to re-evaluate args 2017-06-01 14:44:00 +02:00
Robert Swain
4457ebddca mediaapi/writers/download: Rework remote file download synchronisation
Avoid locking around db requests by only locking around active requests
and always creating an active request if there is none. A nice side
effect of this is that if many parallel requests for remote media come
in (a common case) then only one database query is made for the duration
of the query.
2017-06-01 12:32:15 +02:00
Robert Swain
a3b1c7535a mediaapi/writers/download: Remove unnecesary Unlock, Lock after Cond.Wait 2017-06-01 08:39:35 +02:00
Robert Swain
923013aa55 mediaapi: Add support for downloading of remote files 2017-05-31 17:56:11 +02:00
Robert Swain
07c821daec mediaapi/writers: Fix InternalServerError responses 2017-05-31 17:41:42 +02:00
Robert Swain
2e013e3408 mediaapi/writers/download: Remove unnecessary closeConnection() 2017-05-31 15:39:19 +02:00
Robert Swain
bd96d99a3a mediaapi/writers/download: 500 ISE if disk and db file size differ 2017-05-31 15:08:00 +02:00
Robert Swain
e33438a37e mediaapi/writers/upload: Add standard fields to logger
So that they apply to all log messages thereafter
2017-05-31 14:54:10 +02:00
Robert Swain
5922ad383d mediaapi/writers/upload: Don't use deprecated sql.ErrNoRows check 2017-05-31 14:52:45 +02:00
Robert Swain
f0c717b0a1 mediaapi/writers/download: Escalate corrupt db/file cases to errors 2017-05-31 14:33:49 +02:00
Robert Swain
9606ea28ce mediaapi/writers/download: Always log origin and mediaID 2017-05-31 14:30:57 +02:00
Robert Swain
a45f008c12 mediaapi/storage: Don't leak sql.ErrNoRows out of storage package 2017-05-31 14:29:28 +02:00
Robert Swain
a936ad5063 mediaapi/writers/download: Add local download support 2017-05-31 13:46:21 +02:00
Robert Swain
3f7ef7690b Merge pull request #124 from matrix-org/rob/media-api-upload
/upload handler and storage database
2017-05-31 12:32:10 +02:00
Robert Swain
a0eae6922d mediaapi/writers: Remove unnecessary fmt.Sprintf 2017-05-31 07:12:22 +02:00
Robert Swain
08d1eb9669 mediaapi/upload: Improve HTTP status codes for error cases 2017-05-31 07:11:21 +02:00
Robert Swain
523303277e mediaapi/storage: Refer to RFC instead of golang for base64 format 2017-05-31 07:11:00 +02:00
Robert Swain
61329ee380 mediaapi/fileutils: Comment truncation of data when reading 2017-05-31 07:10:01 +02:00
Robert Swain
a4300eefc4 mediaapi/fileutils: Fix and comment os.IsNotExist bug 2017-05-31 07:08:21 +02:00
Robert Swain
63ccd770c6 mediaapi/fileutils: Use filepath not path for filesystem paths 2017-05-31 07:07:48 +02:00
Robert Swain
0ca2931b62 mediaapi/fileutils: Change path schema to put file in subdir of hash 2017-05-31 07:06:42 +02:00
Robert Swain
8c6f30eadc mediaapi/config: Remove obsolete proxying comment and add default comment 2017-05-31 07:05:07 +02:00
Kegsay
50aacd4f3c Hook up registration/login APIs and implement access token generation (#122) 2017-05-30 17:51:40 +01:00
Mark Haines
65b66a6452 Add API for querying the state after a list of events (#123)
* Add API for querying the state after a list of events

* Implement state after events API

* Actually implement the API

* Reorder .travis.yml

* Comments

* Use common.MakeAPI
2017-05-30 17:44:31 +01:00
Robert Swain
6fc6499848 mediaapi/fileutils: Remove obsolete error variables 2017-05-26 17:50:16 +02:00
Robert Swain
4f2d9a3b69 mediaapi/storage: Simplify descriptions 2017-05-26 17:44:43 +02:00
Robert Swain
9678cb6ea1 mediaapi/writers/upload: Simplify storeFileAndMetadata description 2017-05-26 17:42:08 +02:00
Robert Swain
9ecf620ad9 mediaapi/writers/upload: Factor out doUpload from Upload 2017-05-26 17:34:58 +02:00
Robert Swain
d83359dd51 mediaapi: Remove unnecessary ContentDisposition
Content-Disposition is only used for communicating the filename. It does
not need to be stored in the database as we have upload_name anyway. It
does not need to be in types.MediaMetadata and does not need to be
logged.
2017-05-26 17:24:13 +02:00
Robert Swain
731c10a418 mediaapi/fileutils: Clean up
Reorder functions to have public API functions in alphabetical order at
the top, internal package functions at the bottom in call order.

Use RawURLEncoding to avoid padding the hash with '='.

Use stronger types for paths in public API.

Simplify comments.
2017-05-26 17:15:54 +02:00
Robert Swain
05e88d81cb mediaapi: Add database storage and upload handler 2017-05-26 16:49:54 +02:00
Robert Swain
e21cd5ae66 Merge pull request #120 from matrix-org/rob/media-skeleton
Skeleton for the Media API
2017-05-26 15:57:13 +02:00
Robert Swain
86596488e9 mediaapi/writers/upload: Correct Content-Disposition evaluation
Content-Disposition is not a valid request header and so it must be
evaluated from the upload request's filename query parameter.
2017-05-26 15:26:50 +02:00
Robert Swain
98ef88b668 mediaapi/writers/download: Clarify validation applied to Origin 2017-05-26 14:59:45 +02:00
Robert Swain
4dfbae81cd mediaapi/writers/download: Allow only media IDs matching [A-Za-z0-9_=-]+ 2017-05-26 14:55:51 +02:00
Mark Haines
a96b124158 Move calculate state to the state package (#121) 2017-05-26 13:51:54 +01:00
Robert Swain
0affdae889 mediaapi/types: Rename member to ActiveRemoteRequests.MXCToCond
Clearer that it's a map from mxc URLs to sync.Cond
2017-05-26 14:17:54 +02:00
Robert Swain
e8d3b4648b mediaapi/routing: Use common.MakeAPI 2017-05-26 13:37:07 +02:00
Robert Swain
2fa0ae29d7 mediaapi/writers: Minor improvements to comments 2017-05-26 10:03:16 +02:00
Robert Swain
a3aaa86ea3 mediaapi: Add initial skeleton 2017-05-26 09:57:09 +02:00
Kegsay
db23640f69 cmd: Add create test accounts command (#119) 2017-05-25 17:41:45 +01:00
Mark Haines
84ad4ff9f6 Start implementing /join for room aliases for rooms the server is not in. (#115)
* Start implementing the join room API

* Hacks to get join room working

* Make the TLS fingerprint configurable

* Fix the client API proxy to handle '#' correctly

* Return a 200 OK response

* Write the join event along with current state to the room server

* Comment on the error handling

* Fix typos

* Fix tab

* Add TODO for moving authEventIDs to gomatrixserverlib

* Comment on why we ignore the key ID argument for local keys

* Avoid 'preceeded'

* Neaten the control flow

* Neaten the control flow even more

* Return pointers

* Rename err to lastErr for the loop
2017-05-25 16:08:28 +01:00
Kegsay
445dce14ae Add in devices_table to store device information (#118) 2017-05-25 13:33:50 +01:00
Kegsay
3b9222e8f7 Glue together devices and auth with the current HTTP code (#117)
- Renamed `clientapi/auth/types` to `clientapi/auth/authtypes` for the same
  horrible namespace clashing reasons as `storage`.
- Factored out `makeAPI` to `common`.
- Added in `makeAuthAPI`.
2017-05-23 17:43:05 +01:00
Mark Haines
a7acfa5546 Fix build after #114 2017-05-22 19:28:26 +01:00
Kegsay
d63a1ddc7c Rename packages under /auth (#114)
Previously, all database stuff was under the helpfully named
package 'storage'. However, this convention is used throughout all
of dendrite, which will clash the moment we want to add auth to all
the CS API endpoints. To prevent the package name clash, add
sub-directories which represent what is being stored so the final
usage ends up being:

```
func doThing(db *storage.SyncServerDatabase, authDB *accounts.Database)
{
    // ...
}
```
2017-05-22 16:49:32 +01:00
Kegsay
0325459e7f Implement dummy registration and hook it up to AccountDatabase (#113) 2017-05-22 15:55:39 +01:00
Mark Haines
6605333f6f Start implementing the federation server keys API. (#112)
* Start implementing the federation server keys API.

* Fix copyright

* Fix comments

* Comment on the key format

* Better explain what the ValidityPeriod is

* Return a 200 status code
2017-05-19 16:06:41 +01:00
Kegsay
9d4d18ae7f Add AccountDatabase for storing user accounts (#110)
Including the ability to add new accounts with a user/password and
select accounts using a user/password. Uses bcrypt to hash passwords.
2017-05-19 10:27:03 +01:00
Mark Haines
426a0365cf Rename "make" to "makeAPI" and factor out some more common code into it (#109)
* Rename "make" to "makeAPI" and factor out some more common code into it

Naming a function the same as a go builtin function seems like a bad
idea. Also move the call to `NewJSONRequestHander` inside the function
rather than calling it everywhere.

* Fix typo
2017-05-18 13:47:23 +01:00
Kegan Dougal
cf736d746d hook: Make go vet run all tests and fix warnings 2017-05-18 12:27:11 +01:00
Kegsay
917c433fd2 Minor refactoring (#106)
- `s/Server/OutputRoomEvent/` in `consumers` to accurately reflect what is being consumed.
- `s/set/userIDSet/` in `notifier.go` for clarity.
- Removed lying comments.
2017-05-17 17:29:26 +01:00
Kegsay
c8c5a40ebd Update README.md 2017-05-17 16:37:58 +01:00
Kegsay
0457a3bb69 State of the union update 2017-05-17 16:36:17 +01:00
Kegsay
ccd0eb2851 /sync bugfix: Check transitions to 'leave' do not leak events afterwards (#105) 2017-05-17 16:21:27 +01:00
Kegsay
d5a44fd3e8 Only wake up /sync requests which the event is for (#101) 2017-05-17 15:38:24 +01:00
Kegsay
0a3d44a80a Handle room transitions to 'leave' in incremental /sync requests (#104)
* Add test for transition to leave

* Add test for join/leave in the same /sync response
2017-05-17 10:25:59 +01:00
Kegsay
675759c192 Add invites to /sync responses (#103) 2017-05-15 17:41:54 +01:00
Kegsay
94e1c62745 Remove intermediary /sync response struct (#102)
The logic required to populate the right bits of `RoomData` tends towards
the complete `/sync` response struct, so just use the actual response struct
and save the hassle of mapping between the two. It may not make much difference
in its current form, but the next PR will make use of this.

This PR has no functional changes.
2017-05-15 15:18:08 +01:00
Kegsay
cc7117392f Send the entire room state down when transitioning to 'join' on a /sync response (#100)
This is only 'mostly' correct currently, because what should be no-op dupe
joins will actually trigger the entire room state to be re-sent.

Bizarrely, it's significantly easier to just do that than work out if we should,
and there are no client-visible effects to doing so, so we just do it for now.
2017-05-12 16:56:17 +01:00
Kegsay
2a9abefd92 Add more syncserver tests (#99)
Fixed exactly the same bug as before, but this time for incremental
sync. Yay tests!
2017-05-11 15:51:35 +01:00
Kegsay
a506985a66 Allow multiple sync server test requests (#98)
- Test data for the sync server is now in its own file.
- Rejig the sync server tests to support multiple /sync requests and corresponding
  assertions.
- Fixed a minor bug which resulted in state events to appear twice in /sync
  responses when syncing without a `since` parameter.
2017-05-10 17:48:35 +01:00
Kegsay
04f3c154b8 Split out notifying /sync requests and calculating sync responses (#96)
* Split out notifying /sync requests and calculating sync responses

The logic for notifying /sync requests is about to get really
complicated as we optimise when to wake up requests, so split
out that code into a separate struct to isolate it and make
it easier to unit test.
2017-05-10 10:42:00 +01:00
Kegsay
e226d564ec Fix a race with sync server integration tests (#95) 2017-05-09 15:58:31 +01:00
Kegsay
42564e8ed6 Factor out creating/deleting/writing to kafka topics (#94) 2017-05-09 09:05:05 +01:00
Kegsay
801b9246ce Add basic sync server integration test (#91) 2017-05-05 17:59:45 +01:00
Mark Haines
a56f609b74 Update gomatrixserverlib (#90) 2017-05-05 17:43:42 +01:00
Mark Haines
0309035aad Update gomatrixserverlib (#89) 2017-05-05 16:19:48 +01:00
Robert Swain
9b7defd375 common/log: Always log to stderr 2017-04-21 02:11:24 +02:00
Robert Swain
27beb8a2b3 common/log: Re-colorize terminal log output 2017-04-21 02:06:06 +02:00
Robert Swain
9b7bf8cba8 common/log: Add license header 2017-04-21 01:45:18 +02:00
Robert Swain
b33ed93dd6 Merge branch 'master' into rob/common-logging 2017-04-21 01:43:36 +02:00
Robert Swain
d09a65abbe Add Apache Version 2.0 license and headers to all golang files 2017-04-21 00:40:52 +02:00
Robert Swain
e281fc4530 syncserver: Rename to syncapi 2017-04-20 19:21:15 +02:00
Robert Swain
d8b98cec44 client-api-proxy: roomserver-integration-tests: Fix renamed references 2017-04-20 19:21:15 +02:00
Robert Swain
0939ae3686 Rename dendrite-syncserver to dendrite-sync-api-server 2017-04-20 19:21:15 +02:00
Robert Swain
209ba3f6a3 Rename dendrite-roomserver to dendrite-room-server 2017-04-20 19:21:15 +02:00
Robert Swain
2278848922 Rename dendrite-clientapi to dendrite-client-api-server 2017-04-20 19:21:15 +02:00
Kegan Dougal
2a264db85e Don't tightloop if currentPos is 0 2017-04-20 18:01:45 +01:00
Kegsay
cb5081b332 Yank out clientapi config options to env vars for now (#77) 2017-04-20 17:15:34 +01:00
Kegsay
2fadb9e409 Make it possible to point Riot at Dendrite (#74) 2017-04-20 17:11:53 +01:00
Mark Haines
43d1e94e75 Rename dendrite-sync-server to dendrite-syncserver for consistency (#76) 2017-04-20 16:54:22 +01:00
Robert Swain
be47984c43 common/log: Always output timestamps as UTC 2017-04-20 17:16:16 +02:00
Robert Swain
f65e26bc2a cmd: Configure logging before any log messages 2017-04-20 16:28:15 +02:00
Robert Swain
254e61f727 common/log: Switch to RFC3339 format with nanoseconds and trailing zeros 2017-04-20 16:28:15 +02:00
Robert Swain
e8d2d61cc2 cmd: common/log: Always configure logging
When LOG_DIR or so is not specified, just configure the formatter.
2017-04-20 16:27:48 +02:00
Robert Swain
8010083026 dendrite/common: Move logrus configuration to common 2017-04-20 16:27:44 +02:00
Kegsay
db428174d2 tool: Add client-api-proxy (#73) 2017-04-20 14:00:34 +01:00