Commit Graph

319 Commits (fa362ecef2f71419d3819a763313167c42da9443)

Author SHA1 Message Date
Robert Swain 9678cb6ea1 mediaapi/writers/upload: Simplify storeFileAndMetadata description 2017-05-26 17:42:08 +02:00
Robert Swain 9ecf620ad9 mediaapi/writers/upload: Factor out doUpload from Upload 2017-05-26 17:34:58 +02:00
Robert Swain d83359dd51 mediaapi: Remove unnecessary ContentDisposition
Content-Disposition is only used for communicating the filename. It does
not need to be stored in the database as we have upload_name anyway. It
does not need to be in types.MediaMetadata and does not need to be
logged.
2017-05-26 17:24:13 +02:00
Robert Swain 731c10a418 mediaapi/fileutils: Clean up
Reorder functions to have public API functions in alphabetical order at
the top, internal package functions at the bottom in call order.

Use RawURLEncoding to avoid padding the hash with '='.

Use stronger types for paths in public API.

Simplify comments.
2017-05-26 17:15:54 +02:00
Robert Swain 05e88d81cb mediaapi: Add database storage and upload handler 2017-05-26 16:49:54 +02:00
Robert Swain e21cd5ae66 Merge pull request #120 from matrix-org/rob/media-skeleton
Skeleton for the Media API
2017-05-26 15:57:13 +02:00
Robert Swain 86596488e9 mediaapi/writers/upload: Correct Content-Disposition evaluation
Content-Disposition is not a valid request header and so it must be
evaluated from the upload request's filename query parameter.
2017-05-26 15:26:50 +02:00
Robert Swain 98ef88b668 mediaapi/writers/download: Clarify validation applied to Origin 2017-05-26 14:59:45 +02:00
Robert Swain 4dfbae81cd mediaapi/writers/download: Allow only media IDs matching [A-Za-z0-9_=-]+ 2017-05-26 14:55:51 +02:00
Mark Haines a96b124158 Move calculate state to the state package (#121) 2017-05-26 13:51:54 +01:00
Robert Swain 0affdae889 mediaapi/types: Rename member to ActiveRemoteRequests.MXCToCond
Clearer that it's a map from mxc URLs to sync.Cond
2017-05-26 14:17:54 +02:00
Robert Swain e8d3b4648b mediaapi/routing: Use common.MakeAPI 2017-05-26 13:37:07 +02:00
Robert Swain 2fa0ae29d7 mediaapi/writers: Minor improvements to comments 2017-05-26 10:03:16 +02:00
Robert Swain a3aaa86ea3 mediaapi: Add initial skeleton 2017-05-26 09:57:09 +02:00
Kegsay db23640f69 cmd: Add create test accounts command (#119) 2017-05-25 17:41:45 +01:00
Mark Haines 84ad4ff9f6 Start implementing /join for room aliases for rooms the server is not in. (#115)
* Start implementing the join room API

* Hacks to get join room working

* Make the TLS fingerprint configurable

* Fix the client API proxy to handle '#' correctly

* Return a 200 OK response

* Write the join event along with current state to the room server

* Comment on the error handling

* Fix typos

* Fix tab

* Add TODO for moving authEventIDs to gomatrixserverlib

* Comment on why we ignore the key ID argument for local keys

* Avoid 'preceeded'

* Neaten the control flow

* Neaten the control flow even more

* Return pointers

* Rename err to lastErr for the loop
2017-05-25 16:08:28 +01:00
Kegsay 445dce14ae Add in devices_table to store device information (#118) 2017-05-25 13:33:50 +01:00
Kegsay 3b9222e8f7 Glue together devices and auth with the current HTTP code (#117)
- Renamed `clientapi/auth/types` to `clientapi/auth/authtypes` for the same
  horrible namespace clashing reasons as `storage`.
- Factored out `makeAPI` to `common`.
- Added in `makeAuthAPI`.
2017-05-23 17:43:05 +01:00
Mark Haines a7acfa5546 Fix build after #114 2017-05-22 19:28:26 +01:00
Kegsay d63a1ddc7c Rename packages under /auth (#114)
Previously, all database stuff was under the helpfully named
package 'storage'. However, this convention is used throughout all
of dendrite, which will clash the moment we want to add auth to all
the CS API endpoints. To prevent the package name clash, add
sub-directories which represent what is being stored so the final
usage ends up being:

```
func doThing(db *storage.SyncServerDatabase, authDB *accounts.Database)
{
    // ...
}
```
2017-05-22 16:49:32 +01:00
Kegsay 0325459e7f Implement dummy registration and hook it up to AccountDatabase (#113) 2017-05-22 15:55:39 +01:00
Mark Haines 6605333f6f Start implementing the federation server keys API. (#112)
* Start implementing the federation server keys API.

* Fix copyright

* Fix comments

* Comment on the key format

* Better explain what the ValidityPeriod is

* Return a 200 status code
2017-05-19 16:06:41 +01:00
Kegsay 9d4d18ae7f Add AccountDatabase for storing user accounts (#110)
Including the ability to add new accounts with a user/password and
select accounts using a user/password. Uses bcrypt to hash passwords.
2017-05-19 10:27:03 +01:00
Mark Haines 426a0365cf Rename "make" to "makeAPI" and factor out some more common code into it (#109)
* Rename "make" to "makeAPI" and factor out some more common code into it

Naming a function the same as a go builtin function seems like a bad
idea. Also move the call to `NewJSONRequestHander` inside the function
rather than calling it everywhere.

* Fix typo
2017-05-18 13:47:23 +01:00
Kegan Dougal cf736d746d hook: Make go vet run all tests and fix warnings 2017-05-18 12:27:11 +01:00
Kegsay 917c433fd2 Minor refactoring (#106)
- `s/Server/OutputRoomEvent/` in `consumers` to accurately reflect what is being consumed.
- `s/set/userIDSet/` in `notifier.go` for clarity.
- Removed lying comments.
2017-05-17 17:29:26 +01:00
Kegsay c8c5a40ebd Update README.md 2017-05-17 16:37:58 +01:00
Kegsay 0457a3bb69 State of the union update 2017-05-17 16:36:17 +01:00
Kegsay ccd0eb2851 /sync bugfix: Check transitions to 'leave' do not leak events afterwards (#105) 2017-05-17 16:21:27 +01:00
Kegsay d5a44fd3e8 Only wake up /sync requests which the event is for (#101) 2017-05-17 15:38:24 +01:00
Kegsay 0a3d44a80a Handle room transitions to 'leave' in incremental /sync requests (#104)
* Add test for transition to leave

* Add test for join/leave in the same /sync response
2017-05-17 10:25:59 +01:00
Kegsay 675759c192 Add invites to /sync responses (#103) 2017-05-15 17:41:54 +01:00
Kegsay 94e1c62745 Remove intermediary /sync response struct (#102)
The logic required to populate the right bits of `RoomData` tends towards
the complete `/sync` response struct, so just use the actual response struct
and save the hassle of mapping between the two. It may not make much difference
in its current form, but the next PR will make use of this.

This PR has no functional changes.
2017-05-15 15:18:08 +01:00
Kegsay cc7117392f Send the entire room state down when transitioning to 'join' on a /sync response (#100)
This is only 'mostly' correct currently, because what should be no-op dupe
joins will actually trigger the entire room state to be re-sent.

Bizarrely, it's significantly easier to just do that than work out if we should,
and there are no client-visible effects to doing so, so we just do it for now.
2017-05-12 16:56:17 +01:00
Kegsay 2a9abefd92 Add more syncserver tests (#99)
Fixed exactly the same bug as before, but this time for incremental
sync. Yay tests!
2017-05-11 15:51:35 +01:00
Kegsay a506985a66 Allow multiple sync server test requests (#98)
- Test data for the sync server is now in its own file.
- Rejig the sync server tests to support multiple /sync requests and corresponding
  assertions.
- Fixed a minor bug which resulted in state events to appear twice in /sync
  responses when syncing without a `since` parameter.
2017-05-10 17:48:35 +01:00
Kegsay 04f3c154b8 Split out notifying /sync requests and calculating sync responses (#96)
* Split out notifying /sync requests and calculating sync responses

The logic for notifying /sync requests is about to get really
complicated as we optimise when to wake up requests, so split
out that code into a separate struct to isolate it and make
it easier to unit test.
2017-05-10 10:42:00 +01:00
Kegsay e226d564ec Fix a race with sync server integration tests (#95) 2017-05-09 15:58:31 +01:00
Kegsay 42564e8ed6 Factor out creating/deleting/writing to kafka topics (#94) 2017-05-09 09:05:05 +01:00
Kegsay 801b9246ce Add basic sync server integration test (#91) 2017-05-05 17:59:45 +01:00
Mark Haines a56f609b74 Update gomatrixserverlib (#90) 2017-05-05 17:43:42 +01:00
Mark Haines 0309035aad Update gomatrixserverlib (#89) 2017-05-05 16:19:48 +01:00
Robert Swain 9b7defd375 common/log: Always log to stderr 2017-04-21 02:11:24 +02:00
Robert Swain 27beb8a2b3 common/log: Re-colorize terminal log output 2017-04-21 02:06:06 +02:00
Robert Swain 9b7bf8cba8 common/log: Add license header 2017-04-21 01:45:18 +02:00
Robert Swain b33ed93dd6 Merge branch 'master' into rob/common-logging 2017-04-21 01:43:36 +02:00
Robert Swain d09a65abbe Add Apache Version 2.0 license and headers to all golang files 2017-04-21 00:40:52 +02:00
Robert Swain e281fc4530 syncserver: Rename to syncapi 2017-04-20 19:21:15 +02:00
Robert Swain d8b98cec44 client-api-proxy: roomserver-integration-tests: Fix renamed references 2017-04-20 19:21:15 +02:00
Robert Swain 0939ae3686 Rename dendrite-syncserver to dendrite-sync-api-server 2017-04-20 19:21:15 +02:00
Robert Swain 209ba3f6a3 Rename dendrite-roomserver to dendrite-room-server 2017-04-20 19:21:15 +02:00
Robert Swain 2278848922 Rename dendrite-clientapi to dendrite-client-api-server 2017-04-20 19:21:15 +02:00
Kegan Dougal 2a264db85e Don't tightloop if currentPos is 0 2017-04-20 18:01:45 +01:00
Kegsay cb5081b332 Yank out clientapi config options to env vars for now (#77) 2017-04-20 17:15:34 +01:00
Kegsay 2fadb9e409 Make it possible to point Riot at Dendrite (#74) 2017-04-20 17:11:53 +01:00
Mark Haines 43d1e94e75 Rename dendrite-sync-server to dendrite-syncserver for consistency (#76) 2017-04-20 16:54:22 +01:00
Robert Swain be47984c43 common/log: Always output timestamps as UTC 2017-04-20 17:16:16 +02:00
Robert Swain f65e26bc2a cmd: Configure logging before any log messages 2017-04-20 16:28:15 +02:00
Robert Swain 254e61f727 common/log: Switch to RFC3339 format with nanoseconds and trailing zeros 2017-04-20 16:28:15 +02:00
Robert Swain e8d2d61cc2 cmd: common/log: Always configure logging
When LOG_DIR or so is not specified, just configure the formatter.
2017-04-20 16:27:48 +02:00
Robert Swain 8010083026 dendrite/common: Move logrus configuration to common 2017-04-20 16:27:44 +02:00
Kegsay db428174d2 tool: Add client-api-proxy (#73) 2017-04-20 14:00:34 +01:00
Kegsay fad7e9541b /sync: Handle missing state events and return events in the correct order (#72) 2017-04-20 11:18:26 +01:00
Kegsay 92a0325266 Implement incremental sync (#71)
This has the following known shortcomings:
 - It doesn't handle missing events not in the from/to range.
 - It doesn't order events in the timeline correctly.
 - It doesn't handle invited/left rooms at all.
 - There is no intelligent wakeup logic: events for Bob will wake up Alice's stream.
2017-04-19 16:04:01 +01:00
Kegsay f18d935134 Factor out syncRequest parsing to its own file (#70)
Also ensure every sync response has a `next_batch` token by forcing
it in the constructor, as previously timeouts would not have a token.
2017-04-18 10:32:32 +01:00
Kegsay 53ec4a255b Calculate the complete /sync response (#69) 2017-04-13 16:56:46 +01:00
Kegsay 203e706b99 Move syncserver to its own directory (#66) 2017-04-12 16:06:26 +01:00
Kegsay da0fffcc19 Add /sync API response structs (#58) 2017-04-11 11:52:26 +01:00
Kegsay 9a266762b4 Return events based on 'since' value (#57) 2017-04-10 15:12:18 +01:00
Kegsay de9e3e5417 Add structs for HTTP long-polling (#56) 2017-04-07 14:32:42 +01:00
Kegsay c1c837516a Store current room state (#55) 2017-04-05 10:30:13 +01:00
Kegsay a3c66f7fa0 Write OutputRoomEvents into the database and remember the arrival order (#54) 2017-03-30 15:29:23 +01:00
Kegsay 2d2c7e7169 sync-server config: Read from a YAML file rather than hard-coded variables (#53) 2017-03-29 14:09:27 +01:00
Kegsay a423008987 Read roomserver output log and remember position across restarts (#52) 2017-03-29 14:05:43 +01:00
Kegsay a974b90ee9 Factor out Consumer from the roomserver (#50) 2017-03-27 10:00:28 +01:00
Mark Haines e3f3eb8f3d Add metrics for the various ways of calculating room state (#49) 2017-03-23 14:14:25 +00:00
Kegsay e347aa05fe Make a dendrite-sync-server binary (#48)
Currently the HTTP handlers are attached to an API mux which is created inside
Setup(). In the future, we'll want to split this out so a single mux can have
both sync and clientapi handlers attached to it.
2017-03-22 17:39:08 +00:00
Kegsay d6a03039dc Prefix 'dendrite-' onto dendrite binaries (#47)
This distinguishes them from tools like 'kafka-producer'
which are not required to run a dendrite server.

The prefix also groups together the dendrite binaries nicely
on `ls` as well as making it nicer to tab-complete.
2017-03-21 17:43:13 +00:00
Kegsay 83b9cfeb2b Move binaries to /cmd (#46) 2017-03-21 17:14:17 +00:00
Kegsay 346f5d6334 Use gomatrixserverlib.StateKeyTuple and helper functions for auth (#44)
* Update gomatrixserverib dep

* Use helper functions when creating rooms

* Use gomatrixserverlib.StateKeyTuple
2017-03-17 16:28:15 +00:00
Mark Haines 44c63254a7 Tool for writing messages to a single partition in kafka. (#43) 2017-03-17 16:27:45 +00:00
Mark Haines ac66d07177 Add option to stop processing input after a given number of messages (#38)
* Add option to stop processing input after a given number of messages
2017-03-17 13:01:07 +00:00
Kegsay a7390953dd Implement /state event sending (#41) 2017-03-17 11:21:52 +00:00
Mark Haines f2163c31c6 Implement state conflict resolution (#37)
* Implement state conflict resolution
2017-03-15 13:47:12 +00:00
Kegsay 742e4aca85 Factor out writing events to the roomserver input log (#40) 2017-03-15 13:36:26 +00:00
Kegsay 6739f65752 Implement event sending part of /rooms/$roomid/send/$type/$txnid (#39)
This involves:
 - Parsing the HTTP request
 - Requesting auth events from the roomserver via the Query API
 - Building the event
 - Doing auth checks on the event
 - Sending it to the roomserver input log
2017-03-15 11:22:40 +00:00
Mark Haines aa664f0eb7 Tool for creating room events (#36)
* Tool for creating room events
2017-03-14 17:05:02 +00:00
Kegsay b9a4551075 Create README.md 2017-03-13 15:56:26 +00:00
Kegsay 414ea314a6 Update gomatrixserverlib and use AuthEventProvider (#35) 2017-03-10 17:54:17 +00:00
Kegsay 8ccff1e40f Log fatal errors at error level and return generic 500s (#34)
Previously, the error responses:
 - were not valid matrix errors (no `errcode`)
 - returned the `err.Error()` message which may contain sensitive information.
 - did not get logged (at all, let alone set the level correctly).

Now the error responses:
 - return valid matrix errors (`M_UNKNOWN`)
 - return a generic "Internal Server Error" string
 - get logged at `ERROR` level.
2017-03-10 16:50:41 +00:00
Kegsay 2fcf6fd6eb Send /createRoom events to kafka (#33) 2017-03-10 16:19:23 +00:00
Kegsay 49ed708ca4 Create the initial set of state events for room creation (#32) 2017-03-10 11:32:53 +00:00
Mark Haines e667f17e14 Include the requested current state alongside the latest events in the query API. (#30)
* Return the requested portions of current state in the query API

* Use Unique from github.com/matrix-org/util

* rewrite bulkSelectFilteredStateBlockEntries to use append for clarity

* Add test for stateKeyTupleSorter

* Replace current with a new StateEntryList rather than individually setting the fields
2017-03-09 15:07:18 +00:00
Kegsay e82090e277 Update gomatrixserverlib dep and add basic /createRoom validation (#31) 2017-03-09 11:47:06 +00:00
Mark Haines 1d18da1189 Move the functions for reading room state to a separate package. (#29)
This should:
    1) Make the input package a bit cleaner.
    2) Allow use to reuse the state reading code from the query package.
2017-03-08 15:10:26 +00:00
Mark Haines 84682b33c9 Use Unique from github.com/matrix-org/util (#28)
* Update github.com/matrix-org/util

* Use Unique from github.com/matrix-org/util
2017-03-08 13:27:21 +00:00
Kegsay 8ba9d4af04 Add /createRoom endpoint and unmarshal HTTP body (#27) 2017-03-07 16:11:08 +00:00
Kegsay 5552e1f3a8 Extract access tokens from HTTP requests (#15) 2017-03-07 13:43:32 +00:00
Mark Haines 8084beb6f7 Add helper method for preparing lists of statements. (#25)
* Add helper method for preparing lists of statements.
2017-03-07 10:37:41 +00:00
Mark Haines 96fc9294cc Track the current state of the room (#24)
* Add method for calculating the room state after a list of events

* Fill out AddsState and RemovesState fields in the output event

* comment on return value

* Fix whitespace

* Clarify the meaning of LastEventIDSent

* Be more explicit on what the snapshot is

* Fix comments

* Comment on why we want the state after the previous events
2017-03-07 10:25:01 +00:00
Mark Haines 9a8a8aedcb Add a query API to the roomserver for getting the latest events in a room. (#23)
* Start implementing a query API for go using net/rpc

* Use a conventional JSON POST API rather than go net/rpc

net/rpc doesn't automatically handle reconnecting and we have better
logging and metrics infrastructure for monitoring HTTP apis.

* Implement the query API and add it to the integration tests

* Increase the timeout, travis seems to be a bit slow

* Clarify that state events are the things that are not returned if they are not requested

* Add utility function for converting arrays of numeric event IDs to pq Int64Arrays

* Warn people against requesting empty state keys by accident
2017-03-06 14:29:39 +00:00
Mark Haines 37e0b6c4c6 Add integration tests for roomserver to travis (#21)
* Add integration tests for roomserver to travis

* Try setting the dist trusty to see if that helps with postgres

* Pretty print the test input JSON

* Add comment as to why we are using trusty
2017-03-01 14:55:27 +00:00
Mark Haines 237b912cf5 Write to an output kafka log for new room events in the roomserver. (#20)
* Track which events have been sent and the last event sent for each room

* Write output events

* comment that an event could be sent twice
2017-02-27 11:25:35 +00:00
Kegsay 45d1e61a9d Update to use util.JSONResponse (#18) 2017-02-24 12:32:27 +00:00
Mark Haines a9b296c522 Get the SQL working, and make it more idempotent (#17)
* Get the SQL working, and make it more idempotent
2017-02-22 16:51:10 +00:00
Mark Haines c96d3eec0c Marshal and Unmarshal methods for roomserver input api (#16)
* Marshal and Unmarshal methods for roomserver input api

* Comments for why we don't json.marshal the InputRoomEvent directly

* More comments

* s/m.room.create/the first event/
2017-02-22 16:05:18 +00:00
Mark Haines 51f9741b34 Track which events are the latest events in the room. (#14)
* Track which events are the latest events in the room.
2017-02-21 14:50:30 +00:00
Kegsay 966d3b95b7 Add error package and some Matrix errors (#13)
Terse function names are used to make usage not stutter. For example:

```go
err := error.Forbidden("you shall not pass")
```

At the moment they are all the same fundamental `MatrixError` type. This will
be bad if we ever want to `switch` based on the kind of error. I'm hoping we
won't ever need to introspect into errors like this: ideally these errors would
be created purely for immediately being returned in an HTTP response.

`MatrixError` implements the `error` interface.
2017-02-20 17:20:49 +00:00
Kegsay 5c34caa1c1 Implement logging to file via dugong (#12) 2017-02-20 16:14:23 +00:00
Kegsay f1bb59d24a Use gorilla/mux to route HTTP requests (#11)
* Add basic routing based on matched paths
* Make /sync and /send use the right API paths
2017-02-20 15:41:29 +00:00
Mark Haines 450e0f1570 Move the SQL into separate files for each table. (#7)
* Move partition offsets sql to a separate file

* Move event types sql to a separate file

* Move event state key sql to a separate file

* Move room sql to a separate file

* Move events sql to a separate file

* Move event json sql to a separate file

* Move state snapshot sql to a separate file

* Move state block sql to a separate file
2017-02-15 14:43:19 +00:00
Mark Haines 39264cbf4b Calculate and store the state at each event (#6)
* Calculate and store the state at each event

* Use type aliases for numeric IDs
2017-02-15 11:05:45 +00:00
Mark Haines c72a44097f Move the auth checks to a separate file (#5) 2017-02-10 10:44:38 +00:00
Mark Haines fc4eb85379 Check that events pass authentication checks. (#4)
* Check that events pass authentication checks.

Record the list of events that the event passes authentication checks
against.
2017-02-09 16:48:14 +00:00
Mark Haines 600f56b4b8 Store outlier room events. (#3)
* Storage functions for event types
* Consume outlier room events
2017-02-07 17:20:05 +00:00
Kegan Dougal b04dfae9c5 Add comments 2017-02-03 16:32:55 +00:00
Kegan Dougal 225cce9c77 Prefix /api as per standards 2017-02-03 16:28:11 +00:00
Kegan Dougal aafaf6ede6 Add stub clientapi webserver with readers/writers packages
As per RL discussion with Mjark
2017-02-03 16:05:46 +00:00
Mark Haines 41c6a3737e A kafkaesque room event consumer for the roomserver. (#1)
* A kafkaesque room event consumer for the roomserver.

Implement the main input loop for the roomserver.

It will receive events from a kafkaesque event source and track
where it is in the stream.

It currently does nothing with the events it consumes.
2017-02-03 13:52:32 +00:00