Commit graph

342 commits

Author SHA1 Message Date
Erik Johnston
b72142ace5 Add vet linter (#240) 2017-09-20 13:40:22 +01:00
Erik Johnston
8a36a7e89b Add errcheck linter (#238)
* Add errcheck linter

* Add config for 'fast' linters

* Fix errcheck lint stuff

* Fix errcheck lint
2017-09-20 10:59:19 +01:00
Mark Haines
b91b3e729a Use a shared PostgreSQL sequence to generate ids. (#237)
* Use a shared PostgreSQL sequence to generate ids.

Share an auto incrementing sequnce between the account data and
the room event table.
This means that account data updates can be received independantly of
room events updates.

This should give some basic support for fixing #212

* Remove redundant 'primary key'

* Re-number the SQL arguments

* Fewer lies in comments
2017-09-19 17:15:46 +01:00
Mark Haines
fbc4477be0 Use a read-only snapshot transaction for calculating sync responses (#236)
* Use a read-only snapshot transaction for calculating sync responses

* gb vendor update github.com/lib/pq
2017-09-19 16:22:02 +01:00
Erik Johnston
08b9940dde Make media repo use error rather than jsonErrorResponse (#235)
* Make media repo use error rather than jsonErrorResponse

* Update comments

* gb vendor github.com/pkg/errors

* Fixup error formats
2017-09-19 11:40:21 +01:00
Mark Haines
856bc5b52e Add context to the syncapi database (#234) 2017-09-18 16:52:22 +01:00
Mark Haines
238646ee3c Add contexts to device database (#233)
* Add contexts to device database

* Remove spurious whitespace
2017-09-18 15:51:26 +01:00
Mark Haines
e28ee27605 Add context to the account database (#232) 2017-09-18 14:15:27 +01:00
Mark Haines
5ada8872bb Add context to the federationsender database (#231) 2017-09-18 14:15:17 +01:00
Mark Haines
dc5dd4c5d2 Add contexts to the public rooms database (#230) 2017-09-14 14:46:56 +01:00
Mark Haines
bfcce5bd21 Add contexts to the roomserver storage layer (#229)
* Add contexts to the roomserver storage layer

* Fix rooms_table
2017-09-13 16:30:19 +01:00
Mark Haines
3133bef797 Add contexts to the internal roomserver APIs (#228) 2017-09-13 13:37:50 +01:00
Brendan Abolivier
79adba43f0 Append target profile to invite events created from a 3PID invite (#227)
* Append target profile to invite events created from a 3PID invite

* Don't redeclare err

* Add check on invited server
2017-09-13 13:31:46 +01:00
Brendan Abolivier
4fbafc35fe Specify HTTP methods for public APIs (#226)
* Specify HTTP methods for the client API

* Specify HTTP methods for the federation API

* Specify HTTP methods for the media API

* Specify HTTP methods for the sync API
2017-09-13 11:08:11 +01:00
Mark Haines
029e71828a Add context.Context to the federation client (#225)
* Add context.Context to the federation client

* gb vendor update github.com/matrix-org/gomatrixserverlib
2017-09-13 11:03:41 +01:00
Brendan Abolivier
086683459f Implement federation route PUT /exchange_third_party_invite (#224)
* Add comment

* gb vendor update github.com/matrix-org/gomatrixserverlib

* Add handler for the exchange_third_party_invite endpoint

* Doc

* Use SendEvents to send the invite to the roomserver

* Add missing error check

* Add checks
2017-09-12 17:15:13 +01:00
Mark Haines
82c82a3412 Explain when we should use SendEvents instead of SendInvite.
Because it's easy to get confused between the two methods.
2017-09-12 16:54:37 +01:00
Brendan Abolivier
f1fce55697 Add config and checks for trusted ID servers (#206)
* Add config for trusted ID servers

* Add new error

* Implement check for trusted ID server

* Complete unfinished comment

* Make comment more explicit in the config file

* Use go standard errors in membership.go

* Use standard errors instead of JSON responses in threepid

* Doc errors

* Remove unused parameter
2017-09-11 19:18:19 +01:00
Brendan Abolivier
28346b39e8 3PID invite exchange over federation (#222)
* Use federation to auth the event if the server isn't in the room

* Use MakeAPI for 3pid onbind handler as it isn't a standard federation request

* Error check

* Temporarily disable tests

* Fix return on 3PID invite

* Re-enable tests

* Remove useless else

* gb vendor update github.com/matrix-org/gomatrixserverlib

* gb vendor update github.com/matrix-org/gomatrixserverlib

* Implement same behaviour as synapse

* Fix condition and array initialisation

* Log errors on iteration and throw one if no server could be reached

* Fix err not being initialised

* Fix lint

* Fix import path
2017-09-11 19:08:24 +01:00
Mark Haines
6cb9d900b9 Fix response to federation /invite to match the format expected by synapse (#221)
* Fix response to /invite to match the format expected by synapse

* gb vendor update github.com/matrix-org/gomatrixserverlib

* Use gomatrixserverlib.RespInvite

* gb vendor update github.com/matrix-org/gomatrixserverlib
2017-09-11 18:07:12 +01:00
Mark Haines
5740cb3e58 Redact the event before verifying its signature
... in the federation invite API.
2017-09-11 15:47:50 +01:00
Brendan Abolivier
4d1d503d43 Processing of pending invites on 3PID binding (#218)
* Add missing file headers

* Move the ID server's signatures verification to common

* Allow verification without specifying a server name

* Add third-party structs to membership events content

* Add processing of 3PID onbind requests

* Use reference for third party invite data

* Fix return arguments order

* Revert "Move the ID server's signatures verification to common"

This reverts commit 93442010316ce71a77ac58ffd3613754ce8fe969.

* Revert "Allow verification without specifying a server name"

This reverts commit fd27afbf82eac50fe9f7b83b26cfce3c66d530d2.

* Remove checks that are already occurring in gomatrixserverlib

* Change return type of createInviteFrom3PIDInvite

* Add doc, add checks in fillDisplayName

* Use MakeFedAPI

* Invert condition

* Use AuthEvents to retrieve the 3PID invite

* Update comment

* Remove unused parameter

* gb vendor update github.com/matrix-org/gomatrixserverlib
2017-09-08 15:17:12 +01:00
Erik Johnston
fad997303b Turn on 'unparam' linting (#216)
* Turn on 'unparam' linting

* Increase deadline

* Fix mediaapi integeration test
2017-09-07 12:50:39 +01:00
Erik Johnston
ea80ffb3af Add get event federation API (#209)
* Add get event federation API

* Handle err correctly

* Add missing readers

* Ignore gocyclo lint for routing function

* Add todo

* Fix styling a bit
2017-09-06 12:38:22 +01:00
Erik Johnston
2dcb3a11a5 Use gometalinter (#210)
* Remove unused struct field

* Ignore unused test data

* Remove unused variables

* Remove deadcode

* Fix up vetshadow warnings

* Convert to using gometalinter

* Update travis

* Use vendored versions of gometalinter

* Make gometalinter install its stuff

* Vendor misspell
2017-09-05 17:40:46 +01:00
Mark Haines
643d05b157 Add function for making authed federation APIs (#208) 2017-09-04 13:14:01 +01:00
Brendan Abolivier
960af3d628 Implement client API routes for 3PID handling (#205)
* Create package for handling 3pid processes and move invite processing there

* Add database table and functions for tracking 3PIDs

* Add structures and functions to interact with an ID server

* Add handlers for 3PIDs management

* Fix 3PIDs retrieval sending null if no 3PID known for a user

* Include medium in database requests and function calls

* Publish an association if it has been validated and requested

* Add TODO markers for tursted ID server check

* Use a structure instead of a map to represent a 3PID
2017-09-01 10:13:10 +01:00
Richard van der Hoff
8c2e6273e3 Add command to generate test keys/certs (#204)
Add `generate-keys` command, which can be used to generate TLS keys/certs, and,
more usefully, matrix signing keys.
2017-08-31 12:28:58 +01:00
Richard van der Hoff
fc51f72bf9 Avoid swallowing error from kafka (#203) 2017-08-31 12:27:53 +01:00
Brendan Abolivier
2183712e04 Process requests for 3PID invites (#201)
* Structure for 3PID invite

* Generate invite from 3PID known by ID server

* Load user profile in a separate function

* Generate m.room.third_party_invite if the ID server doesn't know the 3PID

* Fix URLs to the spec in comments

* Move third-party invites to a separate package and doc' it

* Handle non-OK status codes on lookup

* Send display name to identity server when asking to store an invite

* Remove join response structure

* Change the way some variables are declared or passed as argument

* Use gomatrixserverlib.Base64String instead of the builtin base64 package

* Don't copy the public keys array
2017-08-29 15:17:26 +01:00
Brendan Abolivier
685e056ab3 Membership viewing API when user left the room (#194)
* Implement case where user left the room

* Filter by membership event

* Move the logic from the storage to the query API

* Fix check on state entries iteration

* Remove aliases methods from query API

* Use structure for response to match with the spec

* Remove filtering on /members and implement /joined_members
2017-08-24 16:00:14 +01:00
Mark Haines
9c954501a2 Implement /federation/v1/invite/{roomID}/{eventID} (#197)
* Implement /federation/v1/invite/{roomID}/{eventID}

* Use NotJSON instead of BadJSON when the JSON couldn't be decoded
2017-08-23 15:13:47 +01:00
Mark Haines
17c60759c4 Add query API for listing active invites (#196)
* Add query API for listing active invites

This lists the invites for a user in a room that could be used to
join the room over federation.

* s/Lookup/Look up/

* Fix implements comments
2017-08-23 15:08:48 +01:00
Brendan Abolivier
166ac9d092 Fix sync not returning on room join (#195)
* Use BuildEvent method on room join

* Fix building the list of room members in the sync notifier

* Fix building the list of room members in the sync notifier

* Rephrase comment
2017-08-22 14:14:37 +01:00
Brendan Abolivier
b15ce900ab Implement public rooms APIs (#185)
* Move events contents to common

* Basic database structure

* Complete database update

* Support visibility update and retrieval

* Add HTTP methods for visibility update and retrieval

* Add the database for the new component

* Add a listener for the new component

* Fix attribute update statements

* Create public rooms component

* Fix failing test

* Add roomserver consumer

* Fix a bug in aliases creation

* Add a check on type

* Implement public rooms directory

* Use auth API for visibility update

* Support filtering

* Add component to monolith

* Various fixes

* Fix computation of next public rooms batch

* Retrieve state events from the roomserver query API + avoid dupes on join

* Split update of string or boolean attribute in two separate functions

* Use event type to detect duplicate joins

* Improve the joined members counter computation

* Use event.RoomID()
2017-08-22 11:12:51 +01:00
Mark Haines
808c2e09f6 Make txn *sql.Tx arguments optional everywhere using a utility function (#191)
* Make txn *sql.Tx arguments optional everywhere using a utility function

* Clarify that if the txn is nil the stmt will run outside a transaction
2017-08-21 17:20:23 +01:00
Mark Haines
57b7097368 Add input API for adding invites to the roomserver. (#187)
* Add input API for adding invites to the roomserver.

This API handles invites received over federation that occur outside of a room.

* Add some docstring for withTransaction

* Use a nicer pattern for wrapping transactions

* Fix MembershipUpdater method to not commit the transaction before returning it

* Use the Transaction interface from common
2017-08-21 16:37:11 +01:00
Brendan Abolivier
5950293e79 Membership viewing API (#174)
* Basic memberships retrieval

* Change the way the memberships are saved in the client API database

* Retrieve single membership

* Get memberships only if the user is or has been in the room

* Check server name on room ID instead of user ID

* Save the join membership event and updates it when necessary

* Membership events retrieval + update on leave

* Implement the API on the roomserver and client API server

* Fix comments

* Remove the functions and attributes used before the new query API

* Explicitely state what we return in query

* Remove tab
2017-08-21 16:34:26 +01:00
Mark Haines
81179a0595 Fix integration test proxy certificate arguments (#190) 2017-08-21 16:23:59 +01:00
Brendan Abolivier
35df4767c6 Remove defaults for TLS so the proxy can more easily serve over HTTP (#189) 2017-08-21 14:25:47 +01:00
Mark Haines
efbc14f7b9 vars["txnID"] is always empty for state events (#188) 2017-08-18 16:10:28 +01:00
Brendan Abolivier
f607ef29c2 Add a route matching the trailing slash on the state event sending route (#182)
* Add a route matching the trailing slash on the state event sending route

* Use single route for both cases

* Use synapse regexp
2017-08-18 15:33:40 +01:00
Mark Haines
877ea5cb62 Remove StopProcessingAfter from the roomserver consumer as it is unused (#186) 2017-08-18 11:33:10 +01:00
Mark Haines
ba8b5d8bf9 Fix kafka consumer setup in monolith. (#184)
We can't consume the same topic on a single kafka consumer more than
once. So when using kafka we have to create a new consumer for each
component in the monolith.
2017-08-16 16:55:34 +01:00
Mark Haines
c27d1fdfb4 Optionally use naffka in the monolithic server (#183)
* dependency injection for the kafka consumers/producers

* Optionally use naffka in the monolithic server

* remember to call setupKafka()

* tweak imports

* fix integration tests

* Add use_naffka to the example config

* Update comment on the listen APIs
2017-08-16 13:36:41 +01:00
Mark Haines
2071387f3c Add tables for tracking the state of invites to the room server. (#165)
* Storage functions for invite events

* Add table for tracking membership state

* More stuff

* More stuff

* Use utility methods from gomatrixserverlib, rather than reimplementing them

* More stuff

* Return string rather than pointer to string

* Update gomatrixserverlib

* Use HTTP API for roomserver input.

* Use synchronous HTTP API for writing events to the roomserver

* Remove unused config for kafka topic

* Add new output types to roomserver for invites

* Write membership updates

* Separate filtering from pairing up changes in membershipChanges

* Fix SQL

* Fix SQL

* Namespace the tables

* Fix SQL

* Use clearer names for some of the variables

* Rename senderID for consistency

* Restructure update membership

* Comments

* More comment

* Fix SQL

* More comments

* Assign state keys inside the transaction

* Comment on the purpose of the latestEventsUpdater

* Comment on the purpose of updateMembership

* Remove duplicate fields from stateChange

* Attempt to rewrite comment in 'english'

* More comments

* Fix comment

* Comment

* more comments
2017-08-08 16:38:03 +01:00
Brendan Abolivier
c35803c9d8 Add TLS (https) support to the monolithic server (#180)
* Add HTTPS listen and serve, and move both listen and serve to goroutines

* Add some log so we know the initialisation went well
2017-08-07 13:39:53 +01:00
Mark Haines
4d1504ee93 Add prefixes to namespace the SQL tables. (#177)
* Add prefixes to namespace the SQL tables.

This means that multiple components can share a single database schema
without colliding with each other.

Once this lands it will be possible to run a single monolithic dendrite
against a single postgresql schema.

Hopefully this will make trivial deployments and development easier.

* Comment
2017-08-07 11:51:46 +01:00
Brendan Abolivier
8ccc5d108b Implement membership APIs (#171)
* Implement membership endpoints

* Use FillBuilder when possible

* Fix typo in membership event content

* Fix state key invite membership events not being correctly set

* Set membership content to match the profile of the user in state_key

* Move event building and rename common function

* Doc getMembershipStateKey

* Check if user is local before lookin up their profile
2017-08-04 16:32:10 +01:00
Brendan Abolivier
03dd456b47 Fix bad initialisation of sync API notifier in monolith (#176) 2017-08-04 14:54:08 +01:00
Mark Haines
1a28cf9767 Add a monolithic server that combines all the components into one (#175)
* Add a monolithic server that combines all the components into one

* Review comments

* Expose the matrix APIs directly
2017-08-04 13:12:36 +01:00
Mark Haines
4975eb9074 Move setting up the api mux to outside the routing.Setup functions. (#173)
This makes it possible to setup all the component APIs on a single http
listener which is necessary if we want to combine all the components
into a single monolith.
2017-08-03 15:10:39 +01:00
Mark Haines
3b07633326 Simplify preparing the roomserver SQL (#172) 2017-08-02 16:50:34 +01:00
Brendan Abolivier
0fbb8b7824 Make account data sync incremental (#170)
* Clean roomserver consumer

* Make account data sync incremental

* Use a different name for the sync AD table

* Improved error logging

* Created missing topic in tests

* Add client API topic to tests

* Add client API topic to common

* Move data batch retrieval

* Add database index for data retrieval

* Fix typo in table name

* Fix indentation
2017-08-02 16:21:35 +01:00
Brendan Abolivier
7d17df6f51 Add local room join (#169)
* Add local room join

* Update TODO
2017-07-28 15:29:12 +01:00
Brendan Abolivier
34165c7761 Add default config file name for media API and room servers (#168) 2017-07-28 11:32:16 +01:00
Brendan Abolivier
c7e36ba2a1 Implement room aliases (#167)
* Add database functions to interact with aliases

* Save room alias

* Check if alias already exists

* Implement local alias lookup

* Implement alias removal

* Move alias API

* Handle case with no alias set for a room

* Apply requested changes
2017-07-28 11:31:43 +01:00
Brendan Abolivier
3e394e9e21 Save and retrieve account data (#166)
* Save function for account data

* Fix upsert + add empty routes and function

* Save account data

* Retrieval functions

* Implement retrieval in /sync

* Fix arrays not correctly initialised

* Merge account data retrieval functions

* Request DB only once per request

* Initialise array

* Fix comment
2017-07-26 14:53:11 +01:00
Brendan Abolivier
6d073dcf9f Propagate profile update through rooms (#163)
* Use gomatrixserverlib function to split user ID

* Propagate profile update via m.room.member events

* Send profile data on room join

* Send profile data on room creation

* Rename variable

* Move membership update to roomserver consumer

* Improve iteration

* Move event update from client API server to sync API server

* Change the way buildMembershipEvents is called

* Forbid update of someone else's profile

* Use gomatrixserverlib method

* Fix depth and previous events not being set

* Fix wrong removal in latest commit

* Update all events instead of only memberships

* Handle case where there is no state key

* Fix test
2017-07-25 16:10:59 +01:00
Brendan Abolivier
ce311ce0fe Improve room creation (#164) 2017-07-20 13:06:14 +01:00
Mark Haines
e6d77d6bde Use HTTP API for roomserver input. (#161)
* Use HTTP API for roomserver input.

* Use synchronous HTTP API for writing events to the roomserver

* Remove unused config for kafka topic

* Tweak comments
2017-07-18 13:40:03 +01:00
Brendan Abolivier
d9b8e5de45 Keep track of membership in Client API (#159)
* Saving memberships

* Removed unused index

* Removed useless log

* Fixed membership not being saved on the right conditions + added membership removal

* Updated outdated comment

* Use server lib method + check server name + use new roomserver API

* Better handling of events from the room server

* Fixed membership removal

* Corrected indentation

* Fix tests (hopefully)

* Replace broken kafka mirror

* Apply requested changes on database management

* Remove useless check and function

* Moved memberships update to the database package

* Use new common function

* Remove useless function
2017-07-17 18:10:56 +01:00
Mark Haines
b06d1124f7 Factor out runTransaction to common code (#162) 2017-07-17 17:20:57 +01:00
Mark Haines
d3a29b7816 Add a HTTP API for roomserver input (#158)
* Add a HTTP API for roomserver input

* Fix integration tests
2017-07-13 11:41:30 +01:00
Mark Haines
4e7862e3b9 Make the room server RPC API json formats more consistent with our other APIs (#156) 2017-07-12 14:13:10 +01:00
Mark Haines
cea9e31723 Make the roomserver output format more flexible (#155)
* Make the roomserver output format more flexible

* Fix the sync server integration testdata

* Fix roomserver testdata

* Actually fix the sync server tests

* Fix typo
2017-07-12 10:46:29 +01:00
Brendan Abolivier
7d36ca03af Implemented logout (#154) 2017-07-11 16:04:34 +01:00
Brendan Abolivier
355ab5eedf Notify profile update (#153)
* Profile retrieval

* Saving avatar (without propagating it)

* Saving display name (without propagating it)

* Getters for display name and avatar URL

* Doc'd

* Introduced new Kafka topic and producer

* Updated config with new kafka topic

* Switched to samara producer and now sending messages

* Doc'd

* Put kafka update after the database insert

* Doc'd profileUpdate structure

* Removed unused parameter

* Moved user updates producer to clientapi/producers
2017-07-11 14:14:06 +01:00
Brendan Abolivier
1efbad8119 Profile API (#151)
* Profile retrieval

* Saving avatar (without propagating it)

* Saving display name (without propagating it)

* Getters for display name and avatar URL

* Doc'd

* Remove unused import

* Applied requested changes

* Added auth on PUT /profile/{userID}/...

* Improved error handling/reporting

* Using utils log reporting

* Removed useless checks
2017-07-10 14:52:41 +01:00
Mark Haines
69c29172c3 Use utility methods from gomatrixserverlib. (#152)
* Use utility methods from gomatrixserverlib, rather than reimplementing them

* Return string rather than pointer to string

* Update gomatrixserverlib
2017-07-07 14:11:32 +01:00
Brendan Abolivier
b13cbb18fb Response to createRoom is now the room's ID (#149) 2017-07-06 11:44:15 +01:00
Mark Haines
b2f9b8c641 Update the example configs with the new unified config (#146)
* Update the example configs with the new unified config

* Review comments

* Use an int for the config version
2017-07-05 13:10:28 +01:00
Mark Haines
8f1dca4e74 Add a component for sending event to remote matrix servers using fede… (#148)
* Add a component for sending event to remote matrix servers using federation

* Use named constants

* Comment on why we combine the deltas

* Comments

* Copyright headers!

* Fix the copy+Pasted comment

* Add tests for combineDeltas

* Review comments

* More review comments

* More comments

* Even more comments

* Name the mutexes

* Document what the mutexes are protecting

* Simplify the SQL, and more comments
2017-06-28 16:10:17 +01:00
Mark Haines
54e7e3041b Set depth of events and whether they need to be federated. (#145)
* Set depth of events and whether they need to be federated.

Set the depth of each new event to one greater than the maximum depth
of it's direct ancestors.

Add a flag to each event passing through the roomserver that tells us
whether the event needs to be sent over federation.

We do this by passing the name of the server to send the event as.
We will need this capability if we want to support vhosting as it is
not possible to tell from the event alone which server to send it as.

(The reason for this is that sometimes a event needs to be sent on
behalf of a different remote matrix server)

* Update roomserver integration tests

* More comments

* Constants and comments

* Fix comments

* Document the int64 returned by LatestEventIDs
2017-06-27 15:28:44 +01:00
Mark Haines
572f6c399d Include the state before the event in roomserver output (#143)
* Include the state before the event in roomserver output

* Fix roomserver integration test

* Remove VisibilityEventIDs from the JSON

* More comments

* Remove spurious TODO
2017-06-27 13:20:04 +01:00
Mark Haines
524475f8a3 Stub APIs needed to let riot join a room (#147)
* Stub APIs needed to let riot join a room

* Fix comments
2017-06-27 12:37:25 +01:00
Matthew Hodgson
eb029116b0 trivial fix to actually parse fed-api-server's CLI args 2017-06-25 01:20:04 +01:00
Matthew Hodgson
cdd1fdcd3d trivial typo in usage output 2017-06-25 01:06:07 +01:00
Mark Haines
e67f9401be Replace the cmd specific config with common config. (#144)
* Move all the dendrite config in to a single place

* Add tests for config parsing

* replace syncserver config with common config

* Replace client API config with common config

* Replace federation API config with common config

* Replace media api config with common config

* Replace room server config with common config

* Remove unused readKey function

* Fix the integration tests

* Comment on hardcoding roomserver to HTTP

* Add a method for getting RoomServerURL

This moves the hardcoding of HTTPs into one place.
2017-06-19 15:21:04 +01:00
Mark Haines
3fbe728d94 Add utility functions for generating dendrite configs for integration tests (#142) 2017-06-19 12:56:55 +01:00
Mark Haines
c6e59ca967 Add a single config file format for dendrite. (#141)
* Add a single config file format for dendrite.

This config file combines all the configuration needed by the
dendrite components into a single file.

* Resolve relative paths against the current working directory

* Review comments

* more review comments!

* Document what the purpose of the version field is
2017-06-16 16:52:03 +01:00
Mark Haines
5e490e773f Add ineffassign to the pre-commit hooks (#139)
* Add ineffassign to the pre-comiit hooks

* Fix travis.yml

* Only apply ineffassign to the /src directory
2017-06-12 19:52:03 +01:00
Mark Haines
472155837b Add misspell and gofmt simplify to the pre-commit hooks (#138) 2017-06-12 18:30:47 +01:00
Mark Haines
7cbdab30f4 Add a database for storing the server keys (#137)
* Add a database for storing the server keys

* Tweak wording, and comment on the resolution of the timestamp

* Update gomatrixserverlib
2017-06-09 18:07:34 +01:00
Robert Swain
6eae6f7598 Foundation for media API testing (#136)
* cmd/mediaapi-integration-tests: Add foundation for testing

* common/test: Add some server init and client request utilities

* common/test/client: Handle timed out requests for tests that passed

* cmd/syncserver-integration-tests: Port to new common/test infra

* common/test/client: Remove stray debug print

* cmd/mediaapi-integration-tests: Simplify slice initialisation

* cmd/mediaapi-integration-tests: Simplify getMediaURL argument

* cmd/mediaapi-integration-tests: Make startMediaAPI return listen address

* common/test/client: Fix uninitialised LastRequestErr

* common/test/server: Remove redundant argument

* common/test/server: Add StartProxy to create a reverse proxy

* cmd/mediaapi-integration-tests: Add proxies in front of servers

This is needed so that origins can be correctly configured and used for
remote media.

* travis: Enable media API integration tests

* travis: Build the client-api-proxy for media tests

* common/test/client: Don't panic on EOF in CanonicalJSONInput

* cmd/mediaapi-integration-tests: Add upload/download/thumbnail tests

* mediaapi/thumbnailer: Store thumbnail according to requested size

* cmd/mediaapi-integration-tests: Add totem.jpg test file

* cmd/client-api-proxy: Optionally listen for HTTPS

* common/test/client: Do not verify TLS certs for testing

We will commonly use self-signed certs.

* cmd/mediaapi-integration-tests: Make HTTPS requests

* cmd/mediaapi-integration-tests: Log size and method for thumbnails

* mediaapi/thumbnailer: Factor out isThumbnailExists

Appease gocyclo^w^w simplify

* mediaapi/thumbnailer: Check if request is larger than original

* travis: Install openssl and generate server.{crt,key}

* cmd/mediaapi-integration-tests: Add valid dynamic thumbnail test

* cmd/mediaapi-integration-tests: Document state of tests

* cmd/mediaapi-integration-tests: Test remote thumbnail before download

This ordering also exercises the cold cache immediate generation of a
size configured for pregeneration.

* travis: Explain openssl key+cert generation

* common/test/server: Clarify postgresContainerName
2017-06-08 15:40:51 +02:00
Mark Haines
b184a48897 Fetching missing state from the roomserver. (#135)
* Fetching missing state from the roomserver.

Whenever the syncserver receives an event from the room server that adds
state that isn't in the syncserver's local database it should fetch
those state events from the roomserver.

* Fix append

* Put comment back

* Comments

* s/addsStateEvents/lookupStateEvents/

* Fix spelling

* Include the stream position that a state event was added at in the current state tables

* Fix comment

* Review comments
2017-06-07 16:35:41 +01:00
Mark Haines
515cce1a45 Add support for receiving room events over federation. (#130)
* Add API for querying events by ID.

* Fix tense

* Start implementing federation ingress

* More stuff

* Hook up federation event receiving

* Handle the case where we are missing state

* Fix docstring and comments

* Fix infinite loop when printing unknownRoomError
2017-06-07 14:32:53 +01:00
Robert Swain
2d202cec07 mediaapi: Add thumbnail support (#132)
* vendor: Add bimg image processing library

bimg is MIT licensed. It depends on the C library libvips which is LGPL
v2.1+ licensed. libvips must be installed separately.

* mediaapi: Add YAML config file support

* mediaapi: Add thumbnail support

* mediaapi: Add missing thumbnail files

* travis: Add ppa and install libvips-dev

* travis: Another ppa and install libvips-dev attempt

* travis: Add sudo: required for sudo apt* usage

* mediaapi/thumbnailer: Make comparison code more readable

* mediaapi: Simplify logging of thumbnail properties

* mediaapi/thumbnailer: Rename metrics to fitness

Metrics is used in the context of monitoring with Prometheus so renaming
to avoid confusion.

* mediaapi/thumbnailer: Use math.Inf() for max aspect and size

* mediaapi/thumbnailer: Limit number of parallel generators

Fall back to selecting from already-/pre-generated thumbnails or serving
the original.

* mediaapi/thumbnailer: Split bimg code into separate file

* vendor: Add github.com/nfnt/resize pure go image scaler

* mediaapi/thumbnailer: Add nfnt/resize thumbnailer

* travis: Don't install libvips-dev via ppa

* mediaapi: Add notes to README about resizers

* mediaapi: Elaborate on scaling libs in README
2017-06-07 01:12:49 +02:00
Robert Swain
def49400bc cmd/client-api-proxy: Add proxy for media API (#133) 2017-06-05 15:55:04 +02:00
Mark Haines
721c2df484 Clean up syncapi storage to match the coding style set by roomserver storage (#131) 2017-06-05 10:37:04 +01:00
Mark Haines
ef7b934d51 Add Query API for querying events by ID (#129) 2017-06-02 14:32:36 +01:00
Mark Haines
1b6e06aa59 Revert "Add API for querying events by ID. (#127)" (#128)
This reverts commit ce82158abb.
2017-06-02 14:24:13 +01:00
Mark Haines
ce82158abb Add API for querying events by ID. (#127)
* Add API for querying events by ID.

* Fix tense

* Start implementing federation ingress

* More stuff

* Hook up federation event receiving

* Fix comments

* Comment on the order of the arrays
2017-06-02 11:19:34 +01:00
Robert Swain
7244fc15b3 media/writers/download: Make use of CreateMediaDownloadRequest from lib 2017-06-01 17:57:05 +02:00
Robert Swain
a398cdd193 mediaapi/writers/download: Use mime.ParseMediaType to parse Content-Disposition 2017-06-01 16:04:41 +02:00
Robert Swain
2d822c57f9 mediaapi/writers/download: Use DNS address as not all HSes have SRV 2017-06-01 15:30:36 +02:00
Robert Swain
cae309fa26 mediaapi/writers/download: Handle panic to ensure waking of goroutines
If the active request were to panic, we need to ensure all the waiting
goroutines get woken up.
2017-06-01 14:54:59 +02:00
Robert Swain
ce21b32d38 mediaapi/writers/download: Wrap broadcast call in closure to re-evaluate args 2017-06-01 14:44:00 +02:00
Robert Swain
4457ebddca mediaapi/writers/download: Rework remote file download synchronisation
Avoid locking around db requests by only locking around active requests
and always creating an active request if there is none. A nice side
effect of this is that if many parallel requests for remote media come
in (a common case) then only one database query is made for the duration
of the query.
2017-06-01 12:32:15 +02:00
Robert Swain
a3b1c7535a mediaapi/writers/download: Remove unnecesary Unlock, Lock after Cond.Wait 2017-06-01 08:39:35 +02:00
Robert Swain
923013aa55 mediaapi: Add support for downloading of remote files 2017-05-31 17:56:11 +02:00
Robert Swain
07c821daec mediaapi/writers: Fix InternalServerError responses 2017-05-31 17:41:42 +02:00
Robert Swain
2e013e3408 mediaapi/writers/download: Remove unnecessary closeConnection() 2017-05-31 15:39:19 +02:00
Robert Swain
bd96d99a3a mediaapi/writers/download: 500 ISE if disk and db file size differ 2017-05-31 15:08:00 +02:00
Robert Swain
e33438a37e mediaapi/writers/upload: Add standard fields to logger
So that they apply to all log messages thereafter
2017-05-31 14:54:10 +02:00
Robert Swain
5922ad383d mediaapi/writers/upload: Don't use deprecated sql.ErrNoRows check 2017-05-31 14:52:45 +02:00
Robert Swain
f0c717b0a1 mediaapi/writers/download: Escalate corrupt db/file cases to errors 2017-05-31 14:33:49 +02:00
Robert Swain
9606ea28ce mediaapi/writers/download: Always log origin and mediaID 2017-05-31 14:30:57 +02:00
Robert Swain
a45f008c12 mediaapi/storage: Don't leak sql.ErrNoRows out of storage package 2017-05-31 14:29:28 +02:00
Robert Swain
a936ad5063 mediaapi/writers/download: Add local download support 2017-05-31 13:46:21 +02:00
Robert Swain
3f7ef7690b Merge pull request #124 from matrix-org/rob/media-api-upload
/upload handler and storage database
2017-05-31 12:32:10 +02:00
Robert Swain
a0eae6922d mediaapi/writers: Remove unnecessary fmt.Sprintf 2017-05-31 07:12:22 +02:00
Robert Swain
08d1eb9669 mediaapi/upload: Improve HTTP status codes for error cases 2017-05-31 07:11:21 +02:00
Robert Swain
523303277e mediaapi/storage: Refer to RFC instead of golang for base64 format 2017-05-31 07:11:00 +02:00
Robert Swain
61329ee380 mediaapi/fileutils: Comment truncation of data when reading 2017-05-31 07:10:01 +02:00
Robert Swain
a4300eefc4 mediaapi/fileutils: Fix and comment os.IsNotExist bug 2017-05-31 07:08:21 +02:00
Robert Swain
63ccd770c6 mediaapi/fileutils: Use filepath not path for filesystem paths 2017-05-31 07:07:48 +02:00
Robert Swain
0ca2931b62 mediaapi/fileutils: Change path schema to put file in subdir of hash 2017-05-31 07:06:42 +02:00
Robert Swain
8c6f30eadc mediaapi/config: Remove obsolete proxying comment and add default comment 2017-05-31 07:05:07 +02:00
Kegsay
50aacd4f3c Hook up registration/login APIs and implement access token generation (#122) 2017-05-30 17:51:40 +01:00
Mark Haines
65b66a6452 Add API for querying the state after a list of events (#123)
* Add API for querying the state after a list of events

* Implement state after events API

* Actually implement the API

* Reorder .travis.yml

* Comments

* Use common.MakeAPI
2017-05-30 17:44:31 +01:00
Robert Swain
6fc6499848 mediaapi/fileutils: Remove obsolete error variables 2017-05-26 17:50:16 +02:00
Robert Swain
4f2d9a3b69 mediaapi/storage: Simplify descriptions 2017-05-26 17:44:43 +02:00
Robert Swain
9678cb6ea1 mediaapi/writers/upload: Simplify storeFileAndMetadata description 2017-05-26 17:42:08 +02:00
Robert Swain
9ecf620ad9 mediaapi/writers/upload: Factor out doUpload from Upload 2017-05-26 17:34:58 +02:00
Robert Swain
d83359dd51 mediaapi: Remove unnecessary ContentDisposition
Content-Disposition is only used for communicating the filename. It does
not need to be stored in the database as we have upload_name anyway. It
does not need to be in types.MediaMetadata and does not need to be
logged.
2017-05-26 17:24:13 +02:00
Robert Swain
731c10a418 mediaapi/fileutils: Clean up
Reorder functions to have public API functions in alphabetical order at
the top, internal package functions at the bottom in call order.

Use RawURLEncoding to avoid padding the hash with '='.

Use stronger types for paths in public API.

Simplify comments.
2017-05-26 17:15:54 +02:00
Robert Swain
05e88d81cb mediaapi: Add database storage and upload handler 2017-05-26 16:49:54 +02:00
Robert Swain
e21cd5ae66 Merge pull request #120 from matrix-org/rob/media-skeleton
Skeleton for the Media API
2017-05-26 15:57:13 +02:00
Robert Swain
86596488e9 mediaapi/writers/upload: Correct Content-Disposition evaluation
Content-Disposition is not a valid request header and so it must be
evaluated from the upload request's filename query parameter.
2017-05-26 15:26:50 +02:00
Robert Swain
98ef88b668 mediaapi/writers/download: Clarify validation applied to Origin 2017-05-26 14:59:45 +02:00
Robert Swain
4dfbae81cd mediaapi/writers/download: Allow only media IDs matching [A-Za-z0-9_=-]+ 2017-05-26 14:55:51 +02:00
Mark Haines
a96b124158 Move calculate state to the state package (#121) 2017-05-26 13:51:54 +01:00
Robert Swain
0affdae889 mediaapi/types: Rename member to ActiveRemoteRequests.MXCToCond
Clearer that it's a map from mxc URLs to sync.Cond
2017-05-26 14:17:54 +02:00
Robert Swain
e8d3b4648b mediaapi/routing: Use common.MakeAPI 2017-05-26 13:37:07 +02:00
Robert Swain
2fa0ae29d7 mediaapi/writers: Minor improvements to comments 2017-05-26 10:03:16 +02:00
Robert Swain
a3aaa86ea3 mediaapi: Add initial skeleton 2017-05-26 09:57:09 +02:00
Kegsay
db23640f69 cmd: Add create test accounts command (#119) 2017-05-25 17:41:45 +01:00
Mark Haines
84ad4ff9f6 Start implementing /join for room aliases for rooms the server is not in. (#115)
* Start implementing the join room API

* Hacks to get join room working

* Make the TLS fingerprint configurable

* Fix the client API proxy to handle '#' correctly

* Return a 200 OK response

* Write the join event along with current state to the room server

* Comment on the error handling

* Fix typos

* Fix tab

* Add TODO for moving authEventIDs to gomatrixserverlib

* Comment on why we ignore the key ID argument for local keys

* Avoid 'preceeded'

* Neaten the control flow

* Neaten the control flow even more

* Return pointers

* Rename err to lastErr for the loop
2017-05-25 16:08:28 +01:00
Kegsay
445dce14ae Add in devices_table to store device information (#118) 2017-05-25 13:33:50 +01:00
Kegsay
3b9222e8f7 Glue together devices and auth with the current HTTP code (#117)
- Renamed `clientapi/auth/types` to `clientapi/auth/authtypes` for the same
  horrible namespace clashing reasons as `storage`.
- Factored out `makeAPI` to `common`.
- Added in `makeAuthAPI`.
2017-05-23 17:43:05 +01:00
Mark Haines
a7acfa5546 Fix build after #114 2017-05-22 19:28:26 +01:00
Kegsay
d63a1ddc7c Rename packages under /auth (#114)
Previously, all database stuff was under the helpfully named
package 'storage'. However, this convention is used throughout all
of dendrite, which will clash the moment we want to add auth to all
the CS API endpoints. To prevent the package name clash, add
sub-directories which represent what is being stored so the final
usage ends up being:

```
func doThing(db *storage.SyncServerDatabase, authDB *accounts.Database)
{
    // ...
}
```
2017-05-22 16:49:32 +01:00
Kegsay
0325459e7f Implement dummy registration and hook it up to AccountDatabase (#113) 2017-05-22 15:55:39 +01:00
Mark Haines
6605333f6f Start implementing the federation server keys API. (#112)
* Start implementing the federation server keys API.

* Fix copyright

* Fix comments

* Comment on the key format

* Better explain what the ValidityPeriod is

* Return a 200 status code
2017-05-19 16:06:41 +01:00
Kegsay
9d4d18ae7f Add AccountDatabase for storing user accounts (#110)
Including the ability to add new accounts with a user/password and
select accounts using a user/password. Uses bcrypt to hash passwords.
2017-05-19 10:27:03 +01:00
Mark Haines
426a0365cf Rename "make" to "makeAPI" and factor out some more common code into it (#109)
* Rename "make" to "makeAPI" and factor out some more common code into it

Naming a function the same as a go builtin function seems like a bad
idea. Also move the call to `NewJSONRequestHander` inside the function
rather than calling it everywhere.

* Fix typo
2017-05-18 13:47:23 +01:00
Kegan Dougal
cf736d746d hook: Make go vet run all tests and fix warnings 2017-05-18 12:27:11 +01:00
Kegsay
917c433fd2 Minor refactoring (#106)
- `s/Server/OutputRoomEvent/` in `consumers` to accurately reflect what is being consumed.
- `s/set/userIDSet/` in `notifier.go` for clarity.
- Removed lying comments.
2017-05-17 17:29:26 +01:00
Kegsay
c8c5a40ebd Update README.md 2017-05-17 16:37:58 +01:00