2017-07-17 16:20:57 +00:00
|
|
|
// Copyright 2017 Vector Creations Ltd
|
Send-to-device support (#1072)
* Groundwork for send-to-device messaging
* Update sample config
* Add unstable routing for now
* Send to device consumer in sync API
* Start the send-to-device consumer
* fix indentation in dendrite-config.yaml
* Create send-to-device database tables, other tweaks
* Add some logic for send-to-device messages, add them into sync stream
* Handle incoming send-to-device messages, count them with EDU stream pos
* Undo changes to test
* pq.Array
* Fix sync
* Logging
* Fix a couple of transaction things, fix client API
* Add send-to-device test, hopefully fix bugs
* Comments
* Refactor a bit
* Fix schema
* Fix queries
* Debug logging
* Fix storing and retrieving of send-to-device messages
* Try to avoid database locks
* Update sync position
* Use latest sync position
* Jiggle about sync a bit
* Fix tests
* Break out the retrieval from the update/delete behaviour
* Comments
* nolint on getResponseWithPDUsForCompleteSync
* Try to line up sync tokens again
* Implement wildcard
* Add all send-to-device tests to whitelist, what could possibly go wrong?
* Only care about wildcard when targeted locally
* Deduplicate transactions
* Handle tokens properly, return immediately if waiting send-to-device messages
* Fix sync
* Update sytest-whitelist
* Fix copyright notice (need to do more of this)
* Comments, copyrights
* Return errors from Do, fix dendritejs
* Review comments
* Comments
* Constructor for TransactionWriter
* defletions
* Update gomatrixserverlib, sytest-blacklist
2020-06-01 16:50:19 +00:00
|
|
|
// Copyright 2017-2018 New Vector Ltd
|
|
|
|
// Copyright 2019-2020 The Matrix.org Foundation C.I.C.
|
2017-07-17 16:20:57 +00:00
|
|
|
//
|
|
|
|
// Licensed under the Apache License, Version 2.0 (the "License");
|
|
|
|
// you may not use this file except in compliance with the License.
|
|
|
|
// You may obtain a copy of the License at
|
|
|
|
//
|
|
|
|
// http://www.apache.org/licenses/LICENSE-2.0
|
|
|
|
//
|
|
|
|
// Unless required by applicable law or agreed to in writing, software
|
|
|
|
// distributed under the License is distributed on an "AS IS" BASIS,
|
|
|
|
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
|
|
// See the License for the specific language governing permissions and
|
|
|
|
// limitations under the License.
|
|
|
|
|
2020-05-21 13:40:13 +00:00
|
|
|
package internal
|
2017-07-17 16:20:57 +00:00
|
|
|
|
|
|
|
import (
|
|
|
|
"database/sql"
|
Send-to-device support (#1072)
* Groundwork for send-to-device messaging
* Update sample config
* Add unstable routing for now
* Send to device consumer in sync API
* Start the send-to-device consumer
* fix indentation in dendrite-config.yaml
* Create send-to-device database tables, other tweaks
* Add some logic for send-to-device messages, add them into sync stream
* Handle incoming send-to-device messages, count them with EDU stream pos
* Undo changes to test
* pq.Array
* Fix sync
* Logging
* Fix a couple of transaction things, fix client API
* Add send-to-device test, hopefully fix bugs
* Comments
* Refactor a bit
* Fix schema
* Fix queries
* Debug logging
* Fix storing and retrieving of send-to-device messages
* Try to avoid database locks
* Update sync position
* Use latest sync position
* Jiggle about sync a bit
* Fix tests
* Break out the retrieval from the update/delete behaviour
* Comments
* nolint on getResponseWithPDUsForCompleteSync
* Try to line up sync tokens again
* Implement wildcard
* Add all send-to-device tests to whitelist, what could possibly go wrong?
* Only care about wildcard when targeted locally
* Deduplicate transactions
* Handle tokens properly, return immediately if waiting send-to-device messages
* Fix sync
* Update sytest-whitelist
* Fix copyright notice (need to do more of this)
* Comments, copyrights
* Return errors from Do, fix dendritejs
* Review comments
* Comments
* Constructor for TransactionWriter
* defletions
* Update gomatrixserverlib, sytest-blacklist
2020-06-01 16:50:19 +00:00
|
|
|
"errors"
|
2020-02-13 17:27:33 +00:00
|
|
|
"fmt"
|
Add peer-to-peer support into Dendrite via libp2p and fetch (#880)
* Use a fork of pq which supports userCurrent on wasm
* Use sqlite3_js driver when running in JS
* Add cmd/dendritejs to pull in sqlite3_js driver for wasm only
* Update to latest go-sqlite-js version
* Replace prometheus with a stub. sigh
* Hard-code a config and don't use opentracing
* Latest go-sqlite3-js version
* Generate a key for now
* Listen for fetch traffic rather than HTTP
* Latest hacks for js
* libp2p support
* More libp2p
* Fork gjson to allow us to enforce auth checks as before
Previously, all events would come down redacted because the hash
checks would fail. They would fail because sjson.DeleteBytes didn't
remove keys not used for hashing. This didn't work because of a build
tag which included a file which no-oped the index returned.
See https://github.com/tidwall/gjson/issues/157
When it's resolved, let's go back to mainline.
* Use gjson@1.6.0 as it fixes https://github.com/tidwall/gjson/issues/157
* Use latest gomatrixserverlib for sig checks
* Fix a bug which could cause exclude_from_sync to not be set
Caused when sending events over federation.
* Use query variadic to make lookups actually work!
* Latest gomatrixserverlib
* Add notes on getting p2p up and running
Partly so I don't forget myself!
* refactor: Move p2p specific stuff to cmd/dendritejs
This is important or else the normal build of dendrite will fail
because the p2p libraries depend on syscall/js which doesn't work
on normal builds.
Also, clean up main.go to read a bit better.
* Update ho-http-js-libp2p to return errors from RoundTrip
* Add an LRU cache around the key DB
We actually need this for P2P because otherwise we can *segfault*
with things like: "runtime: unexpected return pc for runtime.handleEvent"
where the event is a `syscall/js` event, caused by spamming sql.js
caused by "Checking event signatures for 14 events of room state" which
hammers the key DB repeatedly in quick succession.
Using a cache fixes this, though the underlying cause is probably a bug
in the version of Go I'm on (1.13.7)
* breaking: Add Tracing.Enabled to toggle whether we do opentracing
Defaults to false, which is why this is a breaking change. We need
this flag because WASM builds cannot do opentracing.
* Start adding conditional builds for wasm to handle lib/pq
The general idea here is to have the wasm build have a `NewXXXDatabase`
that doesn't import any postgres package and hence we never import
`lib/pq`, which doesn't work under WASM (undefined `userCurrent`).
* Remove lib/pq for wasm for syncapi
* Add conditional building to remaining storage APIs
* Update build script to set env vars correctly for dendritejs
* sqlite bug fixes
* Docs
* Add a no-op main for dendritejs when not building under wasm
* Use the real prometheus, even for WASM
Instead, the dendrite-sw.js must mock out `process.pid` and
`fs.stat` - which must invoke the callback with an error (e.g `EINVAL`)
in order for it to work:
```
global.process = {
pid: 1,
};
global.fs.stat = function(path, cb) {
cb({
code: "EINVAL",
});
}
```
* Linting
2020-03-06 10:23:55 +00:00
|
|
|
"runtime"
|
2020-05-01 12:34:53 +00:00
|
|
|
"time"
|
Send-to-device support (#1072)
* Groundwork for send-to-device messaging
* Update sample config
* Add unstable routing for now
* Send to device consumer in sync API
* Start the send-to-device consumer
* fix indentation in dendrite-config.yaml
* Create send-to-device database tables, other tweaks
* Add some logic for send-to-device messages, add them into sync stream
* Handle incoming send-to-device messages, count them with EDU stream pos
* Undo changes to test
* pq.Array
* Fix sync
* Logging
* Fix a couple of transaction things, fix client API
* Add send-to-device test, hopefully fix bugs
* Comments
* Refactor a bit
* Fix schema
* Fix queries
* Debug logging
* Fix storing and retrieving of send-to-device messages
* Try to avoid database locks
* Update sync position
* Use latest sync position
* Jiggle about sync a bit
* Fix tests
* Break out the retrieval from the update/delete behaviour
* Comments
* nolint on getResponseWithPDUsForCompleteSync
* Try to line up sync tokens again
* Implement wildcard
* Add all send-to-device tests to whitelist, what could possibly go wrong?
* Only care about wildcard when targeted locally
* Deduplicate transactions
* Handle tokens properly, return immediately if waiting send-to-device messages
* Fix sync
* Update sytest-whitelist
* Fix copyright notice (need to do more of this)
* Comments, copyrights
* Return errors from Do, fix dendritejs
* Review comments
* Comments
* Constructor for TransactionWriter
* defletions
* Update gomatrixserverlib, sytest-blacklist
2020-06-01 16:50:19 +00:00
|
|
|
|
|
|
|
"go.uber.org/atomic"
|
2017-07-17 16:20:57 +00:00
|
|
|
)
|
|
|
|
|
2020-06-01 17:34:29 +00:00
|
|
|
// ErrUserExists is returned if a username already exists in the database.
|
|
|
|
var ErrUserExists = errors.New("Username already exists")
|
|
|
|
|
2017-08-21 15:37:11 +00:00
|
|
|
// A Transaction is something that can be committed or rolledback.
|
|
|
|
type Transaction interface {
|
|
|
|
// Commit the transaction
|
|
|
|
Commit() error
|
|
|
|
// Rollback the transaction.
|
|
|
|
Rollback() error
|
|
|
|
}
|
|
|
|
|
|
|
|
// EndTransaction ends a transaction.
|
|
|
|
// If the transaction succeeded then it is committed, otherwise it is rolledback.
|
2020-02-13 17:27:33 +00:00
|
|
|
// You MUST check the error returned from this function to be sure that the transaction
|
|
|
|
// was applied correctly. For example, 'database is locked' errors in sqlite will happen here.
|
|
|
|
func EndTransaction(txn Transaction, succeeded *bool) error {
|
2017-08-21 15:37:11 +00:00
|
|
|
if *succeeded {
|
2020-02-13 17:27:33 +00:00
|
|
|
return txn.Commit() // nolint: errcheck
|
2017-08-21 15:37:11 +00:00
|
|
|
} else {
|
2020-02-13 17:27:33 +00:00
|
|
|
return txn.Rollback() // nolint: errcheck
|
2017-08-21 15:37:11 +00:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2017-07-17 16:20:57 +00:00
|
|
|
// WithTransaction runs a block of code passing in an SQL transaction
|
|
|
|
// If the code returns an error or panics then the transactions is rolledback
|
|
|
|
// Otherwise the transaction is committed.
|
|
|
|
func WithTransaction(db *sql.DB, fn func(txn *sql.Tx) error) (err error) {
|
|
|
|
txn, err := db.Begin()
|
|
|
|
if err != nil {
|
|
|
|
return
|
|
|
|
}
|
2017-08-21 15:37:11 +00:00
|
|
|
succeeded := false
|
2020-02-13 17:27:33 +00:00
|
|
|
defer func() {
|
|
|
|
err2 := EndTransaction(txn, &succeeded)
|
|
|
|
if err == nil && err2 != nil { // failed to commit/rollback
|
|
|
|
err = err2
|
|
|
|
}
|
|
|
|
}()
|
2017-08-21 15:37:11 +00:00
|
|
|
|
2017-07-17 16:20:57 +00:00
|
|
|
err = fn(txn)
|
2017-08-21 15:37:11 +00:00
|
|
|
if err != nil {
|
|
|
|
return
|
|
|
|
}
|
|
|
|
|
|
|
|
succeeded = true
|
2017-07-17 16:20:57 +00:00
|
|
|
return
|
|
|
|
}
|
2017-08-21 16:20:23 +00:00
|
|
|
|
|
|
|
// TxStmt wraps an SQL stmt inside an optional transaction.
|
|
|
|
// If the transaction is nil then it returns the original statement that will
|
|
|
|
// run outside of a transaction.
|
|
|
|
// Otherwise returns a copy of the statement that will run inside the transaction.
|
|
|
|
func TxStmt(transaction *sql.Tx, statement *sql.Stmt) *sql.Stmt {
|
|
|
|
if transaction != nil {
|
|
|
|
statement = transaction.Stmt(statement)
|
|
|
|
}
|
|
|
|
return statement
|
|
|
|
}
|
2017-12-19 09:49:42 +00:00
|
|
|
|
2020-02-13 17:27:33 +00:00
|
|
|
// Hack of the century
|
|
|
|
func QueryVariadic(count int) string {
|
|
|
|
return QueryVariadicOffset(count, 0)
|
|
|
|
}
|
|
|
|
|
|
|
|
func QueryVariadicOffset(count, offset int) string {
|
|
|
|
str := "("
|
|
|
|
for i := 0; i < count; i++ {
|
|
|
|
str += fmt.Sprintf("$%d", i+offset+1)
|
|
|
|
if i < (count - 1) {
|
|
|
|
str += ", "
|
|
|
|
}
|
|
|
|
}
|
|
|
|
str += ")"
|
|
|
|
return str
|
|
|
|
}
|
Add peer-to-peer support into Dendrite via libp2p and fetch (#880)
* Use a fork of pq which supports userCurrent on wasm
* Use sqlite3_js driver when running in JS
* Add cmd/dendritejs to pull in sqlite3_js driver for wasm only
* Update to latest go-sqlite-js version
* Replace prometheus with a stub. sigh
* Hard-code a config and don't use opentracing
* Latest go-sqlite3-js version
* Generate a key for now
* Listen for fetch traffic rather than HTTP
* Latest hacks for js
* libp2p support
* More libp2p
* Fork gjson to allow us to enforce auth checks as before
Previously, all events would come down redacted because the hash
checks would fail. They would fail because sjson.DeleteBytes didn't
remove keys not used for hashing. This didn't work because of a build
tag which included a file which no-oped the index returned.
See https://github.com/tidwall/gjson/issues/157
When it's resolved, let's go back to mainline.
* Use gjson@1.6.0 as it fixes https://github.com/tidwall/gjson/issues/157
* Use latest gomatrixserverlib for sig checks
* Fix a bug which could cause exclude_from_sync to not be set
Caused when sending events over federation.
* Use query variadic to make lookups actually work!
* Latest gomatrixserverlib
* Add notes on getting p2p up and running
Partly so I don't forget myself!
* refactor: Move p2p specific stuff to cmd/dendritejs
This is important or else the normal build of dendrite will fail
because the p2p libraries depend on syscall/js which doesn't work
on normal builds.
Also, clean up main.go to read a bit better.
* Update ho-http-js-libp2p to return errors from RoundTrip
* Add an LRU cache around the key DB
We actually need this for P2P because otherwise we can *segfault*
with things like: "runtime: unexpected return pc for runtime.handleEvent"
where the event is a `syscall/js` event, caused by spamming sql.js
caused by "Checking event signatures for 14 events of room state" which
hammers the key DB repeatedly in quick succession.
Using a cache fixes this, though the underlying cause is probably a bug
in the version of Go I'm on (1.13.7)
* breaking: Add Tracing.Enabled to toggle whether we do opentracing
Defaults to false, which is why this is a breaking change. We need
this flag because WASM builds cannot do opentracing.
* Start adding conditional builds for wasm to handle lib/pq
The general idea here is to have the wasm build have a `NewXXXDatabase`
that doesn't import any postgres package and hence we never import
`lib/pq`, which doesn't work under WASM (undefined `userCurrent`).
* Remove lib/pq for wasm for syncapi
* Add conditional building to remaining storage APIs
* Update build script to set env vars correctly for dendritejs
* sqlite bug fixes
* Docs
* Add a no-op main for dendritejs when not building under wasm
* Use the real prometheus, even for WASM
Instead, the dendrite-sw.js must mock out `process.pid` and
`fs.stat` - which must invoke the callback with an error (e.g `EINVAL`)
in order for it to work:
```
global.process = {
pid: 1,
};
global.fs.stat = function(path, cb) {
cb({
code: "EINVAL",
});
}
```
* Linting
2020-03-06 10:23:55 +00:00
|
|
|
|
|
|
|
func SQLiteDriverName() string {
|
|
|
|
if runtime.GOOS == "js" {
|
|
|
|
return "sqlite3_js"
|
|
|
|
}
|
|
|
|
return "sqlite3"
|
|
|
|
}
|
2020-05-01 12:34:53 +00:00
|
|
|
|
|
|
|
// DbProperties functions return properties used by database/sql/DB
|
|
|
|
type DbProperties interface {
|
|
|
|
MaxIdleConns() int
|
|
|
|
MaxOpenConns() int
|
|
|
|
ConnMaxLifetime() time.Duration
|
|
|
|
}
|
Send-to-device support (#1072)
* Groundwork for send-to-device messaging
* Update sample config
* Add unstable routing for now
* Send to device consumer in sync API
* Start the send-to-device consumer
* fix indentation in dendrite-config.yaml
* Create send-to-device database tables, other tweaks
* Add some logic for send-to-device messages, add them into sync stream
* Handle incoming send-to-device messages, count them with EDU stream pos
* Undo changes to test
* pq.Array
* Fix sync
* Logging
* Fix a couple of transaction things, fix client API
* Add send-to-device test, hopefully fix bugs
* Comments
* Refactor a bit
* Fix schema
* Fix queries
* Debug logging
* Fix storing and retrieving of send-to-device messages
* Try to avoid database locks
* Update sync position
* Use latest sync position
* Jiggle about sync a bit
* Fix tests
* Break out the retrieval from the update/delete behaviour
* Comments
* nolint on getResponseWithPDUsForCompleteSync
* Try to line up sync tokens again
* Implement wildcard
* Add all send-to-device tests to whitelist, what could possibly go wrong?
* Only care about wildcard when targeted locally
* Deduplicate transactions
* Handle tokens properly, return immediately if waiting send-to-device messages
* Fix sync
* Update sytest-whitelist
* Fix copyright notice (need to do more of this)
* Comments, copyrights
* Return errors from Do, fix dendritejs
* Review comments
* Comments
* Constructor for TransactionWriter
* defletions
* Update gomatrixserverlib, sytest-blacklist
2020-06-01 16:50:19 +00:00
|
|
|
|
|
|
|
// TransactionWriter allows queuing database writes so that you don't
|
|
|
|
// contend on database locks in, e.g. SQLite. Only one task will run
|
|
|
|
// at a time on a given TransactionWriter.
|
|
|
|
type TransactionWriter struct {
|
|
|
|
running atomic.Bool
|
|
|
|
todo chan transactionWriterTask
|
|
|
|
}
|
|
|
|
|
|
|
|
func NewTransactionWriter() *TransactionWriter {
|
|
|
|
return &TransactionWriter{
|
|
|
|
todo: make(chan transactionWriterTask),
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
// transactionWriterTask represents a specific task.
|
|
|
|
type transactionWriterTask struct {
|
|
|
|
db *sql.DB
|
|
|
|
f func(txn *sql.Tx) error
|
|
|
|
wait chan error
|
|
|
|
}
|
|
|
|
|
|
|
|
// Do queues a task to be run by a TransactionWriter. The function
|
|
|
|
// provided will be ran within a transaction as supplied by the
|
|
|
|
// database parameter. This will block until the task is finished.
|
|
|
|
func (w *TransactionWriter) Do(db *sql.DB, f func(txn *sql.Tx) error) error {
|
|
|
|
if w.todo == nil {
|
|
|
|
return errors.New("not initialised")
|
|
|
|
}
|
|
|
|
if !w.running.Load() {
|
|
|
|
go w.run()
|
|
|
|
}
|
|
|
|
task := transactionWriterTask{
|
|
|
|
db: db,
|
|
|
|
f: f,
|
|
|
|
wait: make(chan error, 1),
|
|
|
|
}
|
|
|
|
w.todo <- task
|
|
|
|
return <-task.wait
|
|
|
|
}
|
|
|
|
|
|
|
|
// run processes the tasks for a given transaction writer. Only one
|
|
|
|
// of these goroutines will run at a time. A transaction will be
|
|
|
|
// opened using the database object from the task and then this will
|
|
|
|
// be passed as a parameter to the task function.
|
|
|
|
func (w *TransactionWriter) run() {
|
|
|
|
if !w.running.CAS(false, true) {
|
|
|
|
return
|
|
|
|
}
|
|
|
|
defer w.running.Store(false)
|
|
|
|
for task := range w.todo {
|
|
|
|
task.wait <- WithTransaction(task.db, func(txn *sql.Tx) error {
|
|
|
|
return task.f(txn)
|
|
|
|
})
|
|
|
|
close(task.wait)
|
|
|
|
}
|
|
|
|
}
|