Refine config and docs for sytest (#714)
This PR adds the "passed but expected fail" tests to testfile, as well as instructions on finding which tests to add for new PRs.main
Alex Chen
Andrew Morgan
parent
bc382bba46
commit
adaec7c8e7
|
|
@ -1,8 +1,78 @@
|
|||
# SyTest
|
||||
|
||||
Dendrite uses [SyTest](https://github.com/matrix-org/sytest) for its
|
||||
integration testing. When creating a new PR, add the test IDs that your PR
|
||||
should allow to pass to `testfile` in dendrite's root directory. Not all PRs
|
||||
need to make new tests pass. If we find your PR should be making a test pass we
|
||||
may ask you to add to that file, as generally Dendrite's progress can be
|
||||
integration testing. When creating a new PR, add the test IDs (see below) that
|
||||
your PR should allow to pass to `testfile` in dendrite's root directory. Not all
|
||||
PRs need to make new tests pass. If we find your PR should be making a test pass
|
||||
we may ask you to add to that file, as generally Dendrite's progress can be
|
||||
tracked through the amount of SyTest tests it passes.
|
||||
|
||||
## Finding out which tests to add
|
||||
|
||||
We recommend you run the tests locally by manually setting up SyTest or using a
|
||||
SyTest docker image. After running the tests, a script will print the tests you
|
||||
need to add to `testfile` for you.
|
||||
|
||||
You should proceed after you see no build problems for dendrite after running:
|
||||
|
||||
```sh
|
||||
./build.sh
|
||||
```
|
||||
|
||||
### Manually Setting up SyTest
|
||||
|
||||
Make sure you have Perl v5+ installed, and get SyTest with:
|
||||
|
||||
(Note that this guide assumes your SyTest checkout is next to your
|
||||
`dendrite` checkout.)
|
||||
|
||||
```sh
|
||||
git clone -b develop https://github.com/matrix-org/sytest
|
||||
cd sytest
|
||||
./install-deps.pl
|
||||
```
|
||||
|
||||
Set up the database:
|
||||
|
||||
```sh
|
||||
sudo -u postgres psql -c "CREATE USER dendrite PASSWORD 'itsasecret'"
|
||||
sudo -u postgres psql -c "CREATE DATABASE sytest_template OWNER dendrite"
|
||||
mkdir -p "server-0"
|
||||
cat > "server-0/database.yaml" << EOF
|
||||
args:
|
||||
user: dendrite
|
||||
database: dendrite
|
||||
host: 127.0.0.1
|
||||
type: pg
|
||||
EOF
|
||||
```
|
||||
|
||||
Run the tests:
|
||||
|
||||
```sh
|
||||
./run-tests.pl -I Dendrite::Monolith -d ../dendrite/bin -W ../dendrite/testfile -O tap --all | tee results.tap
|
||||
```
|
||||
|
||||
where `tee` lets you see the results while they're being piped to the file.
|
||||
|
||||
Once the tests are complete, run the helper script to see if you need to add
|
||||
any newly passing test names to `testfile` in the project's root directory:
|
||||
|
||||
```sh
|
||||
../dendrite/show-expected-fail-tests.sh results.tap
|
||||
```
|
||||
|
||||
If the script prints nothing/exits with 0, then you're good to go.
|
||||
|
||||
### Using a SyTest Docker image
|
||||
|
||||
Ensure you have the latest image for SyTest, then run the tests:
|
||||
|
||||
```sh
|
||||
docker pull matrixdotorg/sytest-dendrite
|
||||
docker run --rm -v /path/to/dendrite/:/src/ matrixdotorg/sytest-dendrite
|
||||
```
|
||||
|
||||
where `/path/to/dendrite/` should be replaced with the actual path to your
|
||||
dendrite source code. The output should tell you if you need to add any tests to
|
||||
`testfile`.
|
||||
|
|
|
|||
|
|
@ -0,0 +1,29 @@
|
|||
#! /bin/bash
|
||||
|
||||
results_file=$1
|
||||
passed_but_expected_fail=$(grep ' # TODO passed but expected fail' ${results_file} | sed -E 's/^ok [0-9]+ (\(expected fail\) )?//' | sed -E 's/( \([0-9]+ subtests\))? # TODO passed but expected fail$//')
|
||||
tests_to_add=""
|
||||
already_in_testfile=""
|
||||
|
||||
fail_build=0
|
||||
while read -r test_id; do
|
||||
grep "${test_id}" testfile > /dev/null 2>&1
|
||||
if [ "$?" != "0" ]; then
|
||||
tests_to_add="${tests_to_add}${test_id}\n"
|
||||
fail_build=1
|
||||
else
|
||||
already_in_testfile="${already_in_testfile}${test_id}\n"
|
||||
fi
|
||||
done <<< "${passed_but_expected_fail}"
|
||||
|
||||
if [ -n "${tests_to_add}" ]; then
|
||||
echo "ERROR: The following passed tests are not present in testfile. Please append them to the file:"
|
||||
echo -e "${tests_to_add}"
|
||||
fi
|
||||
|
||||
if [ -n "${already_in_testfile}" ]; then
|
||||
echo "WARN: Tests in testfile still marked as expected fail:"
|
||||
echo -e "${already_in_testfile}"
|
||||
fi
|
||||
|
||||
exit ${fail_build}
|
||||
142
testfile
142
testfile
|
|
@ -1,4 +1,144 @@
|
|||
GET /register yields a set of flows
|
||||
POST /register can create a user
|
||||
POST /register downcases capitals in usernames
|
||||
|
||||
POST /register rejects registration of usernames with '!'
|
||||
POST /register rejects registration of usernames with '"'
|
||||
POST /register rejects registration of usernames with ':'
|
||||
POST /register rejects registration of usernames with '?'
|
||||
POST /register rejects registration of usernames with '\'
|
||||
POST /register rejects registration of usernames with '@'
|
||||
POST /register rejects registration of usernames with '['
|
||||
POST /register rejects registration of usernames with ']'
|
||||
POST /register rejects registration of usernames with '{'
|
||||
POST /register rejects registration of usernames with '|'
|
||||
POST /register rejects registration of usernames with '}'
|
||||
POST /register rejects registration of usernames with '£'
|
||||
POST /register rejects registration of usernames with 'é'
|
||||
POST /register rejects registration of usernames with '\n'
|
||||
POST /register rejects registration of usernames with '''
|
||||
GET /login yields a set of flows
|
||||
POST /login can log in as a user
|
||||
POST /login can log in as a user with just the local part of the id
|
||||
POST /login as non-existing user is rejected
|
||||
POST /login wrong password is rejected
|
||||
GET /events initially
|
||||
GET /initialSync initially
|
||||
Version responds 200 OK with valid structure
|
||||
PUT /profile/:user_id/displayname sets my name
|
||||
GET /profile/:user_id/displayname publicly accessible
|
||||
PUT /profile/:user_id/avatar_url sets my avatar
|
||||
GET /profile/:user_id/avatar_url publicly accessible
|
||||
GET /device/{deviceId} gives a 404 for unknown devices
|
||||
PUT /device/{deviceId} gives a 404 for unknown devices
|
||||
POST /createRoom makes a public room
|
||||
POST /createRoom makes a private room
|
||||
POST /createRoom makes a private room with invites
|
||||
POST /createRoom makes a room with a name
|
||||
POST /createRoom makes a room with a topic
|
||||
Can /sync newly created room
|
||||
GET /rooms/:room_id/state/m.room.member/:user_id fetches my membership
|
||||
GET /rooms/:room_id/state/m.room.power_levels fetches powerlevels
|
||||
POST /join/:room_alias can join a room
|
||||
POST /join/:room_id can join a room
|
||||
POST /join/:room_id can join a room with custom content
|
||||
POST /join/:room_alias can join a room with custom content
|
||||
POST /rooms/:room_id/leave can leave a room
|
||||
POST /rooms/:room_id/invite can send an invite
|
||||
POST /rooms/:room_id/ban can ban a user
|
||||
POST /rooms/:room_id/send/:event_type sends a message
|
||||
PUT /rooms/:room_id/send/:event_type/:txn_id sends a message
|
||||
PUT /rooms/:room_id/send/:event_type/:txn_id deduplicates the same txn id
|
||||
GET /rooms/:room_id/state/m.room.power_levels can fetch levels
|
||||
PUT /rooms/:room_id/state/m.room.power_levels can set levels
|
||||
PUT power_levels should not explode if the old power levels were empty
|
||||
Both GET and PUT work
|
||||
POST /rooms/:room_id/read_markers can create read marker
|
||||
User signups are forbidden from starting with '_'
|
||||
Can logout all devices
|
||||
Request to logout with invalid an access token is rejected
|
||||
Request to logout without an access token is rejected
|
||||
Room creation reports m.room.create to myself
|
||||
Room creation reports m.room.member to myself
|
||||
New room members see their own join event
|
||||
Existing members see new members' join events
|
||||
setting 'm.room.power_levels' respects room powerlevel
|
||||
Unprivileged users can set m.room.topic if it only needs level 0
|
||||
Users cannot set ban powerlevel higher than their own
|
||||
Users cannot set kick powerlevel higher than their own
|
||||
Users cannot set redact powerlevel higher than their own
|
||||
Can get rooms/{roomId}/members for a departed room (SPEC-216)
|
||||
3pid invite join with wrong but valid signature are rejected
|
||||
3pid invite join valid signature but revoked keys are rejected
|
||||
3pid invite join valid signature but unreachable ID server are rejected
|
||||
Room members can override their displayname on a room-specific basis
|
||||
Room members can join a room with an overridden displayname
|
||||
displayname updates affect room member events
|
||||
avatar_url updates affect room member events
|
||||
Real non-joined user cannot call /events on shared room
|
||||
Real non-joined user cannot call /events on invited room
|
||||
Real non-joined user cannot call /events on joined room
|
||||
Real non-joined user cannot call /events on default room
|
||||
Real non-joined users can get state for world_readable rooms
|
||||
Real non-joined users can get individual state for world_readable rooms
|
||||
Real non-joined users can get individual state for world_readable rooms after leaving
|
||||
Real non-joined users cannot send messages to guest_access rooms if not joined
|
||||
Real users can sync from world_readable guest_access rooms if joined
|
||||
Real users can sync from default guest_access rooms if joined
|
||||
Can't forget room you're still in
|
||||
Can get rooms/{roomId}/members
|
||||
Can create filter
|
||||
Can download filter
|
||||
Can sync
|
||||
Can sync a joined room
|
||||
Newly joined room is included in an incremental sync
|
||||
User is offline if they set_presence=offline in their sync
|
||||
Changes to state are included in an incremental sync
|
||||
A change to displayname should appear in incremental /sync
|
||||
Current state appears in timeline in private history
|
||||
Current state appears in timeline in private history with many messages before
|
||||
Rooms a user is invited to appear in an initial sync
|
||||
Rooms a user is invited to appear in an incremental sync
|
||||
Sync can be polled for updates
|
||||
Sync is woken up for leaves
|
||||
Newly left rooms appear in the leave section of incremental sync
|
||||
We should see our own leave event, even if history_visibility is restricted (SYN-662)
|
||||
We should see our own leave event when rejecting an invite, even if history_visibility is restricted (riot-web/3462)
|
||||
Newly left rooms appear in the leave section of gapped sync
|
||||
Previously left rooms don't appear in the leave section of sync
|
||||
Left rooms appear in the leave section of full state sync
|
||||
Newly banned rooms appear in the leave section of incremental sync
|
||||
Newly banned rooms appear in the leave section of incremental sync
|
||||
local user can join room with version 1
|
||||
User can invite local user to room with version 1
|
||||
local user can join room with version 2
|
||||
User can invite local user to room with version 2
|
||||
local user can join room with version 3
|
||||
User can invite local user to room with version 3
|
||||
local user can join room with version 4
|
||||
User can invite local user to room with version 4
|
||||
Should reject keys claiming to belong to a different user
|
||||
Can add account data
|
||||
Can add account data to room
|
||||
Latest account data appears in v2 /sync
|
||||
New account data appears in incremental v2 /sync
|
||||
Checking local federation server
|
||||
Inbound federation can query profile data
|
||||
Outbound federation can send room-join requests
|
||||
Outbound federation can send events
|
||||
Inbound federation can backfill events
|
||||
Backfill checks the events requested belong to the room
|
||||
Can upload without a file name
|
||||
Can download without a file name locally
|
||||
Can upload with ASCII file name
|
||||
Can send image in room message
|
||||
AS cannot create users outside its own namespace
|
||||
Regular users cannot register within the AS namespace
|
||||
AS can't set displayname for random users
|
||||
AS user (not ghost) can join room without registering, with user_id query param
|
||||
Changing the actions of an unknown default rule fails with 404
|
||||
Changing the actions of an unknown rule fails with 404
|
||||
Enabling an unknown default rule fails with 404
|
||||
Trying to get push rules with unknown rule_id fails with 404
|
||||
Events come down the correct room
|
||||
local user can join room with version 5
|
||||
User can invite local user to room with version 5
|
||||
|
|
|
|||
Loading…
Reference in New Issue