Refine config and docs for sytest (#714)
This PR adds the "passed but expected fail" tests to testfile, as well as instructions on finding which tests to add for new PRs.
This commit is contained in:
Alex Chen
Andrew Morgan
parent
bc382bba46
commit
adaec7c8e7
3 changed files with 244 additions and 5 deletions
|
|
@ -1,8 +1,78 @@
|
|||
# SyTest
|
||||
|
||||
Dendrite uses [SyTest](https://github.com/matrix-org/sytest) for its
|
||||
integration testing. When creating a new PR, add the test IDs that your PR
|
||||
should allow to pass to `testfile` in dendrite's root directory. Not all PRs
|
||||
need to make new tests pass. If we find your PR should be making a test pass we
|
||||
may ask you to add to that file, as generally Dendrite's progress can be
|
||||
integration testing. When creating a new PR, add the test IDs (see below) that
|
||||
your PR should allow to pass to `testfile` in dendrite's root directory. Not all
|
||||
PRs need to make new tests pass. If we find your PR should be making a test pass
|
||||
we may ask you to add to that file, as generally Dendrite's progress can be
|
||||
tracked through the amount of SyTest tests it passes.
|
||||
|
||||
## Finding out which tests to add
|
||||
|
||||
We recommend you run the tests locally by manually setting up SyTest or using a
|
||||
SyTest docker image. After running the tests, a script will print the tests you
|
||||
need to add to `testfile` for you.
|
||||
|
||||
You should proceed after you see no build problems for dendrite after running:
|
||||
|
||||
```sh
|
||||
./build.sh
|
||||
```
|
||||
|
||||
### Manually Setting up SyTest
|
||||
|
||||
Make sure you have Perl v5+ installed, and get SyTest with:
|
||||
|
||||
(Note that this guide assumes your SyTest checkout is next to your
|
||||
`dendrite` checkout.)
|
||||
|
||||
```sh
|
||||
git clone -b develop https://github.com/matrix-org/sytest
|
||||
cd sytest
|
||||
./install-deps.pl
|
||||
```
|
||||
|
||||
Set up the database:
|
||||
|
||||
```sh
|
||||
sudo -u postgres psql -c "CREATE USER dendrite PASSWORD 'itsasecret'"
|
||||
sudo -u postgres psql -c "CREATE DATABASE sytest_template OWNER dendrite"
|
||||
mkdir -p "server-0"
|
||||
cat > "server-0/database.yaml" << EOF
|
||||
args:
|
||||
user: dendrite
|
||||
database: dendrite
|
||||
host: 127.0.0.1
|
||||
type: pg
|
||||
EOF
|
||||
```
|
||||
|
||||
Run the tests:
|
||||
|
||||
```sh
|
||||
./run-tests.pl -I Dendrite::Monolith -d ../dendrite/bin -W ../dendrite/testfile -O tap --all | tee results.tap
|
||||
```
|
||||
|
||||
where `tee` lets you see the results while they're being piped to the file.
|
||||
|
||||
Once the tests are complete, run the helper script to see if you need to add
|
||||
any newly passing test names to `testfile` in the project's root directory:
|
||||
|
||||
```sh
|
||||
../dendrite/show-expected-fail-tests.sh results.tap
|
||||
```
|
||||
|
||||
If the script prints nothing/exits with 0, then you're good to go.
|
||||
|
||||
### Using a SyTest Docker image
|
||||
|
||||
Ensure you have the latest image for SyTest, then run the tests:
|
||||
|
||||
```sh
|
||||
docker pull matrixdotorg/sytest-dendrite
|
||||
docker run --rm -v /path/to/dendrite/:/src/ matrixdotorg/sytest-dendrite
|
||||
```
|
||||
|
||||
where `/path/to/dendrite/` should be replaced with the actual path to your
|
||||
dendrite source code. The output should tell you if you need to add any tests to
|
||||
`testfile`.
|
||||
|
|
|
|||
29
show-expected-fail-tests.sh
Executable file
29
show-expected-fail-tests.sh
Executable file
|
|
@ -0,0 +1,29 @@
|
|||
#! /bin/bash
|
||||
|
||||
results_file=$1
|
||||
passed_but_expected_fail=$(grep ' # TODO passed but expected fail' ${results_file} | sed -E 's/^ok [0-9]+ (\(expected fail\) )?//' | sed -E 's/( \([0-9]+ subtests\))? # TODO passed but expected fail$//')
|
||||
tests_to_add=""
|
||||
already_in_testfile=""
|
||||
|
||||
fail_build=0
|
||||
while read -r test_id; do
|
||||
grep "${test_id}" testfile > /dev/null 2>&1
|
||||
if [ "$?" != "0" ]; then
|
||||
tests_to_add="${tests_to_add}${test_id}\n"
|
||||
fail_build=1
|
||||
else
|
||||
already_in_testfile="${already_in_testfile}${test_id}\n"
|
||||
fi
|
||||
done <<< "${passed_but_expected_fail}"
|
||||
|
||||
if [ -n "${tests_to_add}" ]; then
|
||||
echo "ERROR: The following passed tests are not present in testfile. Please append them to the file:"
|
||||
echo -e "${tests_to_add}"
|
||||
fi
|
||||
|
||||
if [ -n "${already_in_testfile}" ]; then
|
||||
echo "WARN: Tests in testfile still marked as expected fail:"
|
||||
echo -e "${already_in_testfile}"
|
||||
fi
|
||||
|
||||
exit ${fail_build}
|
||||
142
testfile
142
testfile
|
|
@ -1,4 +1,144 @@
|
|||
GET /register yields a set of flows
|
||||
POST /register can create a user
|
||||
POST /register downcases capitals in usernames
|
||||
|
||||
POST /register rejects registration of usernames with '!'
|
||||
POST /register rejects registration of usernames with '"'
|
||||
POST /register rejects registration of usernames with ':'
|
||||
POST /register rejects registration of usernames with '?'
|
||||
POST /register rejects registration of usernames with '\'
|
||||
POST /register rejects registration of usernames with '@'
|
||||
POST /register rejects registration of usernames with '['
|
||||
POST /register rejects registration of usernames with ']'
|
||||
POST /register rejects registration of usernames with '{'
|
||||
POST /register rejects registration of usernames with '|'
|
||||
POST /register rejects registration of usernames with '}'
|
||||
POST /register rejects registration of usernames with '£'
|
||||
POST /register rejects registration of usernames with 'é'
|
||||
POST /register rejects registration of usernames with '\n'
|
||||
POST /register rejects registration of usernames with '''
|
||||
GET /login yields a set of flows
|
||||
POST /login can log in as a user
|
||||
POST /login can log in as a user with just the local part of the id
|
||||
POST /login as non-existing user is rejected
|
||||
POST /login wrong password is rejected
|
||||
GET /events initially
|
||||
GET /initialSync initially
|
||||
Version responds 200 OK with valid structure
|
||||
PUT /profile/:user_id/displayname sets my name
|
||||
GET /profile/:user_id/displayname publicly accessible
|
||||
PUT /profile/:user_id/avatar_url sets my avatar
|
||||
GET /profile/:user_id/avatar_url publicly accessible
|
||||
GET /device/{deviceId} gives a 404 for unknown devices
|
||||
PUT /device/{deviceId} gives a 404 for unknown devices
|
||||
POST /createRoom makes a public room
|
||||
POST /createRoom makes a private room
|
||||
POST /createRoom makes a private room with invites
|
||||
POST /createRoom makes a room with a name
|
||||
POST /createRoom makes a room with a topic
|
||||
Can /sync newly created room
|
||||
GET /rooms/:room_id/state/m.room.member/:user_id fetches my membership
|
||||
GET /rooms/:room_id/state/m.room.power_levels fetches powerlevels
|
||||
POST /join/:room_alias can join a room
|
||||
POST /join/:room_id can join a room
|
||||
POST /join/:room_id can join a room with custom content
|
||||
POST /join/:room_alias can join a room with custom content
|
||||
POST /rooms/:room_id/leave can leave a room
|
||||
POST /rooms/:room_id/invite can send an invite
|
||||
POST /rooms/:room_id/ban can ban a user
|
||||
POST /rooms/:room_id/send/:event_type sends a message
|
||||
PUT /rooms/:room_id/send/:event_type/:txn_id sends a message
|
||||
PUT /rooms/:room_id/send/:event_type/:txn_id deduplicates the same txn id
|
||||
GET /rooms/:room_id/state/m.room.power_levels can fetch levels
|
||||
PUT /rooms/:room_id/state/m.room.power_levels can set levels
|
||||
PUT power_levels should not explode if the old power levels were empty
|
||||
Both GET and PUT work
|
||||
POST /rooms/:room_id/read_markers can create read marker
|
||||
User signups are forbidden from starting with '_'
|
||||
Can logout all devices
|
||||
Request to logout with invalid an access token is rejected
|
||||
Request to logout without an access token is rejected
|
||||
Room creation reports m.room.create to myself
|
||||
Room creation reports m.room.member to myself
|
||||
New room members see their own join event
|
||||
Existing members see new members' join events
|
||||
setting 'm.room.power_levels' respects room powerlevel
|
||||
Unprivileged users can set m.room.topic if it only needs level 0
|
||||
Users cannot set ban powerlevel higher than their own
|
||||
Users cannot set kick powerlevel higher than their own
|
||||
Users cannot set redact powerlevel higher than their own
|
||||
Can get rooms/{roomId}/members for a departed room (SPEC-216)
|
||||
3pid invite join with wrong but valid signature are rejected
|
||||
3pid invite join valid signature but revoked keys are rejected
|
||||
3pid invite join valid signature but unreachable ID server are rejected
|
||||
Room members can override their displayname on a room-specific basis
|
||||
Room members can join a room with an overridden displayname
|
||||
displayname updates affect room member events
|
||||
avatar_url updates affect room member events
|
||||
Real non-joined user cannot call /events on shared room
|
||||
Real non-joined user cannot call /events on invited room
|
||||
Real non-joined user cannot call /events on joined room
|
||||
Real non-joined user cannot call /events on default room
|
||||
Real non-joined users can get state for world_readable rooms
|
||||
Real non-joined users can get individual state for world_readable rooms
|
||||
Real non-joined users can get individual state for world_readable rooms after leaving
|
||||
Real non-joined users cannot send messages to guest_access rooms if not joined
|
||||
Real users can sync from world_readable guest_access rooms if joined
|
||||
Real users can sync from default guest_access rooms if joined
|
||||
Can't forget room you're still in
|
||||
Can get rooms/{roomId}/members
|
||||
Can create filter
|
||||
Can download filter
|
||||
Can sync
|
||||
Can sync a joined room
|
||||
Newly joined room is included in an incremental sync
|
||||
User is offline if they set_presence=offline in their sync
|
||||
Changes to state are included in an incremental sync
|
||||
A change to displayname should appear in incremental /sync
|
||||
Current state appears in timeline in private history
|
||||
Current state appears in timeline in private history with many messages before
|
||||
Rooms a user is invited to appear in an initial sync
|
||||
Rooms a user is invited to appear in an incremental sync
|
||||
Sync can be polled for updates
|
||||
Sync is woken up for leaves
|
||||
Newly left rooms appear in the leave section of incremental sync
|
||||
We should see our own leave event, even if history_visibility is restricted (SYN-662)
|
||||
We should see our own leave event when rejecting an invite, even if history_visibility is restricted (riot-web/3462)
|
||||
Newly left rooms appear in the leave section of gapped sync
|
||||
Previously left rooms don't appear in the leave section of sync
|
||||
Left rooms appear in the leave section of full state sync
|
||||
Newly banned rooms appear in the leave section of incremental sync
|
||||
Newly banned rooms appear in the leave section of incremental sync
|
||||
local user can join room with version 1
|
||||
User can invite local user to room with version 1
|
||||
local user can join room with version 2
|
||||
User can invite local user to room with version 2
|
||||
local user can join room with version 3
|
||||
User can invite local user to room with version 3
|
||||
local user can join room with version 4
|
||||
User can invite local user to room with version 4
|
||||
Should reject keys claiming to belong to a different user
|
||||
Can add account data
|
||||
Can add account data to room
|
||||
Latest account data appears in v2 /sync
|
||||
New account data appears in incremental v2 /sync
|
||||
Checking local federation server
|
||||
Inbound federation can query profile data
|
||||
Outbound federation can send room-join requests
|
||||
Outbound federation can send events
|
||||
Inbound federation can backfill events
|
||||
Backfill checks the events requested belong to the room
|
||||
Can upload without a file name
|
||||
Can download without a file name locally
|
||||
Can upload with ASCII file name
|
||||
Can send image in room message
|
||||
AS cannot create users outside its own namespace
|
||||
Regular users cannot register within the AS namespace
|
||||
AS can't set displayname for random users
|
||||
AS user (not ghost) can join room without registering, with user_id query param
|
||||
Changing the actions of an unknown default rule fails with 404
|
||||
Changing the actions of an unknown rule fails with 404
|
||||
Enabling an unknown default rule fails with 404
|
||||
Trying to get push rules with unknown rule_id fails with 404
|
||||
Events come down the correct room
|
||||
local user can join room with version 5
|
||||
User can invite local user to room with version 5
|
||||
|
|
|
|||
Loading…
Reference in a new issue