The gitea cmd/serv.go setup function deletes the console logger to
prevent stdout interference with commands. However, the problem is that
it then calls setting.NewContext which may cause an exit with log.Fatal
- which will then not be shown.
Now, log.Fatal is so disastrous that the interference that logging
causes is wanted. However, we can avoid stdout interference by emitting
on stderr.
This PR forces the creation of a console logger on stderr listening on
Fatal for these commands. Most commands have a `--debug` option now that
will print more reasons too - however, defaulting to give the log for
Fatal seems reasonable.
Signed-off-by: Andrew Thornton <art27@cantab.net>
* LFS support to be stored on minio
* Fix test
* Fix lint
* Fix lint
* Fix check
* Fix test
* Update documents and add migration for LFS
* Fix some bugs
Provides new command: `gitea doctor recreate-table` which will recreate
db tables and copy the old data in to the new table.
This function can be used to remove the old warning of struct defaults being
out of date.
Fix#8868Fix#3265Fix#8894
Signed-off-by: Andrew Thornton <art27@cantab.net>
#12624 missed lowering the provided repoPath.
(Additionally make a few fixes to the way the debug flag works.)
Fix#12659Fix#12667
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Add a storage layer for attachments
* Fix some bug
* fix test
* Fix copyright head and lint
* Fix bug
* Add setting for minio and flags for migrate-storage
* Add documents
* fix lint
* Add test for minio store type on attachments
* fix test
* fix test
* Apply suggestions from code review
Co-authored-by: guillep2k <18600385+guillep2k@users.noreply.github.com>
* Add warning when storage migrated successfully
* Fix drone
* fix test
* rebase
* Fix test
* display the error on console
* Move minio test to amd64 since minio docker don't support arm64
* refactor the codes
* add trace
* Fix test
* remove log on xorm
* Fi download bug
* Add a storage layer for attachments
* Add setting for minio and flags for migrate-storage
* fix lint
* Add test for minio store type on attachments
* Apply suggestions from code review
Co-authored-by: guillep2k <18600385+guillep2k@users.noreply.github.com>
* Fix drone
* fix test
* Fix test
* display the error on console
* Move minio test to amd64 since minio docker don't support arm64
* refactor the codes
* add trace
* Fix test
* Add URL function to serve attachments directly from S3/Minio
* Add ability to enable/disable redirection in attachment configuration
* Fix typo
* Add a storage layer for attachments
* Add setting for minio and flags for migrate-storage
* fix lint
* Add test for minio store type on attachments
* Apply suggestions from code review
Co-authored-by: guillep2k <18600385+guillep2k@users.noreply.github.com>
* Fix drone
* fix test
* Fix test
* display the error on console
* Move minio test to amd64 since minio docker don't support arm64
* don't change unrelated files
* Fix lint
* Fix build
* update go.mod and go.sum
* Use github.com/minio/minio-go/v6
* Remove unused function
* Upgrade minio to v7 and some other improvements
* fix lint
* Fix go mod
Co-authored-by: guillep2k <18600385+guillep2k@users.noreply.github.com>
Co-authored-by: Tyler <tystuyfzand@gmail.com>
#12391 offered to change the default PID file from /var/run/gitea.pid however in discussion it was decided that this could break users of older systems. An alternative was offered that we could make the PID file compile/link time settable.
This PR does this, and changes the name of the setting from CustomPID to simply PIDFile. It also updates the from-source docs to show how to change the compiler settings to do this.
Closes#12391
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: Florian Klink <flokli@flokli.de>
Replace all calls to os.Remove/os.RemoveAll by retrying util.Remove/util.RemoveAll and remove circular dependencies from util.
Fix#12339
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: silverwind <me@silverwind.io>
* Make LogDescriptions race safe
* Add manager commands for pausing, resuming, adding and removing loggers
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Placate lint
* Ensure that file logger is run!
* Add support for smtp and conn
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Add release-and-reopen
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
Co-authored-by: Lauris BH <lauris@nix.lv>
* Add migration to set IsArchived false if it is null
Fix#11824
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Add doctor
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
* Dump: Use mholt/archive/v3 to support tar including many compressions
Signed-off-by: Philipp Homann <homann.philipp@googlemail.com>
* Dump: Allow dump output to stdout
Signed-off-by: Philipp Homann <homann.philipp@googlemail.com>
* Dump: Fixed bug present since #6677 where SessionConfig.Provider is never "file"
Signed-off-by: Philipp Homann <homann.philipp@googlemail.com>
* Dump: never pack RepoRootPath, LFS.ContentPath and LogRootPath when they are below AppDataPath
Signed-off-by: Philipp Homann <homann.philipp@googlemail.com>
* Dump: also dump LFS (fixes#10058)
Signed-off-by: Philipp Homann <homann.philipp@googlemail.com>
* Dump: never dump CustomPath if CustomPath is a subdir of or equal to AppDataPath (fixes#10365)
Signed-off-by: Philipp Homann <homann.philipp@googlemail.com>
* Use log.Info instead of fmt.Fprintf
Signed-off-by: Philipp Homann <homann.philipp@googlemail.com>
* import ordering
* make fmt
Co-authored-by: zeripath <art27@cantab.net>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
Co-authored-by: Matti R <matti@mdranta.net>
needed to fix issue as described in #10280
* rename check-db to check-db-version
* add check-db-consistency:
* find issues without existing repository
* find pulls without existing issues
* find tracked times without existing issues/pulls
* find labels without repository or org reference
Co-authored-by: guillep2k <18600385+guillep2k@users.noreply.github.com>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
InternalTokens are fixed as alphanum strings therefore TrimSpace from these.
Also use isatty to not add a terminal newline when redirecting generate.
Fix#11498
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: Lauris BH <lauris@nix.lv>
There is a potential memory leak in `Workerpool` due to the intricacies of
`time.Timer` stopping.
Whenever a `time.Timer` is `Stop`ped its channel must be cleared using a
`select` if the result of the `Stop()` is `false`.
Unfortunately in `Workerpool` these were checked the wrong way round.
However, there were a few other places that were not being checked.
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
This is a boolean flag; simply checking if it's set isn't enough, we must check the value as well.
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
Not all dumps need to include the logs, in a similar vain to not all dumps
needing to include repositories; these may be subject to different backup
mechanisms/constraints. Add a simple option to let them be excluded from the
dump to simplify workflows that need to exclude them or not collect in the
first place.
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
Co-authored-by: zeripath <art27@cantab.net>
* Add `gitea doctor --list` flag to list the checks that will be run, including those by default
* Add `gitea doctor --run` to run specific checks
* Add `gitea doctor --all` to run all checks
* Add db version checker
* Add non-default recalculate merge bases check/fixer to doctor
* Add hook checker (Fix#9878) and ensure hooks are executable (Fix#6319)
* Fix authorized_keys checker - slight change of functionality here because parsing the command is fragile and we should just check if the authorized_keys file is essentially the same as what gitea would produce. (This is still not perfect as order matters - we should probably just md5sum the two files.)
* Add SCRIPT_TYPE check (Fix#10977)
* Add `gitea doctor --fix` to attempt to fix what is possible to easily fix
* Add `gitea doctor --log-file` to set the log-file, be it a file, stdout or to switch off completely. (Fixes previously undetected bug with certain xorm logging configurations - see @6543 comment.)
Signed-off-by: Andrew Thornton <art27@cantab.net>
* make admin auth list formatting configurable
Signed-off-by: Andrew Thornton <art27@cantab.net>
* As per @guillep2k
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Add more logging in the LFS server
Adds more logging in the LFS server and stops sending internal server
error information to the client
* Add LFS Lock cursor implementation
* Simplify Claims in LFS and remove the float64 casts
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: Lauris BH <lauris@nix.lv>
* Add doctor check of app.ini paths
* Make /custom dir not mandatory
* Fix message and improve interface
* Update cmd/doctor.go
Co-Authored-By: John Olheiser <42128690+jolheiser@users.noreply.github.com>
* Apaise lint
* Isn't the linter a sweet? (1)
* Isn't the linter a sweet? (2)
* Isn't the linter a sweet?? (3)
* Restart CI
Co-authored-by: John Olheiser <42128690+jolheiser@users.noreply.github.com>
Co-authored-by: zeripath <art27@cantab.net>
* Make WorkerPools and Queues flushable
Adds Flush methods to Queues and the WorkerPool
Further abstracts the WorkerPool
Adds a final step to Flush the queues in the defer from PrintCurrentTest
Fixes an issue with Settings inheritance in queues
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Change to for loop
* Add IsEmpty and begin just making the queues composed WorkerPools
* subsume workerpool into the queues and create a flushable interface
* Add manager command
* Move flushall to queue.Manager and add to testlogger
* As per @guillep2k
* as per @guillep2k
* Just make queues all implement flushable and clean up the wrapped queue flushes
* cope with no timeout
Co-authored-by: Lauris BH <lauris@nix.lv>
* Delay printing hook statuses until after 1 second
* Move to a 5s delay, wrapped writer structure and add config
* Update cmd/hook.go
* Apply suggestions from code review
* Update cmd/hook.go
Co-authored-by: Antoine GIRARD <sapk@users.noreply.github.com>
* add doctor
* Add a new command doctor to check if some wrong configurations on gitea instance
* fix import
* use regex match authorized_keys on doctor
* Add documentation
#8982 attempted to enforce the gitea environment for pushes - unfortunately it tested the settings before they were actually read in - and therefore does not do that!
* make notifyWatchers work on multiple actions
* more efficient multiple notifyWatchers
* Make CommitRepoAction take advantage of multiple actions
* Batch post and pre-receive results
* Set batch to 30
* Auto adjust timeout & add logging
* adjust processing message
* Add some messages to pre-receive
* Make any non-200 status code from pre-receive an error
* Add missing hookPrintResults
* Remove shortcut for single action
* mistaken merge fix
* oops
* Move master branch to the front
* If repo was empty and the master branch is pushed ensure that that is set as the default branch
* fixup
* fixup
* Missed HookOptions in setdefaultbranch
* Batch PushUpdateAddTag and PushUpdateDelTag
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
* Fixed errors logging in dump.go
* Replaced overlooked log.Fatal() with fatal()
* Run make fmt on dump.go
* Added missing new line to the fatal log
* Run make fmt on dump.go
* Change graceful to use a singleton obtained through GetManager instead of a global.
* Graceful: Make TestPullRequests shutdownable
* Graceful: Make the cron tasks graceful
* Graceful: AddTestPullRequest run in graceful ctx
* Graceful: SyncMirrors shutdown
* Graceful: SetDefaultContext for Xorm to be HammerContext
* Avoid starting graceful for migrate commands and checkout
* Graceful: DeliverHooks now can be shutdown
* Fix multiple syncing errors in modules/sync/UniqueQueue & Make UniqueQueue closable
* Begin the process of making the repo indexer shutdown gracefully
* Graceful: Create callbacks to with contexts
* Graceful: Say when Gitea is completely finished
* Graceful: Git and Process within HammerTime
Force all git commands to terminate at HammerTime
Force all process commands to terminate at HammerTime
Move almost all git processes to run as git Commands
* Graceful: Always Hammer after Shutdown
* ProcessManager: Add cancel functionality
* Fix tests
* Make sure that process.Manager.Kill() cancels
* Make threadsafe access to Processes and remove own unused Kill
* Remove cmd from the process manager as it is no longer used
* the default context is the correct context
* get rid of double till
* Graceful: Allow graceful restart for fcgi
My previous interpretation was incorrect - we do not handle sockets
being passed in over stdin
* Update web.go
* Graceful Shutdown for windows and others
Restructures modules/graceful, adding shutdown for windows, removing and
replacing the old minwinsvc code.
Creates a new waitGroup - terminate which allows for goroutines to
finish up after the shutdown of the servers.
Shutdown and terminate hooks are added for goroutines.
* Remove unused functions - these can be added in a different PR
* Add startup timeout functionality
* Document STARTUP_TIMEOUT
In investigating #7947 it has become clear that the storage component of go-git repositories needs closing.
This PR adds this Close function and adds the Close functions as necessary.
In TransferOwnership the ctx.Repo.GitRepo is closed if it is open to help prevent the risk of multiple open files.
Fixes#7947
* Only attempt to kill parent once
* Apply suggestions from code review
Co-Authored-By: guillep2k <18600385+guillep2k@users.noreply.github.com>
* Add waitgroup for running servers
Add an option to protected branches to add writing deploy keys to the whitelist for pushing.
Please note this is technically a breaking change: previously if the owner of a repository was on the whitelist then any writing deploy key was effectively on the whitelist. This option will now need to be set if that is desired.
Closes#8472
Details:
* Allow Protected Branches to Whitelist Deploy Keys
* Add migration
* Ensure that IsDeployKey is set to false on the http pushes
* add not null default false
* Prevent deadlock in indexer initialisation during graceful restart
* Move from gracehttp to our own service to add graceful ssh
* Add timeout for start of indexers and make hammer time configurable
* Fix issue with re-initialization in indexer during tests
* move the code to detect use of closed to graceful
* Handle logs gracefully - add a pid suffix just before restart
* Move to using a cond and a holder for indexers
* use time.Since
* Add some comments and attribution
* update modules.txt
* Use zero to disable timeout
* Move RestartProcess to its own file
* Add cleanup routine
Add password complexity checks. The default settings require a lowercase, uppercase, number and a special character within passwords.
Co-Authored-By: T-M-A <maxim.tkachenko@gmail.com>
Co-Authored-By: Lanre Adelowo <adelowomailbox@gmail.com>
Co-Authored-By: guillep2k <18600385+guillep2k@users.noreply.github.com>
Co-Authored-By: Lauris BH <lauris@nix.lv>
Regression in 356854fc5f, where
a log.Fatal call was removed. log.Fatal calls os.Exit(1) as intended,
but without it the fail() function returns normally and execution
continues past the fatal error, causing a panic.
This is visible as a go panic log and stack trace returned to the SSH
client, which is not only ugly, it leaks server and build system
information.
Fix by removing the stray return statement so that the fail() function
always calls os.Exit(1).
Fixes: https://github.com/go-gitea/gitea/issues/7457
Signed-off-by: Allen Wild <allenwild93@gmail.com>
* Fix#6946 by checking PullRequest ID on pushing
* Ensure we have the owner name, the pr attributes and the the issue
* Fix TestSearchRepo by waiting till indexing is done
* Update integrations/repo_search_test.go
* changes as per @mrsdizzie
* missing comma
* Spelling mistake
* Fix full pushing environment
* add CLI commands to manage LDAP authentication source
* delete Gogs copyright
* remove unused return value of func parseLoginSource
* fix comment
Co-Authored-By: ngourdon <31291059+ngourdon@users.noreply.github.com>
* remove config flag already present in global flags
* remove config flag from ldap commands in docs
* remove config flag handling
* Move hook functionality internally
* Internalise serv logic
* Remove old internal paths
* finally remove the gitlogger
* Disallow push on archived repositories
* fix lint error
* Update modules/private/key.go
* Update routers/private/hook.go
* Update routers/private/hook.go
* Update routers/private/hook.go
* Updated routers/private/serv.go
* Fix LFS Locks over SSH
* rev-list needs to be run by the hook process
* fixup
* Improve git test
* Ensure that the lfs files are created with a different prefix
* Reduce the replication in git_test.go
* slight refactor
* Remove unnecessary "/"
* Restore ensureAnonymousClone
* Restore ensureAnonymousClone
* Run rev-list on server side
* Try passing in the alternative directories instead
* Mark test as skipped
* Improve git test
* Ensure that the lfs files are created with a different prefix
* Reduce the replication in git_test.go
* Remove unnecessary "/"
When replicating to gitea from a remote system which makes use of
git refs to store extra data (for example, gerrit), pushing a lot
of refs to gitea can cause problems due to the extra processing
that the pre and post receive hooks perform. But it's still
useful for gitea to be able to serve those refs. This change
skips unecessary processing of refs other than branches or tags.
We don't need to check any ref that isn't a branch for branch
protection (protection will never be enabled). So in the
pre-receive hook, we wrap that check in a test for whether the
ref is a branch.
We also don't need to add information to the activity stream about
pushes to non-standard refs, so we skip that step in the
post-receive hook for refs which are not branches or tags.
For some concrete examples, gerrit maintains a ref for every
patchset of every change in the form refs/changes/XX/YYYY/Z.
Many systems use refs/notes to store additonal data about commits.
This change allows these and other schemes to be used without
affecting gitea.
* Add username flag in create-user command
* Fix the error message
* Change name to username on flag of create-user command
* Include create-user flags of both versions
* delete deprecated example of create-user command
When running the dump command, include the specific configuration file
provided as part of the backup. Currently we only zip up custom/ and this
misses the config file if it is somewhere else, like /etc/gitea/app.ini
Fixes#6034
* Panic don't fatal on create new logger
Fixes#5854
Signed-off-by: Andrew Thornton <art27@cantab.net>
* partial broken
* Update the logging infrastrcture
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Reset the skip levels for Fatal and Error
Signed-off-by: Andrew Thornton <art27@cantab.net>
* broken ncsa
* More log.Error fixes
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Remove nal
* set log-levels to lowercase
* Make console_test test all levels
* switch to lowercased levels
* OK now working
* Fix vetting issues
* Fix lint
* Fix tests
* change default logging to match current gitea
* Improve log testing
Signed-off-by: Andrew Thornton <art27@cantab.net>
* reset error skip levels to 0
* Update documentation and access logger configuration
* Redirect the router log back to gitea if redirect macaron log but also allow setting the log level - i.e. TRACE
* Fix broken level caching
* Refactor the router log
* Add Router logger
* Add colorizing options
* Adjust router colors
* Only create logger if they will be used
* update app.ini.sample
* rename Attribute ColorAttribute
* Change from white to green for function
* Set fatal/error levels
* Restore initial trace logger
* Fix Trace arguments in modules/auth/auth.go
* Properly handle XORMLogger
* Improve admin/config page
* fix fmt
* Add auto-compression of old logs
* Update error log levels
* Remove the unnecessary skip argument from Error, Fatal and Critical
* Add stacktrace support
* Fix tests
* Remove x/sync from vendors?
* Add stderr option to console logger
* Use filepath.ToSlash to protect against Windows in tests
* Remove prefixed underscores from names in colors.go
* Remove not implemented database logger
This was removed from Gogs on 4 Mar 2016 but left in the configuration
since then.
* Ensure that log paths are relative to ROOT_PATH
* use path.Join
* rename jsonConfig to logConfig
* Rename "config" to "jsonConfig" to make it clearer
* Requested changes
* Requested changes: XormLogger
* Try to color the windows terminal
If successful default to colorizing the console logs
* fixup
* Colorize initially too
* update vendor
* Colorize logs on default and remove if this is not a colorizing logger
* Fix documentation
* fix test
* Use go-isatty to detect if on windows we are on msys or cygwin
* Fix spelling mistake
* Add missing vendors
* More changes
* Rationalise the ANSI writer protection
* Adjust colors on advice from @0x5c
* Make Flags a comma separated list
* Move to use the windows constant for ENABLE_VIRTUAL_TERMINAL_PROCESSING
* Ensure matching is done on the non-colored message - to simpify EXPRESSION
* Use PathUnescape instead of QueryUnescape when working with branch names
Currently branch names with a '+' fail in certain situations because
QueryUnescape replaces the + character with a blank space.
Using PathUnescape should be better since it is defined as:
// PathUnescape is identical to QueryUnescape except that it does not
// unescape '+' to ' ' (space).
Fixes#6333
* Change error to match new function name
* Add new util function PathEscapeSegments
This function simply runs PathEscape on each segment of a path without
touching the forward slash itself. We want to use this instead of
PathEscape/QueryEscape in most cases because a forward slash is a valid name for a
branch etc... and we don't want that escaped in a URL.
Putting this in new file url.go and also moving a couple similar
functions into that file as well.
* Use EscapePathSegments where appropriate
Replace various uses of EscapePath/EscapeQuery with new
EscapePathSegments. Also remove uncessary uses of various
escape/unescape functions when the text had already been escaped or was
not escaped.
* Reformat comment to make drone build happy
* Remove no longer used url library
* Requested code changes
1. A key can either be an ssh user key or a deploy key. It cannot be both.
2. If a key is a user key - it can only be associated with one user.
3. If a key is a deploy key - it can be used in multiple repositories and the permissions it has on those repositories can be different.
4. If a repository is deleted, its deploy keys must be deleted too.
We currently don't enforce any of this and multiple repositories access with different permissions doesn't work at all. This PR enforces the following constraints:
- [x] You should not be able to add the same user key as another user
- [x] You should not be able to add a ssh user key which is being used as a deploy key
- [x] You should not be able to add a ssh deploy key which is being used as a user key
- [x] If you add an ssh deploy key to another repository you should be able to use it in different modes without losing the ability to use it in the other mode.
- [x] If you delete a repository you must delete all its deploy keys.
Fix#1357
The gitea prerecieve and postrecieve hooks and the gitea PushUpdate function require that the PusherID and PusherName are real users. Previously, these environment variables were not being set when using a deploy key - the main result being that pushing to empty repositories meant that is_empty status was not changed.
I've also added an integration test to ensure that the is_empty status is updated on pushing with a deploy key.
There is a slight issue in that the deploy key is now considered a proxy for the owner - we don't have a way of separating out the deploy key from the owner at present. This can be fixed in another PR.
Fix#3795
Signed-off-by: Andrew Thornton art27@cantab.net
By default, if `setting.NewContext()` prints out any warning logs, these are printed to the stdout breaking `git receive-pack` etc. meaning that even if there is a warning because of a minor problem in your app.ini but gitea starts despite this - you **CANNOT** push or pull over SSH.
This PR disables the console logger whilst in `serv.go`
Signed-off-by: Andrew Thornton <art27@cantab.net>
* add util method and tests
* make sure the title of an issue cannot be empty
* wiki title cannot be empty
* pull request title cannot be empty
* update to make use of the new util methof
* add migration and basic UI for changing a user's theme
* update user themem
* use right text on button
* load theme based on users' selection
* load theme based on users' selection in pwa too
* update sample config
* delete older theme loading
* implement AfterLoad to set users' theme properly
* set up default theme when creating a user. This uses the installation wide theme
* use flash messages for error
* set default theme when creating a user from the cli
* fix @lunny review
zeripath
6543
Lunny Xiao
John Olheiser
Wim
赵智超
PhilippHomann
Kyle Evans
Lauris BH
John Olheiser
guillep2k
SpaWn2KiLl
Shashvat Kedia
Rychu
Maxim Tkachenko
Tamal Saha
clavinet
Allen Wild
ngourdon
kolaente
Sandro Santilli
James E. Blair
Lanre Adelowo
techknowlogick
ngourdon
mrsdizzie
glaszig
Jonas Franz
Lanre Adelowo
Zsombor
Greg Karékinian