* More efficiently parse shas for shaPostProcessor
The shaPostProcessor currently repeatedly calls git rev-parse --verify on both backends
which is fine if there is only one thing that matches a sha - however if there are
multiple things then this becomes wildly inefficient.
This PR provides functions for both backends which are much faster to use.
Fix#16092
* Add ShaExistCache to RenderContext
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: 6543 <6543@obermui.de>
There is an inefficiency in the design of our processors which means that Emoji
and other processors run in order n^2 time.
This PR forces the processors to process the entirety of text node before passing
back up. The fundamental inefficiency remains but it should be significantly
ameliorated.
Signed-off-by: Andrew Thornton <art27@cantab.net>
Bluemonday sanitizer regexp rules are not additive, so the addition of the icons,
emojis and chroma syntax policy has led to this being stripped.
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Links in markdown should be absolute to the repository not the server
Fix#15075
Signed-off-by: Andrew Thornton <art27@cantab.net>
* match github
Signed-off-by: Andrew Thornton <art27@cantab.net>
* add testcase
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: 6543 <6543@obermui.de>
* Upgrade to bluemonday 1.0.7
Fix#15349
Signed-off-by: Andrew Thornton <art27@cantab.net>
* resolve unit test
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
* _ to unused func options
* rm useless brakets
* rm trifial non used models functions
* rm dead code
* rm dead global vars
* fix routers/api/v1/repo/issue.go
* dont overload import module
Implements request #14320 The rendering of CSV files does match the diff style.
* Moved CSV logic into base package.
* Added method to create a tabular diff.
* Added CSV compare context.
* Added CSV diff template.
* Use new table style in CSV markup.
* Added file size limit for CSV rendering.
* Display CSV parser errors in diff.
* Lazy read single file.
* Lazy read rows for full diff.
* Added unit tests for various CSV changes.
* Fix an issue with panics related to attributes
* Wrap goldmark render in a recovery function
* Reduce memory use in render emoji
* Use a pipe for rendering goldmark - still needs more work and a limiter
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: Lauris BH <lauris@nix.lv>
* Make sure sibling images get a link too
Due a problem with the ast.Walker in the our transformer in goldmark
an image with a sibling image will not be transformed to gain a parent
link. This PR fixes this.
Fix#12925
Signed-off-by: Andrew Thornton <art27@cantab.net>
PostProcess is supposed to be parsing and handling HTML
fragments, but on fuzzing it appears that there is a weird
issue with NUL elements that could cause a memory address
error in downstream libraries.
The simplest solution is to strip out the weird NULs - they
should not be there in any case and would be stripped out
anyway.
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Prevent panic on fuzzer provided string
The fuzzer has found that providing a <body> tag with an attribute to
PostProcess causes a panic. This PR removes any rendered html or body
tags from the output.
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Placate lint
* placate lint again
Signed-off-by: Andrew Thornton <art27@cantab.net>
* minor cleanup
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Markdown task list improvements
- Remove `.ui` class and wrappers to prevent fomantic from messing with it.
- Change rendered HTML to match GitHub.
- Add custom styling for the checkboxes.
* fix unittest
Co-authored-by: Lauris BH <lauris@nix.lv>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
* clickable links in pull request (and issue) titles #13658
reuses the existing logic to render clickable links in commit messages
* dedicated RenderIssueTitle function
applied patch from @mrsdizzie
This PR updates golangci-lint to the latest version 1.31.0.
The upgrade introduced a new check for which I've fixed or disabled most cases.
Signed-off-by: kolaente <k@knt.li>
* fix: media links in org files not liked to media files
* fix: write directly to io.Writer r
as suggested by code review
Co-authored-by: zeripath <art27@cantab.net>
Co-authored-by: zeripath <art27@cantab.net>
* Fix meta parsing and add minimal test
Signed-off-by: jolheiser <john.olheiser@gmail.com>
* Add license
Signed-off-by: jolheiser <john.olheiser@gmail.com>
* Require first separator and add more tests
Signed-off-by: jolheiser <john.olheiser@gmail.com>
* Make fmt
Signed-off-by: jolheiser <john.olheiser@gmail.com>
* Go lets this work gracefully 🙏
Signed-off-by: jolheiser <john.olheiser@gmail.com>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
* Update to go-org 1.3.2
Fix#12727
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Fix unit test
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
- Fix emoji not being replaced in issue title change text
- Make the image attributes consistent, add alt, remove align
Co-authored-by: zeripath <art27@cantab.net>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
Replace all calls to os.Remove/os.RemoveAll by retrying util.Remove/util.RemoveAll and remove circular dependencies from util.
Fix#12339
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: silverwind <me@silverwind.io>
- Add loading spinners on editor and mermaid renderers
- Add error handling and inline error box for mermaid
- Fix Mermaid rendering by using the .init api
* Fix emoji detection certain cases
Previous tests weren't complicated enough so there were some situations where emojis were't detected properly. Find the earliest occurance in addition to checking for the longest combination.
Fixes#12312
* ok spell bot
Co-authored-by: Lauris BH <lauris@nix.lv>
* Server-side syntax hilighting for all code
This PR does a few things:
* Remove all traces of highlight.js
* Use chroma library to provide fast syntax hilighting directly on the server
* Provide syntax hilighting for diffs
* Re-style both unified and split diffs views
* Add custom syntax hilighting styling for both regular and arc-green
Fixes#7729Fixes#10157Fixes#11825Fixes#7728Fixes#3872Fixes#3682
And perhaps gets closer to #9553
* fix line marker
* fix repo search
* Fix single line select
* properly load settings
* npm uninstall highlight.js
* review suggestion
* code review
* forgot to call function
* fix test
* Apply suggestions from code review
suggestions from @silverwind thanks
Co-authored-by: silverwind <me@silverwind.io>
* code review
* copy/paste error
* Use const for highlight size limit
* Update web_src/less/_repository.less
Co-authored-by: Lauris BH <lauris@nix.lv>
* update size limit to 1MB and other styling tweaks
* fix highlighting for certain diff sections
* fix test
* add worker back as suggested
Co-authored-by: silverwind <me@silverwind.io>
Co-authored-by: Lauris BH <lauris@nix.lv>
* Disable all typographic replacements in markdown renderer
Previously we only disabled some of them. This disables all the default
replacements that goldmark's typographer extension offers, matching
GitHub's renderer.
Ref: https://github.com/yuin/goldmark#typographer-extension
Fixes: https://github.com/go-gitea/gitea/issues/11001
* remove typographer extension completely
* fix test
* really fix test
Co-authored-by: Lauris BH <lauris@nix.lv>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
When matching emoji, use a regex built from the data we have instead of something generic using unicode ranges. A generic regex can't tell the difference between two separate emoji next to each other or one emoji that is built out of two separate emoji next to each other.
This means that emoji that are next to each other without space in between will be now accurately spanned individually with proper title etc...
GH has different HardBreaks behaviour for markdown comments and documents.
Comments have hard breaks and documents have soft breaks - therefore Gitea's rendering will always be different from GH's if we only provide one setting.
Here we split the setting in to two - one for documents and one for comments and other things.
Signed-off-by: Andrew Thornton art27@cantab.net
Changes to index.js as per @silverwind
Co-authored-by: silverwind <me@silverwind.io>
Changes to docs as per @guillep2k
Co-authored-by: guillep2k <18600385+guillep2k@users.noreply.github.com>
* Add test
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Restore checkbox rendering and prevent poor sanitization of spans
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Also fix preview context
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Also fix preview context
Signed-off-by: Andrew Thornton <art27@cantab.net>
Now that emojify.js has been removed, get rid of all instances of has-emoji class that was only used for that. Support for rendering shortcodes should remain in all of these places so it should still work the same.
Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: Lauris BH <lauris@nix.lv>
* Support unicode emojis and remove emojify.js
This PR replaces all use of emojify.js and adds unicode emoji support to various areas of gitea.
This works in a few ways:
First it adds emoji parsing support into gitea itself. This allows us to
* Render emojis from valid alias (😄)
* Detect unicode emojis and let us put them in their own class with proper aria-labels and styling
* Easily allow for custom "emoji"
* Support all emoji rendering and features without javascript
* Uses plain unicode and lets the system render in appropriate emoji font
* Doesn't leave us relying on external sources for updates/fixes/features
That same list of emoji is also used to create a json file which replaces the part of emojify.js that populates the emoji search tribute. This file is about 35KB with GZIP turned on and I've set it to load after the page renders to not hinder page load time (and this removes loading emojify.js also)
For custom "emoji" it uses a pretty simple scheme of just looking for /emojis/img/name.png where name is something a user has put in the "allowed reactions" setting we already have. The gitea reaction that was previously hard coded into a forked copy of emojify.js is included and works as a custom reaction under this method.
The emoji data sourced here is from https://github.com/github/gemoji which is the gem library Github uses for their emoji rendering (and a data source for other sites). So we should be able to easily render any emoji and :alias: that Github can, removing any errors from migrated content. They also update it as well, so we can sync when there are new unicode emoji lists released.
I've included a slimmed down and slightly modified forked copy of https://github.com/knq/emoji to make up our own emoji module. The code is pretty straight forward and again allows us to have a lot of flexibility in what happens.
I had seen a few comments about performance in some of the other threads if we render this ourselves, but there doesn't seem to be any issue here. In a test it can parse, convert, and render 1,000 emojis inside of a large markdown table in about 100ms on my laptop (which is many more emojis than will ever be in any normal issue). This also prevents any flickering and other weirdness from using javascript to render some things while using go for others.
Not included here are image fall back URLS. I don't really think they are necessary for anything new being written in 2020. However, managing the emoji ourselves would allow us to add these as a feature later on if it seems necessary.
Fixes: https://github.com/go-gitea/gitea/issues/9182
Fixes: https://github.com/go-gitea/gitea/issues/8974
Fixes: https://github.com/go-gitea/gitea/issues/8953
Fixes: https://github.com/go-gitea/gitea/issues/6628
Fixes: https://github.com/go-gitea/gitea/issues/5130
* add new shared function emojiHTML
* don't increase emoji size in issue title
* Update templates/repo/issue/view_content/add_reaction.tmpl
Co-Authored-By: 6543 <6543@obermui.de>
* Support for emoji rendering in various templates
* Render code and review comments as they should be
* Better way to handle mail subjects
* insert unicode from tribute selection
* Add template helper for plain text when needed
* Use existing replace function I forgot about
* Don't include emoji greater than Unicode Version 12
Only include emoji and aliases in JSON
* Update build/generate-emoji.go
* Tweak regex slightly to really match everything including random invisible characters. Run tests for every emoji we have
* final updates
* code review
* code review
* hard code gitea custom emoji to match previous behavior
* Update .eslintrc
Co-Authored-By: silverwind <me@silverwind.io>
* disable preempt
Co-authored-by: silverwind <me@silverwind.io>
Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: Lauris BH <lauris@nix.lv>
Co-authored-by: guillep2k <18600385+guillep2k@users.noreply.github.com>
zeripath
KN4CK3R
Lunny Xiao
KN4CK3R
silverwind
6543
Nuno Silva
a1012112796
mrsdizzie
Jimmy Praet
kolaente
Pranav Nachnekar
John Olheiser