There was a bug introduced in #21352 due to a change of behaviour caused
by #19280. This causes a panic on running the default doctor checks
because the panic introduced by #19280 assumes that the only way
opts.StdOut and opts.Stderr can be set in RunOpts is deliberately.
Unfortunately, when running a git.Command the provided RunOpts can be
set, therefore if you share a common set of RunOpts these two values can
be set by the previous commands.
This PR stops using common RunOpts for the commands in that doctor check
but secondly stops RunCommand variants from changing the provided
RunOpts.
Signed-off-by: Andrew Thornton <art27@cantab.net>
Fix#19513
This PR introduce a new db method `InTransaction(context.Context)`,
and also builtin check on `db.TxContext` and `db.WithTx`.
There is also a new method `db.AutoTx` has been introduced but could be used by other PRs.
`WithTx` will always open a new transaction, if a transaction exist in context, return an error.
`AutoTx` will try to open a new transaction if no transaction exist in context.
That means it will always enter a transaction if there is no error.
Co-authored-by: delvh <dev.lh@web.de>
Co-authored-by: 6543 <6543@obermui.de>
There are far too many error reports regarding timeouts from migrations.
We should adjust error report to suggest increasing this timeout.
Ref #20680
Signed-off-by: Andrew Thornton <art27@cantab.net>
This PR enhances the CORS middleware usage by allowing for the headers
to be configured in `app.ini`.
Fixes#21746
Co-authored-by: KN4CK3R <admin@oldschoolhack.me>
Co-authored-by: John Olheiser <john.olheiser@gmail.com>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Attempt clarify the AppWorkPath in the documentation by using different
notation and adding a section to the start of the cheat sheet.
Fix#21523
Signed-off-by: Andrew Thornton <art27@cantab.net>
Signed-off-by: Andrew Thornton <art27@cantab.net>
Related #20471
This PR adds global quota limits for the package registry. Settings for
individual users/orgs can be added in a seperate PR using the settings
table.
Co-authored-by: Lauris BH <lauris@nix.lv>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Close https://github.com/go-gitea/gitea/issues/21640
Before: Gitea can create users like ".xxx" or "x..y", which is not
ideal, it's already a consensus that dot filenames have special
meanings, and `a..b` is a confusing name when doing cross repo compare.
After: stricter
Co-authored-by: Jason Song <i@wolfogre.com>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: delvh <dev.lh@web.de>
Fix https://github.com/go-gitea/gitea/issues/21666
Caused by https://github.com/go-gitea/gitea/pull/19663
Before: when install, the INTERNAL_TOKEN was always generated and saved.
But the internal token may be already there by INTERNAL_TOKEN_URI
After: INTERNAL_TOKEN_URI file must be non-empty. When install, skip
internal token generation if the token exists.
_This is a different approach to #20267, I took the liberty of adapting
some parts, see below_
## Context
In some cases, a weebhook endpoint requires some kind of authentication.
The usual way is by sending a static `Authorization` header, with a
given token. For instance:
- Matrix expects a `Bearer <token>` (already implemented, by storing the
header cleartext in the metadata - which is buggy on retry #19872)
- TeamCity #18667
- Gitea instances #20267
- SourceHut https://man.sr.ht/graphql.md#authentication-strategies (this
is my actual personal need :)
## Proposed solution
Add a dedicated encrypt column to the webhook table (instead of storing
it as meta as proposed in #20267), so that it gets available for all
present and future hook types (especially the custom ones #19307).
This would also solve the buggy matrix retry #19872.
As a first step, I would recommend focusing on the backend logic and
improve the frontend at a later stage. For now the UI is a simple
`Authorization` field (which could be later customized with `Bearer` and
`Basic` switches):
The header name is hard-coded, since I couldn't fine any usecase
justifying otherwise.
## Questions
- What do you think of this approach? @justusbunsi @Gusted @silverwind
- ~~How are the migrations generated? Do I have to manually create a new
file, or is there a command for that?~~
- ~~I started adding it to the API: should I complete it or should I
drop it? (I don't know how much the API is actually used)~~
## Done as well:
- add a migration for the existing matrix webhooks and remove the
`Authorization` logic there
_Closes #19872_
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: Gusted <williamzijl7@hotmail.com>
Co-authored-by: delvh <dev.lh@web.de>
I found myself wondering whether a PR I scheduled for automerge was
actually merged. It was, but I didn't receive a mail notification for it
- that makes sense considering I am the doer and usually don't want to
receive such notifications. But ideally I want to receive a notification
when a PR was merged because I scheduled it for automerge.
This PR implements exactly that.
The implementation works, but I wonder if there's a way to avoid passing
the "This PR was automerged" state down so much. I tried solving this
via the database (checking if there's an automerge scheduled for this PR
when sending the notification) but that did not work reliably, probably
because sending the notification happens async and the entry might have
already been deleted. My implementation might be the most
straightforward but maybe not the most elegant.
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: Lauris BH <lauris@nix.lv>
Co-authored-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
A patch to #17335.
Just like AppPath, Gitea writes its own CustomConf into git hook scripts
too. If Gitea's CustomConf changes, then the git push may fail.
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
This patch provide a mechanism to disable
[sitemap](https://github.com/go-gitea/gitea/pull/18407).
Signed-off-by: Xinyu Zhou <i@sourcehut.net>
Co-authored-by: 6543 <6543@obermui.de>
Follow-up to #21352: when testing this out, I found out that the syntax
for the `symbolic-ref` command was different than originally assumed in
that PR. Never got around to submitting a PR until now.
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Our implementation of `IsZero` can't work. An "empty" timestamp (= 0)
calls `time.Unix(int64(ts), 0).IsZero()` which is always `false`. Only
`time.Time{}.IsZero()` is `true`.
We call this method ~~only at one place~~ and there the value
(`UpdatedUnix`) should be always != 0 so this PR may not have
consequences.
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
I noticed an admin is not allowed to upload packages for other users
because `ctx.IsSigned` was not set.
I added a check for `user.IsActive` and `user.ProhibitLogin` too because
both was not checked. Tests enforce this now.
Co-authored-by: Lauris BH <lauris@nix.lv>
The OAuth spec [defines two types of
client](https://datatracker.ietf.org/doc/html/rfc6749#section-2.1),
confidential and public. Previously Gitea assumed all clients to be
confidential.
> OAuth defines two client types, based on their ability to authenticate
securely with the authorization server (i.e., ability to
> maintain the confidentiality of their client credentials):
>
> confidential
> Clients capable of maintaining the confidentiality of their
credentials (e.g., client implemented on a secure server with
> restricted access to the client credentials), or capable of secure
client authentication using other means.
>
> **public
> Clients incapable of maintaining the confidentiality of their
credentials (e.g., clients executing on the device used by the resource
owner, such as an installed native application or a web browser-based
application), and incapable of secure client authentication via any
other means.**
>
> The client type designation is based on the authorization server's
definition of secure authentication and its acceptable exposure levels
of client credentials. The authorization server SHOULD NOT make
assumptions about the client type.
https://datatracker.ietf.org/doc/html/rfc8252#section-8.4
> Authorization servers MUST record the client type in the client
registration details in order to identify and process requests
accordingly.
Require PKCE for public clients:
https://datatracker.ietf.org/doc/html/rfc8252#section-8.1
> Authorization servers SHOULD reject authorization requests from native
apps that don't use PKCE by returning an error message
Fixes#21299
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Previously mentioning a user would link to its profile, regardless of
whether the user existed. This change tests if the user exists and only
if it does - a link to its profile is added.
* Fixes#3444
Signed-off-by: Yarden Shoham <hrsi88@gmail.com>
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
It distractingly shows up on unit tests
* Looks like a leftover from #20571
Signed-off-by: Yarden Shoham <hrsi88@gmail.com>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
At the moment a repository reference is needed for webhooks. With the
upcoming package PR we need to send webhooks without a repository
reference. For example a package is uploaded to an organization. In
theory this enables the usage of webhooks for future user actions.
This PR removes the repository id from `HookTask` and changes how the
hooks are processed (see `services/webhook/deliver.go`). In a follow up
PR I want to remove the usage of the `UniqueQueue´ and replace it with a
normal queue because there is no reason to be unique.
Co-authored-by: 6543 <6543@obermui.de>
A lot of our code is repeatedly testing if individual errors are
specific types of Not Exist errors. This is repetitative and unnecesary.
`Unwrap() error` provides a common way of labelling an error as a
NotExist error and we can/should use this.
This PR has chosen to use the common `io/fs` errors e.g.
`fs.ErrNotExist` for our errors. This is in some ways not completely
correct as these are not filesystem errors but it seems like a
reasonable thing to do and would allow us to simplify a lot of our code
to `errors.Is(err, fs.ErrNotExist)` instead of
`package.IsErr...NotExist(err)`
I am open to suggestions to use a different base error - perhaps
`models/db.ErrNotExist` if that would be felt to be better.
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: delvh <dev.lh@web.de>
Fixes#21379
The commits are capped by `setting.UI.FeedMaxCommitNum` so
`len(commits)` is not the correct number. So this PR adds a new
`TotalCommits` field.
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
This changes the rendering logic of issue titles. If a substring in an
issue title is enclosed with a pair of backticks, it'll be rendered with
a monospace font (HTML `code` tag).
* Closes#20887
Signed-off-by: Yarden Shoham <hrsi88@gmail.com>
Co-authored-by: Gusted <williamzijl7@hotmail.com>
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
Co-authored-by: 6543 <6543@obermui.de>
After some discussion, introduce a new slice `brokenArgs` to make
`gitCmd.Run()` return errors if any dynamic argument is invalid.
Co-authored-by: delvh <dev.lh@web.de>
Due to a bug in presumably an older version of Gitea, multiple of my
repositories still have their HEADs pointing to a `master` branch while
the default branch on the UI is listed as `main`. This adds a `gitea
doctor` command that will fix all of the HEAD references for repos when
they're not synchronized with the default branch in the DB.
This will help with cloning to ensure that git automatically checks out
the right branch, instead of a nonexistent one.
Note: I'm not sure if I actually need to do more other than add a file
here. Will try testing this out on my server soon.
Co-authored-by: zeripath <art27@cantab.net>
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
Fixes#17655
If you rename `user1` to `user2` and visit `/user1` you get redirected
to `/user2`. But if you visit `/user1/repo` you just get a 404 error.
With this PR the user is redirected to `/user2/repo`.
Partially fix#21213.
This PR will get client IP address from SSH_CONNECTION env which should
be the first field of that. And deliver it to the internal API so Gitea
routers could record the real IP from SSH requests.
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: zeripath <art27@cantab.net>
We should only log CheckPath errors if they are not simply due to
context cancellation - and we should add a little more context to the
error message.
Fix#20709
Signed-off-by: Andrew Thornton <art27@cantab.net>
There are repeated issues reported whereby users are unable to interpret
the template errors. This PR adds some (somewhat complex) error handling
to the panic recovery for template renderering but hopefully makes the
interpretation of the error easier.
Reference #21344
Signed-off-by: Andrew Thornton <art27@cantab.net>
Close#20315 (fix the panic when parsing invalid input), Speed up #20231 (use ls-tree without size field)
Introduce ListEntriesRecursiveFast (ls-tree without size) and ListEntriesRecursiveWithSize (ls-tree with size)
Calls to ToCommit are very slow due to fetching diffs, analyzing files.
This patch lets us supply `stat` as false to speed fetching a commit
when we don't need the diff.
/v1/repo/commits has a default `stat` set as true now. Set to false to
experience fetching thousands of commits per second instead of 2-5 per
second.
There is a small bug in #20571 whereby `$a a$b b$` will not be correctly
detected as a math inline block of `a a$b b`. This PR fixes this.
Also reenable test cases as per #21340
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
Only load SECRET_KEY and INTERNAL_TOKEN if they exist.
Never write the config file if the keys do not exist, which was only a fallback for Gitea upgraded from < 1.5
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
This adds an api endpoint `/files` to PRs that allows to get a list of changed files.
built upon #18228, reviews there are included
closes https://github.com/go-gitea/gitea/issues/654
Co-authored-by: Anton Bracke <anton@ju60.de>
Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
This PR would presumably
Fix#20522Fix#18773Fix#19069Fix#21077Fix#13622
-----
1. Check whether unit type is currently enabled
2. Check if it _will_ be enabled via opt
3. Allow modification as necessary
Signed-off-by: jolheiser <john.olheiser@gmail.com>
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
Co-authored-by: 6543 <6543@obermui.de>
The behaviour of `PreventSurroundingPre` has changed in
https://github.com/alecthomas/chroma/pull/618 so that apparently it now
causes line wrapper tags to be no longer emitted, but we need some form
of indication to split the HTML into lines, so I did what
https://github.com/yuin/goldmark-highlighting/pull/33 did and added the
`nopWrapper`.
Maybe there are more elegant solutions but for some reason, just
splitting the HTML string on `\n` did not work.
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
Go creates temporary files as 600, but sometimes we want the group to be able to read them (for example,
for another user to back up the storage)
This PR applies the umask to the renamed tmp files in local storage.
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
Close#20098, in the NPM registry API, implemented to match what's described by https://github.com/npm/registry/blob/master/docs/REGISTRY-API.md#get-v1search
Currently have only implemented the bare minimum to work with the [Unity Package Manager](https://docs.unity3d.com/Manual/upm-ui.html).
Co-authored-by: Jack Vine <jackv@jack-lemur-suse.cat-prometheus.ts.net>
Co-authored-by: KN4CK3R <admin@oldschoolhack.me>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
Fix#20863
When REQUIRE_SIGNIN_VIEW = true, even with public repositories, you can only see them after you login. The packages should not be accessed without login.
Co-authored-by: Lauris BH <lauris@nix.lv>
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
Follows #21119
The manual length check doesn't make sense nowadays:
1. The length check is already done by form's `binding:MaxSize` (then the manual check is unnecessary)
2. The CreateRepository doesn't have such check (then the manual check is inconsistent)
So this PR removes these manual length checks.
Both allow only limited characters. If you input more, you will get a error
message. So it make sense to limit the characters of the input fields.
Slightly relax the MaxSize of repo's Description and Website
Using `append(args, strings.Fields(arg)...)` is dangerous, it may
generate incorrect results.
For example: `arg1 "the dangerous"` will be splitted to 3 arguments:
`arg1`, `"the`, `dangerous"`. In some cases the incorrect arguments may
lead to security problems.
This fixes#5709 and #17316 by changing the order of listed branches
and tags to show the ones with latest commits atop.
It's achieved with changing underlying "show-ref" git command with
"for-each-ref" as suggested in https://stackoverflow.com/a/5188364
Also, it's passing format string so the output matches "show-ref"
command output.
close#5709close#17316
This PR adds mathematical rendering with KaTeX.
The first step is to add a Goldmark extension that detects the latex
(and tex) mathematics delimiters.
The second step to make this extension only run if math support is
enabled.
The second step is to then add KaTeX CSS and JS to the head which will
load after the dom is rendered.
Fix#3445
Signed-off-by: Andrew Thornton <art27@cantab.net>
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: silverwind <me@silverwind.io>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
This commit adds the possibility to click the labels in the message
"User added/removed the X label", which will lead to the issues page,
filtered by the clicked label.
This pull requests aims to fix#19768
_I've tried to follow the contribution guidelines as carefully as
possible, but in case I made a mistake: Please correct me._
<!--
Please check the following:
1. Make sure you are targeting the `main` branch, pull requests on
release branches are only allowed for bug fixes.
2. Read contributing guidelines:
https://github.com/go-gitea/gitea/blob/main/CONTRIBUTING.md
3. Describe what your pull request does and which issue you're targeting
(if any)
-->
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
Remove this small, but unnecessary
[module](https://fomantic-ui.com/elements/image.html) and use `img`
selector over previous `.image`. Did a few tests, could not notice any
visual regression.
Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: Lauris BH <lauris@nix.lv>
The `go-licenses` make task introduced in #21034 is being run on make vendor
and occasionally causes an empty go-licenses file if the vendors need to
change. This should be moved to the generate task as it is a generated file.
Now because of this change we also need to split generation into two separate
steps:
1. `generate-backend`
2. `generate-frontend`
In the future it would probably be useful to make `generate-swagger` part of `generate-frontend` but it's not tolerated with our .drone.yml
Ref #21034
Signed-off-by: Andrew Thornton <art27@cantab.net>
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: delvh <dev.lh@web.de>
Add support for triggering webhook notifications on wiki changes.
This PR contains frontend and backend for webhook notifications on wiki actions (create a new page, rename a page, edit a page and delete a page). The frontend got a new checkbox under the Custom Event -> Repository Events section. There is only one checkbox for create/edit/rename/delete actions, because it makes no sense to separate it and others like releases or packages follow the same schema.
The actions itself are separated, so that different notifications will be executed (with the "action" field). All the webhook receivers implement the new interface method (Wiki) and the corresponding tests.
When implementing this, I encounter a little bug on editing a wiki page. Creating and editing a wiki page is technically the same action and will be handled by the ```updateWikiPage``` function. But the function need to know if it is a new wiki page or just a change. This distinction is done by the ```action``` parameter, but this will not be sent by the frontend (on form submit). This PR will fix this by adding the ```action``` parameter with the values ```_new``` or ```_edit```, which will be used by the ```updateWikiPage``` function.
I've done integration tests with matrix and gitea (http).
Fix#16457
Signed-off-by: Aaron Fischer <mail@aaron-fischer.net>
A testing cleanup.
This pull request replaces `os.MkdirTemp` with `t.TempDir`. We can use the `T.TempDir` function from the `testing` package to create temporary directory. The directory created by `T.TempDir` is automatically removed when the test and all its subtests complete.
This saves us at least 2 lines (error check, and cleanup) on every instance, or in some cases adds cleanup that we forgot.
Reference: https://pkg.go.dev/testing#T.TempDir
```go
func TestFoo(t *testing.T) {
// before
tmpDir, err := os.MkdirTemp("", "")
require.NoError(t, err)
defer os.RemoveAll(tmpDir)
// now
tmpDir := t.TempDir()
}
```
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
When setting.Git.DisablePartialClone is set to false then the web server will add filter support to web http. It does this by using`-c` command arguments but this will not work on gitea serv as the upload-pack and receive-pack commands do not support this.
Instead we move these options into the .gitconfig instead.
Fix#20400
Signed-off-by: Andrew Thornton <art27@cantab.net>
When migrating add several more important sanity checks:
* SHAs must be SHAs
* Refs must be valid Refs
* URLs must be reasonable
Signed-off-by: Andrew Thornton <art27@cantab.net>
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: techknowlogick <matti@mdranta.net>
* feat: extend issue template for yaml
* feat: support yaml template
* feat: render form to markdown
* feat: support yaml template for pr
* chore: rename to Fields
* feat: template unmarshal
* feat: split template
* feat: render to markdown
* feat: use full name as template file name
* chore: remove useless file
* feat: use dropdown of fomantic ui
* feat: update input style
* docs: more comments
* fix: render text without render
* chore: fix lint error
* fix: support use description as about in markdown
* fix: add field class in form
* chore: generate swagger
* feat: validate template
* feat: support is_nummber and regex
* test: fix broken unit tests
* fix: ignore empty body of md template
* fix: make multiple easymde editors work in one page
* feat: better UI
* fix: js error in pr form
* chore: generate swagger
* feat: support regex validation
* chore: generate swagger
* fix: refresh each markdown editor
* chore: give up required validation
* fix: correct issue template candidates
* fix: correct checkboxes style
* chore: ignore .hugo_build.lock in docs
* docs: separate out a new doc for merge templates
* docs: introduce syntax of yaml template
* feat: show a alert for invalid templates
* test: add case for a valid template
* fix: correct attributes of required checkbox
* fix: add class not-under-easymde for dropzone
* fix: use more back-quotes
* chore: remove translation in zh-CN
* fix EasyMDE statusbar margin
* fix: remove repeated blocks
* fix: reuse regex for quotes
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
The recovery, API, Web and package frameworks all create their own HTML
Renderers. This increases the memory requirements of Gitea
unnecessarily with duplicate templates being kept in memory.
Further the reloading framework in dev mode for these involves locking
and recompiling all of the templates on each load. This will potentially
hide concurrency issues and it is inefficient.
This PR stores the templates renderer in the context and stores this
context in the NormalRoutes, it then creates a fsnotify.Watcher
framework to watch files.
The watching framework is then extended to the mailer templates which
were previously not being reloaded in dev.
Then the locales are simplified to a similar structure.
Fix#20210Fix#20211Fix#20217
Signed-off-by: Andrew Thornton <art27@cantab.net>
Adds a new option to only show relevant repo's on the explore page, for bigger Gitea instances like Codeberg this is a nice option to enable to make the explore page more populated with unique and "high" quality repo's. A note is shown that the results are filtered and have the possibility to see the unfiltered results.
Co-authored-by: vednoc <vednoc@protonmail.com>
Co-authored-by: delvh <dev.lh@web.de>
Co-authored-by: 6543 <6543@obermui.de>
* Set no-tags in git fetch on compare
In the compare endpoint the git fetch is restricted to a certain branch however,
this does not completely prevent tag acquisition/pollution as git fetch will collect
any tags on that branch.
This causes pollution of the tag namespace and could cause confusion by users.
This PR adds `--no-tags` to the `git fetch` call.
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Update modules/git/repo_compare.go
* Update modules/git/repo_compare.go
Signed-off-by: Andrew Thornton <art27@cantab.net>
This should solve the main problem of dynamic assets getting stale after
a version upgrade. Everything not affected will use query-string based
cache busting, which includes files loaded via HTML or worker scripts.
This PR adds functionality to allow Gitea to sit behind an
HAProxy and HAProxy protocolled connections directly.
Fix#7508
Signed-off-by: Andrew Thornton <art27@cantab.net>
The graceful manager waits for 4 listeners to be created or to be told that they are not needed. If it is not told about them it will indefinitely and timeout.
This leads to SVC hosts not being told of being in the readyState but on Unix would lead to the termination of the process.
There was an unfortunate regression in #20299 which missed this subtly and in the case whereby SSH is disabled the `builtinUnused()` is not called.
This PR adds a call to `builtinUnused()` when not using the builtin ssh to allow `createServerWaitGroup.Done()` to be called.
In addition it was noted that the if/else clauses for timeout informing of the SVC host were in the wrong order. These have been swapped.
Fix#20609
* Disable doctor logging on panic
If permissions are incorrect for writing to the doctor log simply disable the log file
instead of panicing.
Related #20570
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Update cmd/doctor.go
* Update cmd/doctor.go
Co-authored-by: delvh <dev.lh@web.de>
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: delvh <dev.lh@web.de>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
This adds support for getting the user's full name from the reverse
proxy in addition to username and email.
Tested locally with caddy serving as reverse proxy with Tailscale
authentication.
Signed-off-by: Will Norris <will@tailscale.com>
Signed-off-by: Will Norris <will@tailscale.com>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
The use of `--follow` makes getting these commits very slow on large repositories
as it results in searching the whole commit tree for a blob.
Now as nice as the results of `--follow` are, I am uncertain whether it is really
of sufficient importance to keep around.
Fix#20764
Signed-off-by: Andrew Thornton <art27@cantab.net>
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
By default Gitea will always unlink any sockets that are provided using the `LISTEN_FDS` environment variable. This is because it uses this variable to handle passing when it is doing a graceful restart. However, this same mechanism is used by systemd - which explicitly expects that passed in sockets should not be unlinked by the receiving process.
This PR adjusts Gitea's graceful restart mechanism to use an additional environment variable which tracks if a listening socket was opened by Gitea - and therefore should be unlinked on shutdown by Gitea.
Fix#20490
Co-authored-by: zeripath <art27@cantab.net>
This PR rewrites the invisible unicode detection algorithm to more
closely match that of the Monaco editor on the system. It provides a
technique for detecting ambiguous characters and relaxes the detection
of combining marks.
Control characters are in addition detected as invisible in this
implementation whereas they are not on monaco but this is related to
font issues.
Close#19913
Signed-off-by: Andrew Thornton <art27@cantab.net>
This operation that shifts the content from title to data-content is
useless when we can directly render the expected HTML instead.
This change does prevent these tooltips from working when the user has
JS disabled in their browser, but I think we made it clear by now that
JS is required for gitea to work properly.
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
* merge `CheckLFSVersion` into `InitFull` (renamed from `InitWithSyncOnce`)
* remove the `Once` during git init, no data-race now
* for doctor sub-commands, `InitFull` should only be called in initialization stage
Co-authored-by: zeripath <art27@cantab.net>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
* Added support for Pub packages.
* Update docs/content/doc/packages/overview.en-us.md
Co-authored-by: Gergely Nagy <algernon@users.noreply.github.com>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: Gergely Nagy <algernon@users.noreply.github.com>
Co-authored-by: Lauris BH <lauris@nix.lv>
This enables git.Command's Run to optionally use the given context directly so its deadline will be respected. Otherwise, it falls back to the previous behavior of using the supplied timeout or a default timeout value of 360 seconds.
repo's serviceRPC() calls now use the context's deadline (which is unset/unlimited) instead of the default 6-minute timeout. This means that large repo clones will no longer arbitrarily time out on the upload-pack step, and pushes can take longer than 6 minutes on the receive-pack step.
Fixes#20680
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
* increase default page sizes to 20
* adjust docs to reflect that the setting is used all over the place
* fix tests
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
* Executable check always returns true for windows
Windows doesn't have the concept of "executable" POSIX bits so for now always return true to minimise doctor and logging noise. Addresses #20636
* gofmt tweak
* Update modules/repository/hooks.go
Co-authored-by: silverwind <me@silverwind.io>
* gofmt comment line
Co-authored-by: silverwind <me@silverwind.io>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: Lauris BH <lauris@nix.lv>
- Git only decides to use the Wire 2 protocol when `git
{receive,upload}-pack` receive the `GIT_PROTOCOL` environment with as
value `version=2`. Currently the internal SSH Server wasn't passing this
environment through. The `gitea serv` code already passed all received
environments to the git command, so no code changes there.
* `PROTOCOL`: can be smtp, smtps, smtp+startls, smtp+unix, sendmail, dummy
* `SMTP_ADDR`: domain for SMTP, or path to unix socket
* `SMTP_PORT`: port for SMTP; defaults to 25 for `smtp`, 465 for `smtps`, and 587 for `smtp+startls`
* `ENABLE_HELO`, `HELO_HOSTNAME`: reverse `DISABLE_HELO` to `ENABLE_HELO`; default to false + system hostname
* `FORCE_TRUST_SERVER_CERT`: replace the unclear `SKIP_VERIFY`
* `CLIENT_CERT_FILE`, `CLIENT_KEY_FILE`, `USE_CLIENT_CERT`: clarify client certificates here
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
* Support localized README
* Slightly simplify getting the readme file and add some tests. Ensure that i18n also
works for docs/ etc.
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Update modules/markup/renderer.go
* Update modules/markup/renderer.go
* Update modules/markup/renderer.go
Co-authored-by: Andrew Thornton <art27@cantab.net>
- Add a new push mirror to specific repository
- Sync now ( send all the changes to the configured push mirrors )
- Get list of all push mirrors of a repository
- Get a push mirror by ID
- Delete push mirror by ID
Signed-off-by: Mohamed Sekour <mohamed.sekour@exfo.com>
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: zeripath <art27@cantab.net>
* Add latest commit's SHA to content response
- When requesting the contents of a filepath, add the latest commit's
SHA to the requested file.
- Resolves#12840
* Add swagger
* Fix NPE
* Fix tests
* Hook into LastCommitCache
* Move AddLastCommitCache to a common nogogit and gogit file
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Prevent NPE
Co-authored-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
- Always respect the user's configured mime type map
- Allow more types like image/pdf/video/audio to serve with correct content-type
- Shorten cache duration of raw files to 5 minutes, matching GitHub
- Don't set `content-disposition: attachment`, let the browser decide whether it wants to download or display a file directly
- Implement rfc5987 for filenames, remove previous hack. Confirmed it working in Safari.
- Make PDF attachment work in Safari by removing `sandbox` attribute.
This change will make a lot more file types open directly in browser now. Logic should generally be more readable than before with less `if` nesting and such.
Replaces: https://github.com/go-gitea/gitea/pull/20460
Replaces: https://github.com/go-gitea/gitea/pull/20455
Fixes: https://github.com/go-gitea/gitea/issues/20404
The code in modules/ssh/ssh.go:sessionHandler() currently cause an error to be
logged if `gitea serv` exits with a exit(1). This logging is useless because the
accompanying stderr is not provided and in any case the exit(1) is most likely due
to permissions errors.
Further it then causes the EOF to be logged - even though this is not helpful.
This PR simply checks the errors returned and stops logging them.
In the case of misconfigurations causing `gitea serv` to fail with exit(1)
the current logging is not helpful at determining this and users should simply
review the message passed over the ssh connection.
Fix#20473
Signed-off-by: Andrew Thornton <art27@cantab.net>
Sometimes users want to receive email notifications of messages they create or reply to,
Added an option to personal preferences to allow users to choose
Closes#20149
The LastCommitCache code is a little complex and there is unnecessary
duplication between the gogit and nogogit variants.
This PR adds the LastCommitCache as a field to the git.Repository and
pre-creates it in the ReferencesGit helpers etc. There has been some
simplification and unification of the variant code.
Signed-off-by: Andrew Thornton <art27@cantab.net>
Use Unicode placeholders to replace HTML tags and HTML entities first, then do diff, then recover the HTML tags and HTML entities. Now the code diff with highlight has stable behavior, and won't emit broken tags.
`no-transform` allegedly disables CloudFlare auto-minify and we did not
set caching headers on html or api requests, which seems good to have
regardless.
Transformation is still allowed for asset requests.
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
Co-authored-by: Andrew Thornton <art27@cantab.net>
A lot of existing packages do not conform to SemVer, yet, they should be allowed
in the Conan package registry as-is. To achieve this, remove the SemVer check
from `NewRecipeReference`, and replace it with a simple empty string check.
A unit test with a non-semver version is also included.
Fixes#20405.
Signed-off-by: Gergely Nagy <me@gergo.csillger.hu>
Co-authored-by: KN4CK3R <admin@oldschoolhack.me>
Use body text color in for links in the repository files table
Issue/PR links (`.ref-issue`) will not be affected, as seen in other git services.
Co-authored-by: silverwind <me@silverwind.io>
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
Co-authored-by: Lauris BH <lauris@nix.lv>
Examining Organization membership should not necessarily require sign-in if the organization is public and the members are public. Therefore we should adjust `/org/{org}/members` to not require login.
Fix#7501
Signed-off-by: a1012112796 <1012112796@qq.com>
Co-authored-by: zeripath <art27@cantab.net>
When viewing a subdirectory and the latest commit to that directory in
the table, the commit status icon incorrectly showed the status of the
HEAD commit instead of the latest for that directory.
`login_name` (Authentication Sign-in Name) is not included in the response of `adminUserCreate` API.
This PR is to return user-specified `login_name` if there is one.
`PATCH /repos/{owner}/{repo}` API allows users to update `allow_rebase_update`, `default_delete_branch_after_merge`, but `GET /repos/{owner}/{repo}` API does not return these two options, and API users has no other ways to find the state of these two options.
This PR add `allow_rebase_update`, `default_delete_branch_after_merge` to repository query api response.
* Prevent context deadline error propagation in GetCommitsInfo
Although `WalkGitLog` tries to test for `context.DeadlineExceededErr`
there is a small chance that the error will propagate to the reader
before it is recognised. This will cause the error to propagate up to
`renderDirectoryFiles` and cause a http status 500.
Here we check that the error passed is a `DeadlineExceededErr` via error.Is
Fix#20329
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Use default values when provided values are empty
- When provided values are empty like `:3000` would imply that host is
empty, use the default value.
- Resolves#20316
* Update database.go
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
- Currently the avatar in the navbar is being vertically aligned to the
top, this caused that the icon besides it isn't being at the middle of
the avatar. Use the `vm` helper class to force the `vertical-align` to
be `middle`.
- Resolves#20292
Creating the directory automatically is not correct.
In other places for ssh key writing (RewriteAllPrincipalKeys / appendAuthorizedKeysToFile, etc), the directory will still be created when updating the keys.
This PR will resolve the confusing and annoying problem: the dummy and empty ".ssh" directory in new git home.
Support synchronizing with the push mirrors whenever new commits are pushed or synced from pull mirror.
Related Issues: #18220
Co-authored-by: delvh <dev.lh@web.de>
Co-authored-by: zeripath <art27@cantab.net>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
* Add git.HOME_PATH
* add legacy file check
* Apply suggestions from code review
Co-authored-by: zeripath <art27@cantab.net>
* pass env GNUPGHOME to git command, move the existing .gitconfig to new home, make the fix for 1.17rc more clear.
* set git.HOME_PATH for docker images to default HOME
* Revert "set git.HOME_PATH for docker images to default HOME"
This reverts commit f120101ddc.
* force Gitea to use a stable GNUPGHOME directory
* extra check to ensure only process dir or symlink for legacy files
* refactor variable name
* The legacy dir check (for 1.17-rc1) could be removed with 1.18 release, since users should have upgraded from 1.17-rc to 1.17-stable
* Update modules/git/git.go
Co-authored-by: Steven Kriegler <61625851+justusbunsi@users.noreply.github.com>
* remove initFixGitHome117rc
* Update git.go
* Update docs/content/doc/advanced/config-cheat-sheet.en-us.md
Co-authored-by: zeripath <art27@cantab.net>
Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: Steven Kriegler <61625851+justusbunsi@users.noreply.github.com>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
- This code is only valid when `refNumeric` exist(otherwise we didn't find
such numeric PR and can skip that check) and give a free-pas to the "BEFORE" check when
`ref` is nil.
- Resolves#20109
Unfortunately it appears that 2048 bit RSA keys can occasionally be created in such
a way that they appear to have 2047 bit length. This PR simply changes our defaults to
allow these.
Fix#20249
Signed-off-by: Andrew Thornton <art27@cantab.net>
The recent changes to add live-reloading to the i18n translation files made the i18n code totally non-concurrent when using dev. This will make discovering other concurrency related issues far more difficult. This PR fixes these, adds some more comments to the code and slightly restructures a few functions.
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Add username check to doctor
- Add a new breaking change detector to Gitea's doctor, which checks if
all users still have a valid username according to Gitea. Given from
time-to-time we need to make changes, either due to new routes or due to
security, it's for a instance's admin to check if all users still have a
valid username.
* Fix extra argument
* Apply suggestions from code review
Co-authored-by: Jimmy Praet <jimmy.praet@telenet.be>
* Apply suggestions from code review
Co-authored-by: delvh <dev.lh@web.de>
Co-authored-by: Jimmy Praet <jimmy.praet@telenet.be>
Co-authored-by: John Olheiser <john.olheiser@gmail.com>
Co-authored-by: techknowlogick <matti@mdranta.net>
Co-authored-by: delvh <dev.lh@web.de>
* Refactor `i18n` to `locale`
- Currently we're using the `i18n` variable naming for the `locale`
struct. This contains locale's specific information and cannot be used
for general i18n purpose, therefore refactoring it to `locale` makes
more sense.
- Ref: https://github.com/go-gitea/gitea/pull/20096#discussion_r906699200
* Update routers/install/install.go
- Completely remove U2F support from 1.18.0, 1.17.0 will be the last
release that U2F is somewhat supported. Users who used U2F would already
be warned about using U2F for a while now and should hopefully already
be migrated. But starting 1.18 definitely remove it.
* Prototyping
* Start work on creating offsets
* Modify tests
* Start prototyping with actual MPH
* Twiddle around
* Twiddle around comments
* Convert templates
* Fix external languages
* Fix latest translation
* Fix some test
* Tidy up code
* Use simple map
* go mod tidy
* Move back to data structure
- Uses less memory by creating for each language a map.
* Apply suggestions from code review
Co-authored-by: delvh <dev.lh@web.de>
* Add some comments
* Fix tests
* Try to fix tests
* Use en-US as defacto fallback
* Use correct slices
* refactor (#4)
* Remove TryTr, add log for missing translation key
* Refactor i18n
- Separate dev and production locale stores.
- Allow for live-reloading in dev mode.
Co-authored-by: zeripath <art27@cantab.net>
* Fix live-reloading & check for errors
* Make linter happy
* live-reload with periodic check (#5)
* Fix tests
Co-authored-by: delvh <dev.lh@web.de>
Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
Co-authored-by: zeripath <art27@cantab.net>
This PR adds a doctor command to write the commit-graphs for the repositories:
`gitea doctor --run check-commit-graphs --fix`
Signed-off-by: Andrew Thornton <art27@cantab.net>
This PR adds a new manager command to switch on SQL logging and to turn it off.
```
gitea manager logging log-sql
gitea manager logging log-sql --off
```
Signed-off-by: Andrew Thornton <art27@cantab.net>
If the user-agent starts with git and user must change password but
hasn't return a 401 with the message.
It must be a 401, git doesn't seem to show the contents of the error message
when we return a 403
Fixes#19090
* go.mod: add go-fed/{httpsig,activity/pub,activity/streams} dependency
go get github.com/go-fed/activity/streams@master
go get github.com/go-fed/activity/pub@master
go get github.com/go-fed/httpsig@master
* activitypub: implement /api/v1/activitypub/user/{username} (#14186)
Return informations regarding a Person (as defined in ActivityStreams
https://www.w3.org/TR/activitystreams-vocabulary/#dfn-person).
Refs: https://github.com/go-gitea/gitea/issues/14186
Signed-off-by: Loïc Dachary <loic@dachary.org>
* activitypub: add the public key to Person (#14186)
Refs: https://github.com/go-gitea/gitea/issues/14186
Signed-off-by: Loïc Dachary <loic@dachary.org>
* activitypub: go-fed conformant Clock instance
Signed-off-by: Loïc Dachary <loic@dachary.org>
* activitypub: signing http client
Signed-off-by: Loïc Dachary <loic@dachary.org>
* activitypub: implement the ReqSignature middleware
Signed-off-by: Loïc Dachary <loic@dachary.org>
* activitypub: hack_16834
Signed-off-by: Loïc Dachary <loic@dachary.org>
* Fix CI checks-backend errors with go mod tidy
Signed-off-by: Anthony Wang <ta180m@pm.me>
* Change 2021 to 2022, properly format package imports
Signed-off-by: Anthony Wang <ta180m@pm.me>
* Run make fmt and make generate-swagger
Signed-off-by: Anthony Wang <ta180m@pm.me>
* Use Gitea JSON library, add assert for pkp
Signed-off-by: Anthony Wang <ta180m@pm.me>
* Run make fmt again, fix err var redeclaration
Signed-off-by: Anthony Wang <ta180m@pm.me>
* Remove LogSQL from ActivityPub person test
Signed-off-by: Anthony Wang <ta180m@pm.me>
* Assert if json.Unmarshal succeeds
Signed-off-by: Anthony Wang <ta180m@pm.me>
* Cleanup, handle invalid usernames for ActivityPub person GET request
Signed-off-by: Anthony Wang <ta180m@pm.me>
* Rename hack_16834 to user_settings
Signed-off-by: Anthony Wang <ta180m@pm.me>
* Use the httplib module instead of http for GET requests
* Clean up whitespace with make fmt
* Use time.RFC1123 and make the http.Client proxy-aware
* Check if digest algo is supported in setting module
* Clean up some variable declarations
* Remove unneeded copy
* Use system timezone instead of setting.DefaultUILocation
* Use named constant for httpsigExpirationTime
* Make pubKey IRI #main-key instead of /#main-key
* Move /#main-key to #main-key in tests
* Implemented Webfinger endpoint.
* Add visible check.
* Add user profile as alias.
* Add actor IRI and remote interaction URL to WebFinger response
* fmt
* Fix lint errors
* Use go-ap instead of go-fed
* Run go mod tidy to fix missing modules in go.mod and go.sum
* make fmt
* Convert remaining code to go-ap
* Clean up go.sum
* Fix JSON unmarshall error
* Fix CI errors by adding @context to Person() and making sure types match
* Correctly decode JSON in api_activitypub_person_test.go
* Force CI rerun
* Fix TestActivityPubPersonInbox segfault
* Fix lint error
* Use @mariusor's suggestions for idiomatic go-ap usage
* Correctly add inbox/outbox IRIs to person
* Code cleanup
* Remove another LogSQL from ActivityPub person test
* Move httpsig algos slice to an init() function
* Add actor IRI and remote interaction URL to WebFinger response
* Update TestWebFinger to check for ActivityPub IRI in aliases
* make fmt
* Force CI rerun
* WebFinger: Add CORS header and fix Href -> Template for remote interactions
The CORS header is needed due to https://datatracker.ietf.org/doc/html/rfc7033#section-5 and fixes some Peertube <-> Gitea federation issues
* make lint-backend
* Make sure Person endpoint has Content-Type application/activity+json and includes PreferredUsername, URL, and Icon
Setting the correct Content-Type is essential for federating with Mastodon
* Use UTC instead of GMT
* Rename pkey to pubKey
* Make sure HTTP request Date in GMT
* make fmt
* dont drop err
* Make sure API responses always refer to username in original case
Copied from what I wrote on #19133 discussion: Handling username case is a very tricky issue and I've already encountered a Mastodon <-> Gitea federation bug due to Gitea considering Ta180m and ta180m to be the same user while Mastodon thinks they are two different users. I think the best way forward is for Gitea to only use the original case version of the username for federation so other AP software don't get confused.
* Move httpsig algs constant slice to modules/setting/federation.go
* Add new federation settings to app.example.ini and config-cheat-sheet
* Return if marshalling error
* Make sure Person IRIs are generated correctly
This commit ensures that if the setting.AppURL is something like "http://127.0.0.1:42567" (like in the integration tests), a trailing slash will be added after that URL.
* If httpsig verification fails, fix Host header and try again
This fixes a very rare bug when Gitea and another AP server (confirmed to happen with Mastodon) are running on the same machine, Gitea fails to verify incoming HTTP signatures. This is because the other AP server creates the sig with the public Gitea domain as the Host. However, when Gitea receives the request, the Host header is instead localhost, so the signature verification fails. Manually changing the host header to the correct value and trying the veification again fixes the bug.
* Revert "If httpsig verification fails, fix Host header and try again"
This reverts commit f53e46c721.
The bug was actually caused by nginx messing up the Host header when reverse-proxying since I didn't have the line `proxy_set_header Host $host;` in my nginx config for Gitea.
* Go back to using ap.IRI to generate inbox and outbox IRIs
* use const for key values
* Update routers/web/webfinger.go
* Use ctx.JSON in Person response to make code cleaner
* Revert "Use ctx.JSON in Person response to make code cleaner"
This doesn't work because the ctx.JSON() function already sends the response out and it's too late to edit the headers.
This reverts commit 95aad98897.
* Use activitypub.ActivityStreamsContentType for Person response Content Type
* Limit maximum ActivityPub request and response sizes to a configurable setting
* Move setting key constants to models/user/setting_keys.go
* Fix failing ActivityPubPerson integration test by checking the correct field for username
* Add a warning about changing settings that can break federation
* Add better comments
* Don't multiply Federation.MaxSize by 1<<20 twice
* Add more better comments
* Fix failing ActivityPubMissingPerson test
We now use ctx.ContextUser so the message printed out when a user does not exist is slightly different
* make generate-swagger
For some reason I didn't realize that /templates/swagger/v1_json.tmpl was machine-generated by make generate-swagger... I've been editing it by hand for three months! 🤦
* Move getting the RFC 2616 time to a separate function
* More code cleanup
* Update go-ap to fix empty liked collection and removed unneeded HTTP headers
* go mod tidy
* Add ed25519 to httpsig algorithms
* Use go-ap/jsonld to add @context and marshal JSON
* Change Gitea user agent from the default to Gitea/Version
* Use ctx.ServerError and remove all remote interaction code from webfinger.go
It appears possible that stopwatches can become orphaned or have been orphaned in
the past.
This PR adds Orphan checks for Stopwatches.
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
* Empty log queue on flush and close
It is possible for log events to remain in the buffer off the multichannelledlog
and thus not be logged despite close or flush.
This PR simply adds a function to empty the queue before closing or flushing.
(Except when the logger is paused.)
Reference #19982
Signed-off-by: Andrew Thornton <art27@cantab.net>
* and do similar for ChannelledLog
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
If there are dangling stopwatches with missing issues there will be repeated
logging of Unable to APIFormat stopwatches. These are unhelpful and instead
we should only log if the error is not an issue not exist error.
And we should also prevent an error on missing issue in GetActiveStopwatch too
Signed-off-by: Andrew Thornton <art27@cantab.net>
Add fetch.writeCommitGraph to gitconfig to ensure that a commit-graph will be written
on git fetch calls.
Signed-off-by: Andrew Thornton <art27@cantab.net>
Allow git push to work when networked file systems with mixed
ownership are used with Gitea docker images >= 1.16.6 or Gitea
binaries running alongside git versions published after 04/2022.
There are circumstances independent of Gitea (networked file systems
with various permission systems) by which the git repositories managed
by Gitea may have mixed owners. It is not a behavior that Gitea have
control over nor is it a problem as long as the permissions for Gitea to
operate are correct. Gitea instances have been operating under these
conditions for a number of years.
It is detected as a potential security risk ( see
GHSA-vw2c-22j4-2fh2
) by the most recent git versions. However, Gitea always runs git
commands with a current directory matching the repository on
which it operates. That makes Gitea immune from this security problem
and it is safe to ignore the mixed owner permission check.
This gitconfig modification is done on a file dedicated to the user
exclusively used by Gitea.
Fixes: #19455
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
Co-authored-by: zeripath <art27@cantab.net>
* clean git support for ver < 2.0
* fine tune tests for markup (which requires git module)
* remove unnecessary comments
* try to fix tests
* try test again
* use const for GitVersionRequired instead of var
* try to fix integration test
* Refactor CheckAttributeReader to make a *git.Repository version
* update document for commit signing with Gitea's internal gitconfig
* update document for commit signing with Gitea's internal gitconfig
Co-authored-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
* Allow render HTML with css/js external links
* Fix bug because of filename escape chars
* Fix lint
* Update docs about new configuration item
* Fix bug of render HTML in sub directory
* Add CSP head for displaying iframe in rendering file
* Fix test
* Apply suggestions from code review
Co-authored-by: delvh <dev.lh@web.de>
* Some improvements
* some improvement
* revert change in SanitizerDisabled of external renderer
* Add sandbox for iframe and support allow-scripts and allow-same-origin
* refactor
* fix
* fix lint
* fine tune
* use single option RENDER_CONTENT_MODE, use sandbox=allow-scripts
* fine tune CSP
* Apply suggestions from code review
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
Co-authored-by: delvh <dev.lh@web.de>
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
* Move access and repo permission to models/perm/access
* fix test
* fix git test
* Move functions sequence
* Some improvements per @KN4CK3R and @delvh
* Move issues related code to models/issues
* Move some issues related sub package
* Merge
* Fix test
* Fix test
* Fix test
* Fix test
* Rename some files
* Move access and repo permission to models/perm/access
* fix test
* Move some git related files into sub package models/git
* Fix build
* fix git test
* move lfs to sub package
* move more git related functions to models/git
* Move functions sequence
* Some improvements per @KN4CK3R and @delvh
* Prettify number of issues
- Use the PrettyNumber function to add commas in large amount of issues.
* Use client-side formatting
* prettify on both server and client
* remove unused i18n entries
* handle more cases, support other int types in PrettyNumber
* specify locale to avoid issues with node default locale
* remove superfluos argument
* introduce template helper, octicon tweaks, js refactor
* Update modules/templates/helper.go
* Apply some suggestions.
* Add comment
* Update templates/user/dashboard/issues.tmpl
Co-authored-by: silverwind <me@silverwind.io>
Co-authored-by: silverwind <me@silverwind.io>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
* Fix copy/paste of empty newlines again
Fixes: https://github.com/go-gitea/gitea/issues/19331
Regressed by: https://github.com/go-gitea/gitea/pull/18270
Needed to do another newline addition to the Chroma output HTML to get
copy/paste work again. The previous replacement conditions are probably
obsolete, but as I'm not 100% sure, I opted to keep them.
Specifically, the Chroma HTML change mentioned in
https://github.com/go-gitea/gitea/pull/18270#issuecomment-1013350246
broke our previous newline replacement for such empty lines.
Also included are a few changes to make the test more pleasant to work
with.
* run go mod tidy
* add util.Dedent
* copy in the code
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: Lauris BH <lauris@nix.lv>
A pr.Reviewer may be nil when migrating from Gitea if this is a team
request review.
We do not migrate teams therefore we cannot map these requests, but we can
migrate user requests.
Signed-off-by: Andrew Thornton <art27@cantab.net>
Automatically add sidebar in the wiki view containing a TOC for the wiki page.
Make the TOC collapsable
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Add breaking change check in doctor
- This patch introduces a new kind of doctor type, breaking. This file
is made to register checks that helps with detecting when a breaking
change might impact a Gitea instance.
- For now the only check here(and the reason of creating this) is to
check if all users in the database has a valid email address, which
might not be the case after
https://github.com/go-gitea/gitea/pull/17688. This _simply_ uses the
validation function to detect and report these cases.
- Helps admins with detecting #19897.
- I have no clue which priority should be and IsDefault is true, because
when breaking change happen and we have a doctor check for it, we can
say "run `gitea doctor` to help you with this and maybe you find other
errors 😉".
* Makes no sense tbh
* Fix copyright
* Update modules/doctor/breaking.go
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: techknowlogick <matti@mdranta.net>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
* Fix GetNote
* Only log errors if the error is not ErrNotExist
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: Andrew Thornton <art27@cantab.net>
* Make AppDataPath absolute against the AppWorkPath if it is not
There are multiple repeated issues whereby a non-absolute provided
APP_DATA_PATH causes strange issues.
This PR simply absolutes the APP_DATA_PATH against the AppWorkPath if
its not so. It also ensures that AppWorkPath is also always absolute.
Ref #19367
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Add logging
Signed-off-by: Andrew Thornton <art27@cantab.net>
* absolute workpath against pwd instead of app path first
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
* Move some repository related code into sub package
* Move more repository functions out of models
* Fix lint
* Some performance optimization for webhooks and others
* some refactors
* Fix lint
* Fix
* Update modules/repository/delete.go
Co-authored-by: delvh <dev.lh@web.de>
* Fix test
* Merge
* Fix test
* Fix test
* Fix test
* Fix test
Co-authored-by: delvh <dev.lh@web.de>
Fixes#12338
This allows use to talk to the API with our ssh certificate (and/or ssh-agent) without needing to fetch an API key or tokens.
It will just automatically work when users have added their ssh principal in gitea.
This needs client code in tea
Update: also support normal pubkeys
ref: https://tools.ietf.org/html/draft-cavage-http-signatures
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: zeripath <art27@cantab.net>
Signed-off-by: Andrew Thornton <art27@cantab.net>
When Gitea is running as PID 1 git will occassionally orphan child processes leading
to (defunct) processes. This PR simply sets Setpgid to true on these child processes
meaning that these defunct processes will also be correctly reaped.
Fix#19077
Signed-off-by: Andrew Thornton <art27@cantab.net>
A `repo_model.Mirror` repository field (`.Repo`) will not automatically
be set, but is used without checking in mirror_pull.go:UpdateAddress.
This will cause an NPE.
This PR changes UpdateAddress to use the helper function GetRepository()
helping prevent future NPEs but also changes modules/context/repo.go to
ensure that the Mirror.Repo is set.
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
In order for web.Wrap to be able to detect if a response has been written
we need to wrap any non-context.ResponseWriters as a such. Otherwise
responses will be incorrectly detected as non-written to and handlers can
double run.
In the case of GZip this handler will change the response to a non-context.RW
and this failure to correctly detect response writing causes fallthrough and
a NPE.
Fix#19839
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Fix inconsistency in doctor output
- Use `logger.Info` instead of `logger.Warn` when no errors were found.
* Update modules/doctor/fix16961.go
Co-authored-by: delvh <dev.lh@web.de>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: delvh <dev.lh@web.de>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
Our character detection algorithm can potentially incorrectly detect utf-8 as iso-8859-x
if there is a truncated character at the end of the partially read file.
This PR changes the detection algorithm to truncated utf8 characters at the end of the
buffer.
Fix#19743
Signed-off-by: Andrew Thornton <art27@cantab.net>
The permissions created in convertRepo use a minimal perm.AccessModeRead instead of
correctly computing the permission for the repository. This incorrect permission is
then reported to the user.
I do not believe that reporting the permissions is helpful and therefore I propose
we simply null these out. The user can check their permissions using a different
endpoint.
Fix#19759
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Update go tool dependencies
Updated all tool dependencies to latest tags, hoping CI will like it.
* fix new lint errors
* handle more strings.Title cases
* remove lint skip
Although the use of LastModified dates for caching of git objects should be
discouraged (as it is not native to git - and there are a LOT of ways this
could be incorrect) - LastModified dates can be a helpful somewhat more human
way of caching for simple cases.
This PR adds this header and handles the If-Modified-Since header to the /raw/
routes.
Fix#18354
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: 6543 <6543@obermui.de>
* Fix indention
Signed-off-by: kolaente <k@knt.li>
* Add option to merge a pr right now without waiting for the checks to succeed
Signed-off-by: kolaente <k@knt.li>
* Fix lint
Signed-off-by: kolaente <k@knt.li>
* Add scheduled pr merge to tables used for testing
Signed-off-by: kolaente <k@knt.li>
* Add status param to make GetPullRequestByHeadBranch reusable
Signed-off-by: kolaente <k@knt.li>
* Move "Merge now" to a seperate button to make the ui clearer
Signed-off-by: kolaente <k@knt.li>
* Update models/scheduled_pull_request_merge.go
Co-authored-by: 赵智超 <1012112796@qq.com>
* Update web_src/js/index.js
Co-authored-by: 赵智超 <1012112796@qq.com>
* Update web_src/js/index.js
Co-authored-by: 赵智超 <1012112796@qq.com>
* Re-add migration after merge
* Fix frontend lint
* Fix version compare
* Add vendored dependencies
* Add basic tets
* Make sure the api route is capable of scheduling PRs for merging
* Fix comparing version
* make vendor
* adopt refactor
* apply suggestion: User -> Doer
* init var once
* Fix Test
* Update templates/repo/issue/view_content/comments.tmpl
* adopt
* nits
* next
* code format
* lint
* use same name schema; rm CreateUnScheduledPRToAutoMergeComment
* API: can not create schedule twice
* Add TestGetBranchNamesForSha
* nits
* new go routine for each pull to merge
* Update models/pull.go
Co-authored-by: a1012112796 <1012112796@qq.com>
* Update models/scheduled_pull_request_merge.go
Co-authored-by: a1012112796 <1012112796@qq.com>
* fix & add renaming sugestions
* Update services/automerge/pull_auto_merge.go
Co-authored-by: a1012112796 <1012112796@qq.com>
* fix conflict relicts
* apply latest refactors
* fix: migration after merge
* Update models/error.go
Co-authored-by: delvh <dev.lh@web.de>
* Update options/locale/locale_en-US.ini
Co-authored-by: delvh <dev.lh@web.de>
* Update options/locale/locale_en-US.ini
Co-authored-by: delvh <dev.lh@web.de>
* adapt latest refactors
* fix test
* use more context
* skip potential edgecases
* document func usage
* GetBranchNamesForSha() -> GetRefsBySha()
* start refactoring
* ajust to new changes
* nit
* docu nit
* the great check move
* move checks for branchprotection into own package
* resolve todo now ...
* move & rename
* unexport if posible
* fix
* check if merge is allowed before merge on scheduled pull
* debugg
* wording
* improve SetDefaults & nits
* NotAllowedToMerge -> DisallowedToMerge
* fix test
* merge files
* use package "errors"
* merge files
* add string names
* other implementation for gogit
* adapt refactor
* more context for models/pull.go
* GetUserRepoPermission use context
* more ctx
* use context for loading pull head/base-repo
* more ctx
* more ctx
* models.LoadIssueCtx()
* models.LoadIssueCtx()
* Handle pull_service.Merge in one DB transaction
* add TODOs
* next
* next
* next
* more ctx
* more ctx
* Start refactoring structure of old pull code ...
* move code into new packages
* shorter names ... and finish **restructure**
* Update models/branches.go
Co-authored-by: zeripath <art27@cantab.net>
* finish UpdateProtectBranch
* more and fix
* update datum
* template: use "svg" helper
* rename prQueue 2 prPatchCheckerQueue
* handle automerge in queue
* lock pull on git&db actions ...
* lock pull on git&db actions ...
* add TODO notes
* the regex
* transaction in tests
* GetRepositoryByIDCtx
* shorter table name and lint fix
* close transaction bevore notify
* Update models/pull.go
* next
* CheckPullMergable check all branch protections!
* Update routers/web/repo/pull.go
* CheckPullMergable check all branch protections!
* Revert "PullService lock via pullID (#19520)" (for now...)
This reverts commit 6cde7c9159a5ea75a10356feb7b8c7ad4c434a9a.
* Update services/pull/check.go
* Use for a repo action one database transaction
* Apply suggestions from code review
* Apply suggestions from code review
Co-authored-by: delvh <dev.lh@web.de>
* Update services/issue/status.go
Co-authored-by: delvh <dev.lh@web.de>
* Update services/issue/status.go
Co-authored-by: delvh <dev.lh@web.de>
* use db.WithTx()
* gofmt
* make pr.GetDefaultMergeMessage() context aware
* make MergePullRequestForm.SetDefaults context aware
* use db.WithTx()
* pull.SetMerged only with context
* fix deadlock in `test-sqlite\#TestAPIBranchProtection`
* dont forget templates
* db.WithTx allow to set the parentCtx
* handle db transaction in service packages but not router
* issue_service.ChangeStatus just had caused another deadlock :/
it has to do something with how notification package is handled
* if we merge a pull in one database transaktion, we get a lock, because merge infoce internal api that cant handle open db sessions to the same repo
* ajust to current master
* Apply suggestions from code review
Co-authored-by: delvh <dev.lh@web.de>
* dont open db transaction in router
* make generate-swagger
* one _success less
* wording nit
* rm
* adapt
* remove not needed test files
* rm less diff & use attr in JS
* ...
* Update services/repository/files/commit.go
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
* ajust db schema for PullAutoMerge
* skip broken pull refs
* more context in error messages
* remove webUI part for another pull
* remove more WebUI only parts
* API: add CancleAutoMergePR
* Apply suggestions from code review
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
* fix lint
* Apply suggestions from code review
* cancle -> cancel
Co-authored-by: delvh <dev.lh@web.de>
* change queue identifyer
* fix swagger
* prevent nil issue
* fix and dont drop error
* as per @zeripath
* Update integrations/git_test.go
Co-authored-by: delvh <dev.lh@web.de>
* Update integrations/git_test.go
Co-authored-by: delvh <dev.lh@web.de>
* more declarative integration tests (dedup code)
* use assert.False/True helper
Co-authored-by: 赵智超 <1012112796@qq.com>
Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: delvh <dev.lh@web.de>
Co-authored-by: zeripath <art27@cantab.net>
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
* chore: add health check endpoint
docs: update document about health check
fix: fix up Sqlite3 ping. current ping will success even if the db file is missing
fix: do not expose privacy information in output field
* refactor: remove HealthChecker struct
* Added `/api/healthz` to install routes.
This was needed for using /api/healthz endpoint in Docker healthchecks,
otherwise, Docker would never become healthy if using healthz endpoint
and users would not be able to complete the installation of Gitea.
* Update modules/cache/cache.go
* fine tune
* Remove unnecessary test code. Now there are 2 routes for installation (and maybe more in future)
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: Marcos de Oliveira <marcossantos@furb.br>
* Only check for non-finished migrating task
- Only check if a non-finished migrating task exists for a mirror before
fetching the mirror details from the database.
- Resolves#19600
- Regression: #19588
* Clarify function
- When a repository is still being migrated, don't try to fetch the
Mirror from the database. Instead skip it. This allows to visit
repositories that are still being migrated and were configured to be
mirrored.
- Resolves#19585
- Regression: #19295
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
* Apply DefaultUserIsRestricted in CreateUser
* Enforce system defaults in CreateUser
Allow for overwrites with CreateUserOverwriteOptions
* Fix compilation errors
* Add "restricted" option to create user command
* Add "restricted" option to create user admin api
* Respect default setting.Service.RegisterEmailConfirm and setting.Service.RegisterManualConfirm where needed
* Revert "Respect default setting.Service.RegisterEmailConfirm and setting.Service.RegisterManualConfirm where needed"
This reverts commit ee95d3e8dc.
Targeting #14936, #15332
Adds a collaborator permissions API endpoint according to GitHub API: https://docs.github.com/en/rest/collaborators/collaborators#get-repository-permissions-for-a-user to retrieve a collaborators permissions for a specific repository.
### Checks the repository permissions of a collaborator.
`GET` `/repos/{owner}/{repo}/collaborators/{collaborator}/permission`
Possible `permission` values are `admin`, `write`, `read`, `owner`, `none`.
```json
{
"permission": "admin",
"role_name": "admin",
"user": {}
}
```
Where `permission` and `role_name` hold the same `permission` value and `user` is filled with the user API object. Only admins are allowed to use this API endpoint.
Adds a feature [like GitHub has](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/creating-a-pull-request-from-a-fork) (step 7).
If you create a new PR from a forked repo, you can select (and change later, but only if you are the PR creator/poster) the "Allow edits from maintainers" option.
Then users with write access to the base branch get more permissions on this branch:
* use the update pull request button
* push directly from the command line (`git push`)
* edit/delete/upload files via web UI
* use related API endpoints
You can't merge PRs to this branch with this enabled, you'll need "full" code write permissions.
This feature has a pretty big impact on the permission system. I might forgot changing some things or didn't find security vulnerabilities. In this case, please leave a review or comment on this PR.
Closes#17728
Co-authored-by: 6543 <6543@obermui.de>
There is a potential rare race possible whereby the c.running channel could
be closed twice. Looking at the code I do not see a need for this c.running
channel and therefore I think we can remove this. (I think the c.running
might have been some attempt to prevent a hang but the use of os.Pipes should
prevent that.)
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: 6543 <6543@obermui.de>
- Doing 64-bit atomic operations on 32-bit machines is a bit tricky by
golang, as they can only be done under certain set of
conditions(https://pkg.go.dev/sync/atomic#pkg-note-BUG).
- This PR fixes such case whereby the conditions weren't met, it moves
the int64 to the first field of the struct, which will 64-bit operations
happening on this property on 32-bit machines.
- Resolves#19518
Within doArchive there is a service goroutine that performs the
archiving function. This goroutine reports its error using a `chan
error` called `done`. Prior to this PR this channel had 0 capacity
meaning that the goroutine would block until the `done` channel was
cleared - however there are a couple of ways in which this channel might
not be read.
The simplest solution is to add a single space of capacity to the
goroutine which will mean that the goroutine will always complete and
even if the `done` channel is not read it will be simply garbage
collected away.
(The PR also contains two other places when setting up the indexers
which do not leak but where the blocking of the sending goroutine is
also unnecessary and so we should just add a small amount of capacity
and let the sending goroutine complete as soon as it can.)
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: 6543 <6543@obermui.de>
If an `os/exec.Command` is passed non `*os.File` as an input/output, go
will create `os.Pipe`s and wait for their closure in `cmd.Wait()`. If
the code following this is responsible for closing `io.Pipe`s or other
handlers then on process death from context cancellation the `Wait` can
hang.
There are two possible solutions:
1. use `os.Pipe` as the input/output as `cmd.Wait` does not wait for these.
2. create a goroutine waiting on the context cancellation that will close the inputs.
This PR provides the second option - which is a simpler change that can
be more easily backported.
Closes#19448
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Set correct PR status on 3way on conflict checking
- When 3-way merge is enabled for conflict checking, it has a new
interesting behavior that it doesn't return any error when it found a
conflict, so we change the condition to not check for the error, but
instead check if conflictedfiles is populated, this fixes a issue
whereby PR status wasn't correctly on conflicted PR's.
- Refactor the mergeable property(which was incorrectly set and lead me this
bug) to be more maintainable.
- Add a dedicated test for conflicting checking, so it should prevent
future issues with this.
* Fix linter
When a mirror repo interval is updated by the UI it is rescheduled with that interval
however the API does not do this. The API also lacks the enable_prune option.
This PR adds this functionality in to the API Edit Repo endpoint.
Signed-off-by: Andrew Thornton <art27@cantab.net>
Do a refactoring to the CSRF related code, remove most unnecessary functions.
Parse the generated token's issue time, regenerate the token every a few minutes.
Reusing `/api/v1` from Gitea UI Pages have pros and cons.
Pros:
1) Less code copy
Cons:
1) API/v1 have to support shared session with page requests.
2) You need to consider for each other when you want to change something about api/v1 or page.
This PR moves all dependencies to API/v1 from UI Pages.
Partially replace #16052
Remove two unmaintained vendor packages `i18n` and `paginater`. Changes:
* Rewrite `i18n` package with a more clear fallback mechanism. Fix an unstable `Tr` behavior, add more tests.
* Refactor the legacy `Paginater` to `Paginator`, test cases are kept unchanged.
Trivial enhancement (no breaking for end users):
* Use the first locale in LANGS setting option as the default, add a log to prevent from surprising users.
There appears to be an intermittent NPE in queue tests relating to the deferred
shutdown/terminate functions.
This PR more formally asserts that shutdown and termination occurs before starting
and finishing the tests but leaves the defer in place to ensure that if there is an
issue shutdown/termination will occur.
Signed-off-by: Andrew Thornton <art27@cantab.net>
Follows: #19284
* The `CopyDir` is only used inside test code
* Rewrite `ToSnakeCase` with more test cases
* The `RedisCacher` only put strings into cache, here we use internal `toStr` to replace the legacy `ToStr`
* The `UniqueQueue` can use string as ID directly, no need to call `ToStr`
Right now, a pull-mirror repo does not get marked as such until *after* the
mirroring completes. In the meantime, it will show up (in API and UI) as a
regular repo.
The main purpose is to refactor the legacy `unknwon/com` package.
1. Remove most imports of `unknwon/com`, only `util/legacy.go` imports the legacy `unknwon/com`
2. Use golangci's depguard to process denied packages
3. Fix some incorrect values in golangci.yml, eg, the version should be quoted string `"1.18"`
4. Use correctly escaped content for `go-import` and `go-source` meta tags
5. Refactor `com.Expand` to our stable (and the same fast) `vars.Expand`, our `vars.Expand` can still return partially rendered content even if the template is not good (eg: key mistach).
Follows #19266, #8553, Close#18553, now there are only three `Run..(&RunOpts{})` functions.
* before: `stdout, err := RunInDir(path)`
* now: `stdout, _, err := RunStdString(&git.RunOpts{Dir:path})`
- Upgrade all JS dependencies minus vue and vue-loader
- Adapt to breaking change of octicons
- Update eslint rules
- Tested Swagger UI, sortablejs and prod build
Continues on from #19202.
Following the addition of pprof labels we can now more easily understand the relationship between a goroutine and the requests that spawn them.
This PR takes advantage of the labels and adds a few others, then provides a mechanism for the monitoring page to query the pprof goroutine profile.
The binary profile that results from this profile is immediately piped in to the google library for parsing this and then stack traces are formed for the goroutines.
If the goroutine is within a context or has been created from a goroutine within a process context it will acquire the process description labels for that process.
The goroutines are mapped with there associate pids and any that do not have an associated pid are placed in a group at the bottom as unbound.
In this way we should be able to more easily examine goroutines that have been stuck.
A manager command `gitea manager processes` is also provided that can export the processes (with or without stacktraces) to the command line.
Signed-off-by: Andrew Thornton <art27@cantab.net>
This addresses https://github.com/go-gitea/gitea/issues/18352
It aims to improve performance (and resource use) of the `SyncReleasesWithTags` operation for pull-mirrors.
For large repositories with many tags, `SyncReleasesWithTags` can be a costly operation (taking several minutes to complete). The reason is two-fold:
1. on sync, every upstream repo tag is compared (for changes) against existing local entries in the release table to ensure that they are up-to-date.
2. the procedure for getting _each tag_ involves a series of git operations
```bash
git show-ref --tags -- v8.2.4477
git cat-file -t 29ab6ce9f36660cffaad3c8789e71162e5db5d2f
git cat-file -p 29ab6ce9f36660cffaad3c8789e71162e5db5d2f
git rev-list --count 29ab6ce9f36660cffaad3c8789e71162e5db5d2f
```
of which the `git rev-list --count` can be particularly heavy.
This PR optimizes performance for pull-mirrors. We utilize the fact that a pull-mirror is always identical to its upstream and rebuild the entire release table on every sync and use a batch `git for-each-ref .. refs/tags` call to retrieve all tags in one go.
For large mirror repos, with hundreds of annotated tags, this brings down the duration of the sync operation from several minutes to a few seconds. A few unscientific examples run on my local machine:
- https://github.com/spring-projects/spring-boot (223 tags)
- before: `0m28,673s`
- after: `0m2,244s`
- https://github.com/kubernetes/kubernetes (890 tags)
- before: `8m00s`
- after: `0m8,520s`
- https://github.com/vim/vim (13954 tags)
- before: `14m20,383s`
- after: `0m35,467s`
I added a `foreachref` package which contains a flexible way of specifying which reference fields are of interest (`git-for-each-ref(1)`) and to produce a parser for the expected output. These could be reused in other places where `for-each-ref` is used. I'll add unit tests for those if the overall PR looks promising.
This follows
* https://github.com/go-gitea/gitea/issues/18553
Introduce `RunWithContextString` and `RunWithContextBytes` to help the refactoring. Add related unit tests. They keep the same behavior to save stderr into err.Error() as `RunInXxx` before.
Remove `RunInDirTimeoutPipeline` `RunInDirTimeoutFullPipeline` `RunInDirTimeout` `RunInDirTimeoutEnv` `RunInDirPipeline` `RunInDirFullPipeline` `RunTimeout`, `RunInDirTimeoutEnvPipeline`, `RunInDirTimeoutEnvFullPipeline`, `RunInDirTimeoutEnvFullPipelineFunc`.
Then remaining `RunInDir` `RunInDirBytes` `RunInDirWithEnv` can be easily refactored in next PR with a simple search & replace:
* before: `stdout, err := RunInDir(path)`
* next: `stdout, _, err := RunWithContextString(&git.RunContext{Dir:path})`
Other changes:
1. When `timeout <= 0`, use default. Because `timeout==0` is meaningless and could cause bugs. And now many functions becomes more simple, eg: `GitGcRepos` 9 lines to 1 line. `Fsck` 6 lines to 1 line.
2. Only set defaultCommandExecutionTimeout when the option `setting.Git.Timeout.Default > 0`
Gitea was not able to supply any authentication parameters to it. So this brings support to do that, along with some light extraction of a couple of bits into some separate functions for easier testing.
I looked at other libraries supporting similar RedisUri-style connection strings (e.g. Lettuce), but it looks like this type of configuration is beyond what would typically be done in a connection string. Since gitea doesn't have configuration options for manually specifying all this redis connection detail, I went ahead and just chose straightforward names for these new parameters.
Strangely #19038 appears to relate to an issue whereby a tag appears to
be listed in `git show-ref --tags` but then does not appear when `git
show-ref --tags -- short_name` is called.
As a solution though I propose to stop the second call as it is
unnecessary and only likely to cause problems.
I've also noticed that the tags calls are wildly inefficient and aren't using the common cat-files - so these have been added.
I've also noticed that the git commit-graph is not being written on mirroring - so I've also added writing this to the migration which should improve mirror rendering somewhat.
Fix#19038
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: 6543 <6543@obermui.de>
The last PR about clone buttons introduced an JS error when visiting an empty repo page:
* https://github.com/go-gitea/gitea/pull/19028
* `Uncaught ReferenceError: isSSH is not defined`, because the variables are scoped and doesn't share between sub templates.
This:
1. Simplify `templates/repo/clone_buttons.tmpl` and make code clear
2. Move most JS code into `initRepoCloneLink`
3. Remove unused `CloneLink.Git`
4. Remove `ctx.Data["DisableSSH"] / ctx.Data["ExposeAnonSSH"] / ctx.Data["DisableHTTP"]`, and only set them when is is needed (eg: deploy keys / ssh keys)
5. Introduce `Data["CloneButton*"]` to provide data for clone buttons and links
6. Introduce `Data["RepoCloneLink"]` for the repo clone link (not the wiki)
7. Remove most `ctx.Data["PageIsWiki"]` because it has been set in the `/wiki` middleware
8. Remove incorrect `quickstart` class in `migrating.tmpl`
So whilst #19225 fixes one issue it caused another. We need to initialise the Git
module first.
Related #19225Fix#19162
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: 6543 <6543@obermui.de>
The git command by default adds a number of global arguments. These are not
helpful to be displayed in the process manager and so should be skipped for
default process descriptions.
Signed-off-by: Andrew Thornton <art27@cantab.net>
The RepoIndexerTest is failing with considerable frequency due to a race inherrent in
its design. This PR adjust this test to avoid the reliance on waiting for the populate
repo indexer to run and forcibly adds the repo to the queue. It then flushes the queue.
It may be worth separating out the tests somewhat by testing the Index function
directly away from the queue however, this forceful method should solve the current
problem.
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Set the default branch for repositories generated from templates
* Allows default branch to be set through the API for repos generated from templates
* Update swagger API template
* Only set default branch to the one from the template if not specified
* Use specified default branch if it exists while generating git commits
Fix#19082
Co-authored-by: John Olheiser <john.olheiser@gmail.com>
Co-authored-by: zeripath <art27@cantab.net>
* Add auto logging of goroutine pid label
This PR uses unsafe to export the hidden runtime_getProfLabel function from the
runtime package and then casts the result to a map[string]string.
We can then interrogate this map to get the pid label from the goroutine allowing
us to log it with any logging request.
Reference #19202
Signed-off-by: Andrew Thornton <art27@cantab.net>
This PR adds a middleware which sets a ContextUser (like GetUserByParams before) in a single place which can be used by other methods. For routes which represent a repo or org the respective middlewares set the field too.
Also fix a bug in modules/context/org.go during refactoring.
Unhelpfully Locations starting with `/\` will be converted by the
browser to `//` because ... well I do not fully understand. Certainly
the RFCs and MDN do not indicate that this would be expected. Providing
"compatibility" with the (mis)behaviour of a certain proprietary OS is
my suspicion. However, we clearly have to protect against this.
Therefore we should reject redirection locations that match the regular
expression: `^/[\\\\/]+`
Reference #9678
Signed-off-by: Andrew Thornton <art27@cantab.net>
Unfortunately #19169 causing a panic at startup in prod mode. This was hidden by dev
mode because the templates are compiled dynamically there. The issue is that DotEscape
is not in the original FuncMap at the time of compilation which causes a panic.
Ref #19169
Signed-off-by: Andrew Thornton <art27@cantab.net>
Redirect .wiki/* ui link to /wiki
fix#18590
Signed-off-by: a1012112796 <1012112796@qq.com>
Signed-off-by: Andrew Thornton <art27@cantab.net>
Co-authored-by: Andrew Thornton <art27@cantab.net>
Unfortunately many email readers will (helpfully) detect url or url-like names and
automatically create links to them, even in HTML emails. This is not ideal when
usernames can have dots in them.
This PR tries to prevent this behaviour by sticking ZWJ characters between dots and
also set the meta tag to prevent format detection.
Not every email template has been changed in this way - just the activation emails but
it may be that we should be setting the above meta tag in all of our emails too.
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Clean paths when looking in Storage
Ensure paths are clean for minio aswell as local storage.
Use url.Path not RequestURI/EscapedPath in storageHandler.
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Apply suggestions from code review
Co-authored-by: Lauris BH <lauris@nix.lv>
* Remove `db.DefaultContext` usage in routers, use `ctx` directly
* Use `ctx` directly if there is one, remove some `db.DefaultContext` in `services`
* Use ctx instead of db.DefaultContext for `cmd` and some `modules` packages
* fix incorrect context usage
* Clean up protected_branches when deleting user
fixes#19094
* Clean up protected_branches when deleting teams
* fix issue
Co-authored-by: Lauris BH <lauris@nix.lv>
Make SKIP_TLS_VERIFY apply to git data migrations too through adding the `-c http.sslVerify=false` option to the git clone command.
Fix#18998
Signed-off-by: Andrew Thornton <art27@cantab.net>
Storing the foreign identifier of an imported issue in the database is a prerequisite to implement idempotent migrations or mirror for issues. It is a baby step towards mirroring that introduces a new table.
At the moment when an issue is created by the Gitea uploader, it fails if the issue already exists. The Gitea uploader could be modified so that, instead of failing, it looks up the database to find an existing issue. And if it does it would update the issue instead of creating a new one. However this is not currently possible because an information is missing from the database: the foreign identifier that uniquely represents the issue being migrated is not persisted. With this change, the foreign identifier is stored in the database and the Gitea uploader will then be able to run a query to figure out if a given issue being imported already exists.
The implementation of mirroring for issues, pull requests, releases, etc. can be done in three steps:
1. Store an identifier for the element being mirrored (issue, pull request...) in the database (this is the purpose of these changes)
2. Modify the Gitea uploader to be able to update an existing repository with all it contains (issues, pull request...) instead of failing if it exists
3. Optimize the Gitea uploader to speed up the updates, when possible.
The second step creates code that does not yet exist to enable idempotent migrations with the Gitea uploader. When a migration is done for the first time, the behavior is not changed. But when a migration is done for a repository that already exists, this new code is used to update it.
The third step can use the code created in the second step to optimize and speed up migrations. For instance, when a migration is resumed, an issue that has an update time that is not more recent can be skipped and only newly created issues or updated ones will be updated. Another example of optimization could be that a webhook notifies Gitea when an issue is updated. The code triggered by the webhook would download only this issue and call the code created in the second step to update the issue, as if it was in the process of an idempotent migration.
The ForeignReferences table is added to contain local and foreign ID pairs relative to a given repository. It can later be used for pull requests and other artifacts that can be mirrored. Although the foreign id could be added as a single field in issues or pull requests, it would need to be added to all tables that represent something that can be mirrored. Creating a new table makes for a simpler and more generic design. The drawback is that it requires an extra lookup to obtain the information. However, this extra information is only required during migration or mirroring and does not impact the way Gitea currently works.
The foreign identifier of an issue or pull request is similar to the identifier of an external user, which is stored in reactions, issues, etc. as OriginalPosterID and so on. The representation of a user is however different and the ability of users to link their account to an external user at a later time is also a logic that is different from what is involved in mirroring or migrations. For these reasons, despite some commonalities, it is unclear at this time how the two tables (foreign reference and external user) could be merged together.
The ForeignID field is extracted from the issue migration context so that it can be dumped in files with dump-repo and later restored via restore-repo.
The GetAllComments downloader method is introduced to simplify the implementation and not overload the Context for the purpose of pagination. It also clarifies in which context the comments are paginated and in which context they are not.
The Context interface is no longer useful for the purpose of retrieving the LocalID and ForeignID since they are now both available from the PullRequest and Issue struct. The Reviewable and Commentable interfaces replace and serve the same purpose.
The Context data member of PullRequest and Issue becomes a DownloaderContext to clarify that its purpose is not to support in memory operations while the current downloader is acting but is not otherwise persisted. It is, for instance, used by the GitLab downloader to store the IsMergeRequest boolean and sort out issues.
---
[source](https://lab.forgefriends.org/forgefriends/forgefriends/-/merge_requests/36)
Signed-off-by: Loïc Dachary <loic@dachary.org>
Co-authored-by: Loïc Dachary <loic@dachary.org>
* use go1.18 to build gitea& update min go version to 1.17
* bump in a few more places
* add a few simple tests for isipprivate
* update go.mod
* update URL to https://go.dev/dl/
* golangci-lint
* attempt golangci-lint workaround
* change version
* bump fumpt version
* skip strings.title test
* go mod tidy
* update tests as some aren't private??
* update tests
Unfortunately #18642 does not work because a `*net.OpError` does not implement
the `Is` interface to make `errors.Is` work correctly - thus leading to the
irritating conclusion that a `*net.OpError` is not a `*net.OpError`.
Here we keep the `errors.Is` because presumably this will be fixed at
some point in the golang main source code but also we add a simply type
cast to also check.
Fix#18629
Signed-off-by: Andrew Thornton <art27@cantab.net>
Yet another issue has come up where the logging from SyncMirrors does not provide
enough context. This PR adds more context to these logging events.
Related #19038
Signed-off-by: Andrew Thornton <art27@cantab.net>
Add new feature to delete issues and pulls via API
Co-authored-by: fnetx <git@fralix.ovh>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: Gusted <williamzijl7@hotmail.com>
Co-authored-by: 6543 <6543@obermui.de>
- Add helper method to reduce redundancy
- Expand the scope from displaying days to years
- Reduce irrelevance by not displaying small units (hours, minutes, seconds) when bigger ones apply (years)
This PR adjusts the error returned when there is failure to lock the level db, and
permits a connections to the same leveldb where there is a different connection string.
Reference #18921
Reference #18917
Signed-off-by: Andrew Thornton <art27@cantab.net>
* Don't treat BOM escape sequence as hidden character.
- BOM sequence is a common non-harmfull escape sequence, it shouldn't be
shown as hidden character.
- Follows GitHub's behavior.
- Resolves#18837
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
The service worker causes a lot of issues with JS errors after instance
upgrades while not bringing any real performance gain over regular HTTP
caching.
Disable it by default for this reason. Maybe later we can remove it
completely, as I simply see no benefit in having it.
* Add tests for references with dashes
This commit adds tests for full URLs referencing repos names and user
names containing a dash.
* Extend regex to match URLs to repos/users with dashes
* logs: add the buffer logger to inspect logs during testing
Signed-off-by: Loïc Dachary <loic@dachary.org>
* migrations: add test for importing pull requests in gitea uploader
Signed-off-by: Loïc Dachary <loic@dachary.org>
* for each git.OpenRepositoryCtx, call Close
* Content is expected to return the content of the log
* test for errors before defer
Co-authored-by: Loïc Dachary <loic@dachary.org>
Co-authored-by: zeripath <art27@cantab.net>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
