mirror of
https://github.com/go-gitea/gitea.git
synced 2024-08-25 21:44:20 -04:00
This addresses https://github.com/go-gitea/gitea/issues/18352 It aims to improve performance (and resource use) of the `SyncReleasesWithTags` operation for pull-mirrors. For large repositories with many tags, `SyncReleasesWithTags` can be a costly operation (taking several minutes to complete). The reason is two-fold: 1. on sync, every upstream repo tag is compared (for changes) against existing local entries in the release table to ensure that they are up-to-date. 2. the procedure for getting _each tag_ involves a series of git operations ```bash git show-ref --tags -- v8.2.4477 git cat-file -t 29ab6ce9f36660cffaad3c8789e71162e5db5d2f git cat-file -p 29ab6ce9f36660cffaad3c8789e71162e5db5d2f git rev-list --count 29ab6ce9f36660cffaad3c8789e71162e5db5d2f ``` of which the `git rev-list --count` can be particularly heavy. This PR optimizes performance for pull-mirrors. We utilize the fact that a pull-mirror is always identical to its upstream and rebuild the entire release table on every sync and use a batch `git for-each-ref .. refs/tags` call to retrieve all tags in one go. For large mirror repos, with hundreds of annotated tags, this brings down the duration of the sync operation from several minutes to a few seconds. A few unscientific examples run on my local machine: - https://github.com/spring-projects/spring-boot (223 tags) - before: `0m28,673s` - after: `0m2,244s` - https://github.com/kubernetes/kubernetes (890 tags) - before: `8m00s` - after: `0m8,520s` - https://github.com/vim/vim (13954 tags) - before: `14m20,383s` - after: `0m35,467s` I added a `foreachref` package which contains a flexible way of specifying which reference fields are of interest (`git-for-each-ref(1)`) and to produce a parser for the expected output. These could be reused in other places where `for-each-ref` is used. I'll add unit tests for those if the overall PR looks promising. |
||
---|---|---|
.. | ||
activitypub | ||
analyze | ||
appstate | ||
auth | ||
avatar | ||
base | ||
cache | ||
charset | ||
container | ||
context | ||
convert | ||
csv | ||
doctor | ||
emoji | ||
eventsource | ||
generate | ||
git | ||
gitgraph | ||
graceful | ||
hcaptcha | ||
highlight | ||
hostmatcher | ||
httpcache | ||
httplib | ||
indexer | ||
json | ||
lfs | ||
log | ||
markup | ||
metrics | ||
migration | ||
nosql | ||
notification | ||
options | ||
packages | ||
password | ||
pprof | ||
private | ||
process | ||
proxy | ||
public | ||
queue | ||
recaptcha | ||
references | ||
repository | ||
secret | ||
session | ||
setting | ||
ssh | ||
storage | ||
structs | ||
svg | ||
sync | ||
templates | ||
test | ||
timeutil | ||
translation | ||
typesniffer | ||
updatechecker | ||
upload | ||
uri | ||
user | ||
util | ||
validation | ||
web |