1
0
mirror of https://github.com/go-gitea/gitea.git synced 2024-12-04 14:46:57 -05:00
gitea/modules/setting
Jason Song 33cc5837a6
Support compression for Actions logs (#31761)
Support compression for Actions logs to save storage space and
bandwidth. Inspired by
https://github.com/go-gitea/gitea/issues/24256#issuecomment-1521153015

The biggest challenge is that the compression format should support
[seekable](https://github.com/facebook/zstd/blob/dev/contrib/seekable_format/zstd_seekable_compression_format.md).
So when users are viewing a part of the log lines, Gitea doesn't need to
download the whole compressed file and decompress it.

That means gzip cannot help here. And I did research, there aren't too
many choices, like bgzip and xz, but I think zstd is the most popular
one. It has an implementation in Golang with
[zstd](https://github.com/klauspost/compress/tree/master/zstd) and
[zstd-seekable-format-go](https://github.com/SaveTheRbtz/zstd-seekable-format-go),
and what is better is that it has good compatibility: a seekable format
zstd file can be read by a regular zstd reader.

This PR introduces a new package `zstd` to combine and wrap the two
packages, to provide a unified and easy-to-use API.

And a new setting `LOG_COMPRESSION` is added to the config, although I
don't see any reason why not to use compression, I think's it's a good
idea to keep the default with `none` to be consistent with old versions.

`LOG_COMPRESSION` takes effect for only new log files, it adds `.zst` as
an extension to the file name, so Gitea can determine if it needs
decompression according to the file name when reading. Old files will
keep the format since it's not worth converting them, as they will be
cleared after #31735.

<img width="541" alt="image"
src="https://github.com/user-attachments/assets/e9598764-a4e0-4b68-8c2b-f769265183c9">
2024-08-09 10:10:30 +08:00
..
config Customizable "Open with" applications for repository clone (#29320) 2024-02-24 13:12:17 +00:00
actions_test.go
actions.go Support compression for Actions logs (#31761) 2024-08-09 10:10:30 +08:00
admin.go Allow disabling authentication related user features (#31535) 2024-07-09 17:36:31 +00:00
api.go
asset_dynamic.go
asset_static.go
attachment_test.go
attachment.go Update allowed attachment types (#29688) 2024-03-09 12:41:32 +00:00
cache.go Always enable caches (#28527) 2023-12-19 09:29:05 +00:00
camo.go
config_env_test.go Fix environment-to-ini inherited key bug (#27543) 2023-10-10 01:10:37 +08:00
config_env.go Enable unparam linter (#31277) 2024-06-11 18:47:45 +00:00
config_provider_test.go Fix INI parsing for value with trailing slash (#26995) 2023-09-10 16:15:51 +00:00
config_provider.go Initial support for colorblindness-friendly themes (#30625) 2024-04-24 00:18:41 +08:00
config.go Customizable "Open with" applications for repository clone (#29320) 2024-02-24 13:12:17 +00:00
cors.go Refactor CORS handler (#28587) 2023-12-25 20:13:18 +08:00
cron_test.go
cron.go Replace interface{} with any (#25686) 2023-07-04 18:36:08 +00:00
database_sqlite.go
database_test.go Fix incorrect PostgreSQL connection string for Unix sockets (#28865) 2024-01-20 16:04:47 +00:00
database.go Add slow SQL query warning (#27545) 2024-02-23 00:57:24 +00:00
federation.go
git_test.go
git.go
glob.go Improve grep search (#30843) 2024-05-03 09:13:48 +00:00
global.go Fix duplicate sub-path for avatars (#31365) 2024-06-15 11:43:57 +08:00
highlight.go
i18n.go
incoming_email.go Resolve lint for unused parameter and unnecessary type arguments (#30750) 2024-04-29 08:47:56 +00:00
indexer_test.go
indexer.go Improve grep search (#30843) 2024-05-03 09:13:48 +00:00
lfs_test.go Display deprecated warning in admin panel pages as well as in the log file (#26094) 2023-07-26 03:53:37 +00:00
lfs.go Use general token signing secret (#29205) 2024-02-18 17:39:04 +00:00
log_test.go Replace interface{} with any (#25686) 2023-07-04 18:36:08 +00:00
log.go Refactor "dump" sub-command (#30240) 2024-04-03 02:16:46 +00:00
mailer_test.go
mailer.go Add option to change mail from user display name (#31528) 2024-07-14 23:27:00 +02:00
markup.go
metrics.go
migrations.go
mime_type_map.go
mirror.go
oauth2_test.go Always load or generate oauth2 jwt secret (#30942) 2024-05-14 14:21:38 +00:00
oauth2.go Always load or generate oauth2 jwt secret (#30942) 2024-05-14 14:21:38 +00:00
other.go Add [other].SHOW_FOOTER_POWERED_BY setting to hide Powered by (#30253) 2024-04-03 16:01:50 +00:00
packages_test.go
packages.go Add signature support for the RPM module (#27069) 2024-08-06 09:03:33 -04:00
path_test.go
path.go Do not allow different storage configurations to point to the same directory (#30169) 2024-03-31 03:03:24 +00:00
picture.go
project.go
proxy.go
queue.go Increase queue length (#27555) 2023-10-10 18:47:49 +08:00
repository_archive_test.go
repository_archive.go
repository.go Refactor startup deprecation messages (#30305) 2024-04-07 01:11:25 +00:00
security.go Remember login for a month by default (#30150) 2024-03-28 03:13:42 +00:00
server.go Fix duplicate sub-path for avatars (#31365) 2024-06-15 11:43:57 +08:00
service_test.go Fix allowed user types setting problem (#26200) 2023-07-28 12:15:39 -04:00
service.go Add reverseproxy auth for API back with default disabled (#26703) 2023-09-07 08:31:46 +00:00
session.go Refactor startup deprecation messages (#30305) 2024-04-07 01:11:25 +00:00
setting_test.go
setting.go Fix duplicate sub-path for avatars (#31365) 2024-06-15 11:43:57 +08:00
ssh.go Expanded minimum RSA Keylength to 3072 (#26604) 2023-08-28 00:53:16 +00:00
storage_test.go Azure blob storage support (#30995) 2024-05-30 07:33:50 +00:00
storage.go Enable unparam linter (#31277) 2024-06-11 18:47:45 +00:00
task.go
time.go Enable more revive linter rules (#30608) 2024-04-22 11:48:42 +00:00
ui.go Add MAX_ROWS option for CSV rendering (#30268) 2024-06-06 16:06:59 +08:00
webhook.go