1
0
mirror of https://github.com/go-gitea/gitea.git synced 2024-10-29 08:17:31 -04:00
Commit Graph

30 Commits

Author SHA1 Message Date
KN4CK3R
c03e488e14
Add LFS Migration and Mirror (#14726)
* Implemented LFS client.

* Implemented scanning for pointer files.

* Implemented downloading of lfs files.

* Moved model-dependent code into services.

* Removed models dependency. Added TryReadPointerFromBuffer.

* Migrated code from service to module.

* Centralised storage creation.

* Removed dependency from models.

* Moved ContentStore into modules.

* Share structs between server and client.

* Moved method to services.

* Implemented lfs download on clone.

* Implemented LFS sync on clone and mirror update.

* Added form fields.

* Updated templates.

* Fixed condition.

* Use alternate endpoint.

* Added missing methods.

* Fixed typo and make linter happy.

* Detached pointer parser from gogit dependency.

* Fixed TestGetLFSRange test.

* Added context to support cancellation.

* Use ReadFull to probably read more data.

* Removed duplicated code from models.

* Moved scan implementation into pointer_scanner_nogogit.

* Changed method name.

* Added comments.

* Added more/specific log/error messages.

* Embedded lfs.Pointer into models.LFSMetaObject.

* Moved code from models to module.

* Moved code from models to module.

* Moved code from models to module.

* Reduced pointer usage.

* Embedded type.

* Use promoted fields.

* Fixed unexpected eof.

* Added unit tests.

* Implemented migration of local file paths.

* Show an error on invalid LFS endpoints.

* Hide settings if not used.

* Added LFS info to mirror struct.

* Fixed comment.

* Check LFS endpoint.

* Manage LFS settings from mirror page.

* Fixed selector.

* Adjusted selector.

* Added more tests.

* Added local filesystem migration test.

* Fixed typo.

* Reset settings.

* Added special windows path handling.

* Added unit test for HTTPClient.

* Added unit test for BasicTransferAdapter.

* Moved into util package.

* Test if LFS endpoint is allowed.

* Added support for git://

* Just use a static placeholder as the displayed url may be invalid.

* Reverted to original code.

* Added "Advanced Settings".

* Updated wording.

* Added discovery info link.

* Implemented suggestion.

* Fixed missing format parameter.

* Added Pointer.IsValid().

* Always remove model on error.

* Added suggestions.

* Use channel instead of array.

* Update routers/repo/migrate.go

* fmt

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: zeripath <art27@cantab.net>
2021-04-08 18:25:57 -04:00
6543
81c833d92d
Add support to migrate from gogs (#14342)
Add support to migrate gogs:

  *  issues
  *  comments
  *  labels
  *  milestones
  *  wiki


Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
Co-authored-by: Andrew Thornton <art27@cantab.net>
2021-01-21 20:33:58 +01:00
Paul Barton
7576e37a65
Add MirrorInterval to the API (#14163)
* Added MirrorInterval to the API

* Remove MirrorInterval from CreateRepository

* Removed Duplicate UpdateMirror Function

* Updated Error Logging

* Update Log Message for is not Mirror

Co-authored-by: 6543 <6543@obermui.de>

* Delete Debug Statement that snuck in

Co-authored-by: zeripath <art27@cantab.net>

* Add Check for If Interval is too small

* Output to API Call

* Add Error Object when time is Less than Min Interval

* Frequency Error Message

Co-authored-by: zeripath <art27@cantab.net>

* Allow Zero Mirror Interval

Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: zeripath <art27@cantab.net>
2021-01-02 18:47:47 -05:00
Lunny Xiao
dd08853b10
Dump github/gitlab/gitea repository data to a local directory and restore to gitea (#12244)
* Dump github/gitlab repository data to a local directory

* Fix lint

* Adjust directory structure

* Allow migration special units

* Allow migration ignore release assets

* Fix lint

* Add restore repository

* stage the changes

* Merge

* Fix lint

* Update the interface

* Add some restore methods

* Finish restore

* Add comments

* Fix restore

* Add a token flag

* Fix bug

* Fix test

* Fix test

* Fix bug

* Fix bug

* Fix lint

* Fix restore

* refactor downloader

* fmt

* Fix bug isEnd detection on getIssues

* Refactor maxPerPage

* Remove unused codes

* Remove unused codes

* Fix bug

* Fix restore

* Fix dump

* Uploader should not depend downloader

* use release attachment name but not id

* Fix restore bug

* Fix lint

* Fix restore bug

* Add a method of DownloadFunc for base.Release to make uploader not depend on downloader

* fix Release yml marshal

* Fix trace information

* Fix bug when dump & restore

* Save relative path on yml file

* Fix bug

* Use relative path

* Update docs

* Use git service string but not int

* Recognize clone addr to service type
2020-12-27 11:34:19 +08:00
zeripath
825efa2605
[API] Add Ref to Issue (#13946)
* Add Ref to api.Issue

This PR adds Ref to api.Issue and adds the backend parts to the migrator.

Fix #13918

Signed-off-by: Andrew Thornton <art27@cantab.net>
2020-12-13 11:34:11 +00:00
6543
49b1948cb1
Gitea 2 Gitea migration (#12657)
* first draft

* update gitea sdk to 9e280adb4da

* adapt feat of updated sdk

* releases now works

* break the Reactions loop

* use convertGiteaLabel

* fix endless loop because paggination is not supported there !!!

* rename gitea local uploader files

* pagination can bite you in the ass

* Version Checks

* lint

* docs

* rename gitea sdk import to miss future conficts

* go-swagger: dont scan the sdk structs

* make sure gitea can shutdown gracefully

* make GetPullRequests and GetIssues similar

* rm useles

* Add Test: started ...

* ... add tests ...

* Add tests and Fixing things

* Workaround missing SHA

* Adapt: Ensure that all migration requests are cancellable
(714ab71ddc)

* LINT: fix misspells in test set

* adapt ListMergeRequestAwardEmoji

* update sdk

* Return error when creating giteadownloader failed

* update sdk

* adapt new sdk

* adopt new features

* check version before err

* adapt: 'migrate service type switch page'

* optimize

* Fix DefaultBranch

* impruve

* handle subPath

* fix test

* Fix ReviewCommentPosition

* test GetReviews

* add DefaultBranch int test set

* rm unused

* Update SDK to v0.13.0

* addopt sdk changes

* found better link

* format template

* Update Docs

* Update Gitea SDK (v0.13.1)
2020-10-14 07:06:00 +03:00
6543
6c61f498ea
On Migration respect old DefaultBranch (#12843)
* On Migration respect old DefaultBranch

* add DefaultBranch int test set

Co-authored-by: zeripath <art27@cantab.net>
2020-09-15 22:37:44 +08:00
6543
fd60ebfe14
[API] Migration: Change ServiceType String (#12672)
* use different structs for MigrateRepoOptions on UI and API

* Fix TokenAuth and rename UID to an understandable Name

* fix swagger doc

* simplify & mk redable

* R E F A C T O R:
migration has now internal 3 structs to store its options:
 * the Options for WebUI: modules/auth/repo_form.go
 * the Options for API: modules/structs/repo.go
 * the option struct with after validation for internal prossessing: modules/migrations/base/options.go

* Copyright Header

* Deprecate UID - add RepoOwner

* adopt repo.go -> migrate.go

* add comment about each struct purpose

* lint
2020-09-10 23:29:19 +01:00
zeripath
714ab71ddc
Ensure that all migration requests are cancellable (#12669)
* Ensure that all migration requests are cancellable

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Use WithContext as RequestWithContext is go 1.14

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: techknowlogick <techknowlogick@gitea.io>
2020-09-02 13:49:25 -04:00
John Olheiser
211321fb93
Git migration UX (#12619)
* Initial work

Signed-off-by: jolheiser <john.olheiser@gmail.com>

* Implementation

Signed-off-by: jolheiser <john.olheiser@gmail.com>

* Fix gitlab and token cloning

Signed-off-by: jolheiser <john.olheiser@gmail.com>

* Imports and JS

Signed-off-by: jolheiser <john.olheiser@gmail.com>

* Fix test

Signed-off-by: jolheiser <john.olheiser@gmail.com>

* Linting

Signed-off-by: jolheiser <john.olheiser@gmail.com>

* Generate swagger

Signed-off-by: jolheiser <john.olheiser@gmail.com>

* Move mirror toggle and rename options

Signed-off-by: jolheiser <john.olheiser@gmail.com>

Co-authored-by: Lauris BH <lauris@nix.lv>
Co-authored-by: Lunny Xiao <xiaolunwen@gmail.com>
2020-08-28 09:36:37 +08:00
6543
5bfb9bc2b6
When migrating from Gitlab map Approvals to approving Reviews (#11147)
When migrating from Gitlab map Gitlab Approvals to approving Reviews

Co-Authored-By: zeripath <art27@cantab.net>
2020-04-20 13:30:46 +01:00
6543
e37e217602
[Fix] RetryDownloader (add GetReviews()) (#11093)
* add GetReviews() to RetryDownloader
2020-04-20 03:23:28 +01:00
Lunny Xiao
f6067a8465 Migrate reviews when migrating repository from github (#9463)
* fix typo

* Migrate reviews when migrating repository from github

* fix lint

* Added test and migration when external user login

* fix test

* fix commented state

* Some improvements

* fix bug when get pull request and ref original author on code comments

* Fix migrated line; Added comment for review

* Don't load all pull requests attributes

* Fix typo

* wrong change copy head

* fix tests

* fix reactions

* Fix test

* fix fmt

* fix review comment reactions
2020-01-23 19:28:15 +02:00
Lunny Xiao
2b3e931cde Migrate reactions when migrating repository from github (#9599)
* Migrate reactions when migrating repository from github

* fix missed sleep

* fix tests

* update reactions when external user binding

* Fix test

* fix tests

* change the copy head

* fix test

* fix migrator add/delete reaction
2020-01-15 12:14:07 +01:00
Lunny Xiao
7f869c061e Fix missing updated time on migrated issues and comments (#9744)
* Fix missing updated time on migrated issues and comments

* Fix testing and missing updated on migrating pullrequest

Co-authored-by: Antoine GIRARD <sapk@users.noreply.github.com>
2020-01-14 10:29:22 +00:00
Lunny Xiao
ffc904b1e0 Sleep longer if request speed is over github limitation (#9335)
* Sleep longer if request speed is over github limitation

* improve code

* remove unused code

* fix lint

* Use github's rate limit remain value to determine how long to sleep

* Save reset time when finished github api request

* fix bug

* fix lint

* Add context.Context for sleep

* fix test

* improve code

* fix bug and lint

* fix import order
2019-12-16 23:16:54 -05:00
John Olheiser
382936a668 Add SyncTags to uploader interface (#9326)
* Add sync tags to interface

Signed-off-by: jolheiser <john.olheiser@gmail.com>

* Fix revive

Signed-off-by: jolheiser <john.olheiser@gmail.com>
2019-12-12 08:20:11 +08:00
John Olheiser
3bf5a07537 Only sync tags after all batches (#9319)
Signed-off-by: jolheiser <john.olheiser@gmail.com>
2019-12-11 14:09:06 +08:00
Lunny Xiao
1b7182e5ec
Add retry for migration http/https requests (#9019)
* Add retry for migration http/https requests

* give the more suitable name for retry configuraion items

* fix docs and lint

* Only use retryDownloader when setting > 1
2019-11-16 16:30:06 +08:00
zeripath
722a7c902d
Add Close() method to gogitRepository (#8901)
In investigating #7947 it has become clear that the storage component of go-git repositories needs closing.

This PR adds this Close function and adds the Close functions as necessary.

In TransferOwnership the ctx.Repo.GitRepo is closed if it is open to help prevent the risk of multiple open files.

Fixes #7947
2019-11-13 07:01:19 +00:00
Lunny Xiao
e3e44a59d0 Update migrated repositories' issues/comments/prs poster id if user has a github external user saved (#7751)
* update migrated issues/comments when login as github

* add get userid when migrating or login with github oauth2

* fix lint

* add migrations for repository service type

* fix build

* remove unnecessary dependencies on migrations

* add cron task to update migrations poster ids and fix posterid when migrating

* fix lint

* fix lint

* improve code

* fix lint

* improve code

* replace releases publish id to actual author id

* fix import

* fix bug

* fix lint

* fix rawdata definition

* fix some bugs

* fix error message
2019-10-14 09:10:42 +03:00
Lunny Xiao
f2a3abc683
Move migrating repository from frontend to backend (#6200)
* move migrating to backend

* add loading image when migrating and fix tests

* fix format

* fix lint

* add redis task queue support and improve docs

* add redis vendor

* fix vet

* add database migrations and fix app.ini sample

* add comments for task section on app.ini.sample

* Update models/migrations/v84.go

Co-Authored-By: lunny <xiaolunwen@gmail.com>

* Update models/repo.go

Co-Authored-By: lunny <xiaolunwen@gmail.com>

* move migrating to backend

* add loading image when migrating and fix tests

* fix fmt

* add redis task queue support and improve docs

* fix fixtures

* fix fixtures

* fix duplicate function on index.js

* fix tests

* rename repository statuses

* check if repository is being create when SSH request

* fix lint

* fix template

* some improvements

* fix template

* unified migrate options

* fix lint

* fix loading page

* refactor

* When gitea restart, don't restart the running tasks because we may have servel gitea instances, that may break the migration

* fix js

* Update models/repo.go

Co-Authored-By: guillep2k <18600385+guillep2k@users.noreply.github.com>

* Update docs/content/doc/advanced/config-cheat-sheet.en-us.md

Co-Authored-By: guillep2k <18600385+guillep2k@users.noreply.github.com>

* fix tests

* rename ErrTaskIsNotExist to ErrTaskDoesNotExist

* delete release after add one on tests to make it run happy

* fix tests

* fix tests

* improve codes

* fix lint

* fix lint

* fix migrations
2019-10-13 21:23:14 +08:00
Lunny Xiao
bd41a04a56 Add extra user information when migrating release (#8331)
* add extra user information when migrating release

* add migrations

* fix tests
2019-10-05 14:09:27 +03:00
Antoine GIRARD
46d6b92ad0 Import topics during migration (#7851)
* add GetTopics interface

* CreateTopics

* remove un-needed comment
2019-08-14 14:16:12 +08:00
mrsdizzie
1f1ecda541 Display original author and URL information when showing migrated issues/comments (#7352)
* Store original author info for migrated issues and comments

Keep original author name for displaying in Gitea interface and also
store original author user ID for potential future use in linking
accounts from old location.

* Add original_url for repo

Store the original URL for a migrated repo

Clean up migrations/tests

* fix migration

* fix golangci-lint

* make 'make revive' happy also

* Modify templates to use OriginalAuthor if set

Use the original author name in templates if it is set rather than the
user who migrated/currently owns the issues

* formatting fixes

* make generate-swagger

* Use default avatar for imported comments

* Remove no longer used IgnoreIssueAuthor option

* Add OriginalAuthorID to swagger also
2019-07-08 10:14:12 +08:00
Lunny Xiao
337d6915ff Detect migrating batch size (#7353)
* Make migrating batch size as configurable

* detect different table batch insert size and remove config item

* remove unrelated changes
2019-07-06 15:24:50 -04:00
Lunny Xiao
e728b55812 Fix bug conflict between SyncReleasesWithTags and InsertReleases (#7337)
* fix bug conflict between SyncReleasesWithTags and InsertReleases

* fix tests

* fix fmt
2019-07-01 22:17:16 +01:00
Lunny Xiao
462284e2f5
Use batch insert on migrating repository to make the process faster (#7050)
* Use batch insert on migrating repository to make the process faster

* fix lint

* fix tests

* fix comments
2019-06-29 21:38:22 +08:00
Lunny Xiao
7d12ec2abd improve github downloader on migrations (#7049)
* improve github downloader on migrations

* fix tests

* fix  uppercase function parameters
2019-05-30 16:26:57 -04:00
Lunny Xiao
08069dc465
Improve migrations to support migrating milestones/labels/issues/comments/pullrequests (#6290)
* add migrations

* fix package dependency

* fix lints

* implements migrations except pull requests

* add releases

* migrating releases

* fix bug

* fix lint

* fix migrate releases

* fix tests

* add rollback

* pull request migtations

* fix import

* fix go module vendor

* add tests for upload to gitea

* more migrate options

* fix swagger-check

* fix misspell

* add options on migration UI

* fix log error

* improve UI options on migrating

* add support for username password when migrating from github

* fix tests

* remove comments and fix migrate limitation

* improve error handles

* migrate API will also support migrate milestones/labels/issues/pulls/releases

* fix tests and remove unused codes

* add DownloaderFactory and docs about how to create a new Downloader

* fix misspell

* fix migration docs

* Add hints about migrate options on migration page

* fix tests
2019-05-07 09:12:51 +08:00