Gitlab版本升级13.8.3到16.7.7
版本升级中会遇到很多问题，切记一定要备份，以便及时回退。Gitlab升级要跟着官方的升级顺序进行升级。本次生产项目准备升级docker版本和rpm安装方式升级，但docker版本遇到很多报错，且消耗大量时间。转为rpm安装方式升级。
一、升级前准备
1，13.8.3备份
[root@gitlab ~]# gitlab-rakegitlab:backup:create
此命令会根据/etc/gitlab/gitlab.rb中配置的路径备份，如果没有默认是在/var/opt/gitlab/backup/里。一定要注意磁盘空间大小。
2，gitlab官方升级顺序
官方提供的升级顺序提示网站
https://gitlab-com.gitlab.io/support/toolbox/upgrade-path/
3，国内gitlab源
国内的gitlab源选清华的，速度快。直接从官方仓库取太慢了。
https://mirrors.tuna.tsinghua.edu.cn/gitlab-ce/yum/el7/
4, 开启二次验证，以及临时停用二次验证
gitlab代码仓库，安全最重要。所以二次验证需要强制执行双重认证。
在开启双重认证后，如果是迁移gitlab服务，在恢复后就要使用到二次验证。此时我们可以通过命令暂时禁用双重认证，来检查gitlab迁移是否正确。
[root@gitlab ~]# gitlab-rails console
--------------------------------------------------------------------------------
Ruby: ruby 2.7.2p137 (2020-10-01 revision 5445e04352)[x86_64-linux]
GitLab: 13.8.3 (02529530a3d) FOSS
GitLab Shell: 13.15.0
PostgreSQL: 12.5
--------------------------------------------------------------------------------
irb(main):004:0> user = User.find_by(username: 'simon.li')
=> #
irb(main):005:0> user.disable_two_factor!
=> []
irb(main):006:0> exit
5, 13.8.3版本中存在使用legacy存储方式，gitlab14版本起，全面启用哈希存储，如果在升级过程中未进行转换，升级将会失败。
所以我们需要在13.x版本的最后一个版(13.11.12)，将传统存储转换为哈希存储
# 列出传统存储的项目以及附件
gitlab-rake gitlab:storage:list_legacy_projects
gitlab-rake gitlab:storage:list_legacy_attachments（附件可以不转换）
a, 存储库迁移
gitlab-rake gitlab:storage:migrate_to_hashed
b,转换单个仓库
Found 2 projects using Legacy Storage
root/devin (id: 29)
parker.lin/ua (id: 42)
c,进入console:
[root@gitlab ~]# gitlab-rails console
Ruby: ruby 2.7.4p223 (2020-03-30 revision 957bb7cb81) [x86_64-linux]
GitLab: 13.8.3 (6231cc1c609) FOSS
GitLab Shell: 14.34.0
PostgreSQL: 11.14
------------------------------------------------------------[ booted in 33.51s ]
Loading production environment (Rails 7.0.8.1)
irb(main):001:0> project = Project.find(29)
=> #<Project id:29 root/devin>>
irb(main):002:0> project.storage_version = 1
=> 1
irb(main):003:0> project.save!
=> true
在转换过程中遇到报错，需要进入数据库清空token
[root@gitlab ~]# gitlab-rails dbconsole
psql (11.14)
Type "help" for help.
gitlabhq_production=> UPDATE projects SET runners_token = null, runners_token_encrypted = null;
gitlabhq_production=> exit
# 全部迁移成功，以下命令查看所列出的项目总数与页面的理应一致
gitlab-rake gitlab:storage:hashed_projects
# 查看，全部迁移成功以下两条命令应该为 0
gitlab-rake gitlab:storage:legacy_projects
gitlab-rake gitlab:storage:legacy_attachments
6, gitlab备份和恢复命令
遇到备份文件特别大，在恢复过程中出现问题，可以选择只恢复数据库或某些文件。
gitlab 单独备份数据库：
gitlab-rake gitlab:backup:create SKIP=repositories,uploads,builds,artifacts,lfs,registry,pages
gitlab 单独导出仓库
gitlab-rake gitlab:backup:create SKIP=db,uploads,artifacts,lfs,registry,pages,builds
gitlab 直接恢复仓库目录：
参考网站：https://blog.csdn.net/qq_25760623/article/details/120786723
gitlab-rake gitlab:import:repos['/var/opt/gitlab/git-data/repositories']
gitlab 只恢复仓库：
gitlab-rake gitlab:backup:restore BACKUP=1493107454_2017_04_25_13.8.3 SKIP=db,uploads,artifacts,lfs,registry,pages
gitlab 直接恢复数据库：
sudo gitlab-rake gitlab:backup:restore BACKUP=1493107454_2017_04_25_13.8.3 SKIP=repositories,uploads,artifacts,lfs,registry,pages
7，gitlab迁移时runner报500错误解决方法
a. Reset CI/CD variables
[root@gitlab ~]# gitlab-rails dbconsole
SELECT * FROM public."ci_group_variables";
SELECT * FROM public."ci_variables";
DELETE FROM ci_group_variables;
DELETE FROM ci_variables;
b. Reset runner registration tokens
[root@gitlab ~]# gitlab-rails dbconsole
-- Clear project tokens
UPDATE projects SET runners_token = null, runners_token_encrypted = null;
-- Clear group tokens
UPDATE namespaces SET runners_token = null, runners_token_encrypted = null;
-- Clear instance tokens
UPDATE application_settings SET runners_registration_token_encrypted = null;
UPDATE application_settings SET encrypted_ci_jwt_signing_key = null;
-- Clear runner tokens
UPDATE ci_runners SET token = null, token_encrypted = null;
c. Reset pending pipeline jobs
[root@gitlab ~]# gitlab-rails dbconsole
-- Clear build tokens
UPDATE ci_builds SET token = null, token_encrypted = null;
d. Fix project integrations
[root@gitlab ~]# gitlab-rails dbconsole
-- truncate web_hooks table
TRUNCATE web_hooks CASCADE;
8，gitlab13.12.15 解决报500错：
ActionView::Template::Error (PG::UndefinedTable: ERROR: relation "services" does not exist
参考网站：https://stackoverflow.com/questions/71927588/gitlab-ce-500-error-after-upgrade-downgrade
[root@gitlab ~]# gitlab-psql
psql (12.6)
Type "help" for help.
gitlabhq_production=# ALTER TABLE Integrations RENAME TO Services;
ALTER TABLE
gitlabhq_production=#
9，gitlab 14.3.6 解决502报错,可以通过gitlab-ctl reconfigure 报错看到有关services表的相关信息。
先进入gitlab14.0.12，改表
[root@gitlab ~]# gitlab-psql
psql (12.6)
Type "help" for help.
gitlabhq_production=# ALTER TABLE services RENAME TO integrations;
ALTER TABLE
gitlabhq_production=#
10, gitlab 升级过程中需要对pg数据库升级，需要执行migrate, 不是必须要执行的，一般在gitlab-ctl reconfigure会自动migrate数据库。
[root@gitlab ~]# gitlab-rake db:migrate
[root@gitlab ~]# gitlab-rake db:migrate:status
[root@gitlab ~]# gitlab-ctl restart
11, 15.11.13升级遇到报错：CheckViolation: ERROR: check constraint "check_70f294ef54" of relation "oauth_access_tokens" is violated by some row
参考网站：https://blog.csdn.net/islandstar/article/details/130684138
[root@gitlab ~]# gitlab-psql
psql (13.14)
Type "help" for help.
gitlabhq_production=# UPDATE oauth_access_tokens SET expires_in = '7200' WHERE expires_in IS NULL;
12, gitlab 15.4.6 需要升级PostgreSQL数据库,需要将PostgreSQL12,升级到PostgreSQL13
[root@gitlab ~]# gitlab-ctl pg-upgrade
[root@gitlab ~]# gitlab-psql --version
psql (PostgreSQL) 13.14
13，gitlab安装完成后出现gitlab-ctl status的相关服务无法启动，可执行gitlab-ctl reconfigure有没有相关异常及报错，查找google相关报错及解决方法
[root@gitlab ~]# gitlab-ctl start
fail: alertmanager: runsv not running
fail: gitaly: runsv not running
出现服务无法启动,可执行以下命令
[root@gitlab ~]# systemctl start gitlab-runsvdir
[root@gitlab ~]# gitlab-ctl start
ok: run: alertmanager: (pid 4399) 7s
ok: run: gitaly: (pid 4407) 7s
ok: run: gitlab-exporter: (pid 4402) 7s
ok: run: gitlab-kas: (pid 4419) 7s
[root@gitlab ~]# systemctl status gitlab-runsvdir
● gitlab-runsvdir.service - GitLab Runit supervision process
Loaded: loaded (/usr/lib/systemd/system/gitlab-runsvdir.service; enabled; vendor preset: disabled)
Active: active (running) since Tue 2024-04-16 15:43:56 CST; 6s ago
Main PID: 3966 (runsvdir)
CGroup: /system.slice/gitlab-runsvdir.service
三、安装gitlab的rpm 包
a, 停止gitlab服务,然后执行安装命令
[root@gitlab ~]# gitlab-ctl stop
ok: down: alertmanager: 0s, normally up
ok: down: gitaly: 0s, normally up
ok: down: gitlab-exporter: 0s, normally up
ok: down: gitlab-kas: 0s, normally up
ok: down: gitlab-workhorse: 1s, normally up
ok: down: logrotate: 0s, normally up
ok: down: nginx: 1s, normally up
ok: down: postgres-exporter: 0s, normally up
ok: down: postgresql: 0s, normally up
ok: down: prometheus: 0s, normally up
ok: down: puma: 0s, normally up
ok: down: redis: 0s, normally up
ok: down: redis-exporter: 1s, normally up
ok: down: sidekiq: 0s, normally up
ok: down: sshd: 1s, normally up
b, 执行rpm安装
卸载，不要删除相关目录
[root@gitlab log]# rpm -evh `rpm -qa gitlab-ce`
Preparing... ################################# [100%]
Cleaning up / removing...
1:gitlab-ce-13.8.3-ce.0.el7 ################################# [100%]
[root@gitlab log]#
安装
[root@localhost ~]# rpm -ivh gitlab-ce-13.8.3-ce.0.el7.x86_64.rpm
warning: gitlab-ce-13.8.3-ce.0.el7.x86_64.rpm: Header V4 RSA/SHA1 Signature, key ID f27eab47: NOKEY
Preparing... ################################# [100%]
Updating / installing...
1:gitlab-ce-13.8.3-ce.0.el7 ################################# [100%]
It looks like GitLab has not been configured yet; skipping the upgrade script.
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
Thank you for installing GitLab!
GitLab should be available at http://git.mez100.com.cn
For a comprehensive list of configuration options please see the Omnibus GitLab readme
https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/README.md
Help us improve the installation experience, let us know how we did with a 1 minute survey:
https://gitlab.fra1.qualtrics.com/jfe/form/SV_6kVqZANThUQ1bZb?installation=omnibus&release=13-8
[root@localhost ~]#
在安装完成后，需要执行gitlab-ctl reconfigure出现以下提示就表示编译完成
gitlab Reconfigured!
[root@localhost ~]# echo $?
0
[root@localhost ~]#
正常启动gitlab后所运行的gitlab服务及相关端口
[root@localhost ~]# ss -tnl
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 128 *:22 *:*
LISTEN 0 128 [::]:22 [::]:*
[root@localhost ~]# gitlab-ctl start
ok: run: alertmanager: (pid 7208) 0s
ok: run: gitaly: (pid 7218) 0s
ok: run: gitlab-exporter: (pid 7236) 1s
ok: run: gitlab-kas: (pid 7311) 0s
ok: run: gitlab-workhorse: (pid 7319) 1s
ok: run: logrotate: (pid 7331) 0s
ok: run: nginx: (pid 7337) 1s
ok: run: node-exporter: (pid 7354) 0s
ok: run: postgres-exporter: (pid 7360) 0s
ok: run: postgresql: (pid 7374) 1s
ok: run: prometheus: (pid 7385) 0s
ok: run: puma: (pid 7396) 1s
ok: run: redis: (pid 7410) 0s
ok: run: redis-exporter: (pid 7417) 1s
ok: run: sidekiq: (pid 7435) 0s