Draft: Show columns that should be dropped via rake db:obsolete_ignored_columns
What does this MR do and why?
This MR enhances the existing rake task bin/rake db:obsolete_ignored_columns to list database columns which are safe to be dropped according to https://docs.gitlab.com/ee/development/avoiding_downtime_in_migrations.html#dropping-columns.
It also fixes all RuboCop offenses and refactors the code and specs.
Contributes to #382533.
Caveats
We had to side-step the following roadblocks:
- Some models are ignoring columns without using
IgnorableColumnsconcern- See !120880 (merged)
- If Geo database is not setup we have to exclude models which inherit from
Geo::TrackingBaseto avoid `SecondaryNotConfigured failures
Screenshots or screen recordings
The huge difference between Before and After stems from:
- Showing columns which should be dropped
- Inspecting more models by using
Rails.application.eager_load!
Before
$ bin/rake db:obsolete_ignored_columns
The following `ignored_columns` definitions are obsolete and can be removed:
ApplicationSetting:
- user_email_lookup_limit Remove after 2022-04-18 with 15.0
- send_user_confirmation_email Remove after 2022-12-18 with 15.8
- database_apdex_settings Remove after 2023-08-22 with 16.4
- relay_state_domain_allowlist Remove after 2023-10-22 with 16.6
- in_product_marketing_emails_enabled Remove after 2023-10-22 with 16.6
- encrypted_product_analytics_clickhouse_connection_string Remove after 2023-09-22 with 16.5
- encrypted_product_analytics_clickhouse_connection_string_iv Remove after 2023-09-22 with 16.5
- encrypted_jitsu_administrator_password Remove after 2023-09-22 with 16.5
- encrypted_jitsu_administrator_password_iv Remove after 2023-09-22 with 16.5
- jitsu_host Remove after 2023-09-22 with 16.5
- jitsu_project_xid Remove after 2023-09-22 with 16.5
- jitsu_administrator_email Remove after 2023-09-22 with 16.5
GeoNodeStatus:
- wikis_checksum_failed_count Remove after 2023-09-22 with 16.5
- wikis_checksum_mismatch_count Remove after 2023-09-22 with 16.5
- wikis_checksummed_count Remove after 2023-09-22 with 16.5
- wikis_failed_count Remove after 2023-09-22 with 16.5
- wikis_retrying_verification_count Remove after 2023-09-22 with 16.5
- wikis_synced_count Remove after 2023-09-22 with 16.5
- wikis_verification_failed_count Remove after 2023-09-22 with 16.5
- wikis_verified_count Remove after 2023-09-22 with 16.5
- design_repositories_count Remove after 2023-09-22 with 16.5
- design_repositories_synced_count Remove after 2023-09-22 with 16.5
- design_repositories_failed_count Remove after 2023-09-22 with 16.5
- design_repositories_registry_count Remove after 2023-09-22 with 16.5
Ml::Candidate:
- iid Remove after 2023-05-01 with 16.0
User:
- email_opted_in Remove after 2023-10-22 with 16.6
- email_opted_in_ip Remove after 2023-10-22 with 16.6
- email_opted_in_source_id Remove after 2023-10-22 with 16.6
- email_opted_in_at Remove after 2023-10-22 with 16.6
UserPreference:
- experience_level Remove after 2021-03-22 with 14.10
WARNING: Removing columns is tricky because running GitLab processes may still be using the columns.
See also https://docs.gitlab.com/ee/development/avoiding_downtime_in_migrations.html#dropping-columnsAfter
WARNING: Removing columns is tricky because running GitLab processes may still be using the columns.
See also https://docs.gitlab.com/ee/development/avoiding_downtime_in_migrations.html#dropping-columns
Analytics::DevopsAdoption::EnabledNamespace:
- last_recorded_at Drop column after 2021-06-22
Analytics::DevopsAdoption::Snapshot:
- segment_id Remove `ignore_columns` after 2021-07-22 with 14.2
ApplicationSetting:
- elasticsearch_replicas Drop column after 2021-08-22
- elasticsearch_shards Drop column after 2021-08-22
- static_objects_external_storage_auth_token Drop column after 2022-02-22
- user_email_lookup_limit Remove `ignore_columns` after 2022-04-18 with 15.0
- send_user_confirmation_email Remove `ignore_columns` after 2022-12-18 with 15.8
- web_ide_clientside_preview_enabled Drop column after 2023-03-22
- instance_administration_project_id Drop column after 2023-05-22
- instance_administrators_group_id Drop column after 2023-05-22
- database_apdex_settings Remove `ignore_columns` after 2023-08-22 with 16.4
- encrypted_jitsu_administrator_password Remove `ignore_columns` after 2023-09-22 with 16.5
- encrypted_jitsu_administrator_password_iv Remove `ignore_columns` after 2023-09-22 with 16.5
- encrypted_product_analytics_clickhouse_connection_string Remove `ignore_columns` after 2023-09-22 with 16.5
- encrypted_product_analytics_clickhouse_connection_string_iv Remove `ignore_columns` after 2023-09-22 with 16.5
- jitsu_administrator_email Remove `ignore_columns` after 2023-09-22 with 16.5
- jitsu_host Remove `ignore_columns` after 2023-09-22 with 16.5
- jitsu_project_xid Remove `ignore_columns` after 2023-09-22 with 16.5
- in_product_marketing_emails_enabled Remove `ignore_columns` after 2023-10-22 with 16.6
- relay_state_domain_allowlist Remove `ignore_columns` after 2023-10-22 with 16.6
Ci::Minutes::NamespaceMonthlyUsage:
- shared_runners_duration_convert_to_bigint Remove `ignore_columns` after 2023-09-22 with 16.5
Ci::Minutes::ProjectMonthlyUsage:
- shared_runners_duration_convert_to_bigint Remove `ignore_columns` after 2023-09-22 with 16.5
Ci::Pipeline:
- id_convert_to_bigint Drop column after 2023-07-22
- auto_canceled_by_id_convert_to_bigint Drop column after 2023-09-22
Ci::PipelineChatData:
- pipeline_id_convert_to_bigint Remove `ignore_columns` after 2023-10-22 with 16.5
Ci::PipelineMessage:
- pipeline_id_convert_to_bigint Drop column after 2023-08-22
Ci::PipelineVariable:
- pipeline_id_convert_to_bigint Drop column after 2023-09-22
Ci::Sources::Pipeline:
- pipeline_id_convert_to_bigint Drop column after 2023-09-22
- source_pipeline_id_convert_to_bigint Drop column after 2023-09-22
Ci::Stage:
- pipeline_id_convert_to_bigint Drop column after 2023-09-22
Ci::Trigger:
- ref Remove `ignore_columns` after 2023-05-22 with 16.1
Dast::PreScanVerificationStep:
- name Drop column after 2023-04-17
GeoNodeStatus:
- design_repositories_count Remove `ignore_columns` after 2023-09-22 with 16.5
- design_repositories_failed_count Remove `ignore_columns` after 2023-09-22 with 16.5
- design_repositories_registry_count Remove `ignore_columns` after 2023-09-22 with 16.5
- design_repositories_synced_count Remove `ignore_columns` after 2023-09-22 with 16.5
- repositories_checksum_failed_count Drop column after 2023-09-22
- repositories_checksum_mismatch_count Drop column after 2023-09-22
- repositories_checksummed_count Drop column after 2023-09-22
- repositories_failed_count Drop column after 2023-09-22
- repositories_retrying_verification_count Drop column after 2023-09-22
- repositories_synced_count Drop column after 2023-09-22
- repositories_verification_failed_count Drop column after 2023-09-22
- repositories_verified_count Drop column after 2023-09-22
- wikis_checksum_failed_count Remove `ignore_columns` after 2023-09-22 with 16.5
- wikis_checksum_mismatch_count Remove `ignore_columns` after 2023-09-22 with 16.5
- wikis_checksummed_count Remove `ignore_columns` after 2023-09-22 with 16.5
- wikis_failed_count Remove `ignore_columns` after 2023-09-22 with 16.5
- wikis_retrying_verification_count Remove `ignore_columns` after 2023-09-22 with 16.5
- wikis_synced_count Remove `ignore_columns` after 2023-09-22 with 16.5
- wikis_verification_failed_count Remove `ignore_columns` after 2023-09-22 with 16.5
- wikis_verified_count Remove `ignore_columns` after 2023-09-22 with 16.5
- container_repositories_count Drop column after 2023-10-22
- container_repositories_failed_count Drop column after 2023-10-22
- container_repositories_registry_count Drop column after 2023-10-22
- container_repositories_synced_count Drop column after 2023-10-22
- job_artifacts_count Drop column after 2023-10-22
- job_artifacts_failed_count Drop column after 2023-10-22
- job_artifacts_synced_count Drop column after 2023-10-22
- job_artifacts_synced_missing_on_primary_count Drop column after 2023-10-22
- lfs_objects_count Drop column after 2023-10-22
- lfs_objects_failed_count Drop column after 2023-10-22
- lfs_objects_synced_count Drop column after 2023-10-22
- lfs_objects_synced_missing_on_primary_count Drop column after 2023-10-22
Gitlab::BackgroundMigration::BackfillIntegrationsEnableSslVerification::Integration:
- template Remove `ignore_columns` after 2022-04-22 with 15.0
- type Remove `ignore_columns` after 2022-04-22 with 15.0
- properties Remove `ignore_columns` after 2022-05-22 with 15.1
Group:
- unlock_membership_to_ldap Drop column after 2023-10-16
MergeRequestDiffCommit:
- author_email Remove `ignore_columns` after 2021-11-22 with 14.6
- author_name Remove `ignore_columns` after 2021-11-22 with 14.6
- committer_email Remove `ignore_columns` after 2021-11-22 with 14.6
- committer_name Remove `ignore_columns` after 2021-11-22 with 14.6
MergeRequests::ExternalStatusCheck:
- external_approval_rule_id Remove `ignore_columns` after 2021-09-22 with 14.3
Ml::Candidate:
- iid Remove `ignore_columns` after 2023-05-01 with 16.0
Namespace:
- unlock_membership_to_ldap Drop column after 2023-10-16
Namespaces::ProjectNamespace:
- unlock_membership_to_ldap Drop column after 2023-10-16
Namespaces::UserNamespace:
- unlock_membership_to_ldap Drop column after 2023-10-16
PlanLimits:
- ci_max_artifact_size_running_container_scanning Drop column after 2021-07-22
- web_hook_calls_high Remove `ignore_columns` after 2022-02-22 with 15.10
Project:
- emails_disabled Drop column after 2023-07-22
ProjectSetting:
- encrypted_jitsu_administrator_password Remove `ignore_columns` after 2023-09-22 with 16.5
- encrypted_jitsu_administrator_password_iv Remove `ignore_columns` after 2023-09-22 with 16.5
- encrypted_product_analytics_clickhouse_connection_string Remove `ignore_columns` after 2023-09-22 with 16.5
- encrypted_product_analytics_clickhouse_connection_string_iv Remove `ignore_columns` after 2023-09-22 with 16.5
- jitsu_administrator_email Remove `ignore_columns` after 2023-09-22 with 16.5
- jitsu_host Remove `ignore_columns` after 2023-09-22 with 16.5
- jitsu_project_xid Remove `ignore_columns` after 2023-09-22 with 16.5
- jitsu_key Drop column after 2023-10-17
RequirementsManagement::Requirement:
- author_id Drop column after 2022-10-22
- cached_markdown_version Drop column after 2022-10-22
- created_at Drop column after 2022-10-22
- description Drop column after 2022-10-22
- description_html Drop column after 2022-10-22
- state Drop column after 2022-10-22
- title Drop column after 2022-10-22
- title_html Drop column after 2022-10-22
- updated_at Drop column after 2022-10-22
SamlProvider:
- relay_state_domain_allowlist Remove `ignore_columns` after 2023-10-22 with 16.6
Security::OrchestrationPolicyConfiguration:
- bot_user_id Drop column after 2023-10-22
SentNotification:
- id_convert_to_bigint Remove `ignore_columns` after 2023-09-22 with 16.5
ServiceDeskSetting:
- custom_email_smtp_address Drop column after 2023-04-22
- custom_email_smtp_port Drop column after 2023-04-22
- custom_email_smtp_username Drop column after 2023-04-22
- encrypted_custom_email_smtp_password Drop column after 2023-04-22
- encrypted_custom_email_smtp_password_iv Drop column after 2023-04-22
User:
- email_opted_in Remove `ignore_columns` after 2023-10-22 with 16.6
- email_opted_in_at Remove `ignore_columns` after 2023-10-22 with 16.6
- email_opted_in_ip Remove `ignore_columns` after 2023-10-22 with 16.6
- email_opted_in_source_id Remove `ignore_columns` after 2023-10-22 with 16.6
UserDetail:
- requires_credit_card_verification Drop column after 2023-05-22
UserPreference:
- experience_level Remove `ignore_columns` after 2021-03-22 with 14.10
- use_legacy_web_ide Drop column after 2023-05-22
Users::InProductMarketingEmail:
- campaign Drop column after 2023-10-15
Users::PhoneNumberValidation:
- verification_attempts Drop column after 2023-10-17
Vulnerabilities::Finding::Evidence:
- summary Remove `ignore_columns` after 2022-03-17 with 14.9How to set up and validate locally
bin/rake db:obsolete_ignored_columnsMR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.
Edited by Peter Leitzen