Closes https://dev.gitlab.org/gitlab/organization/issues/842
/cc: @pcarranza, @yorickpeterse, @rspeicher, @rymai