Draft: add keep-latest-script

Script Purpose: The purpose of this script is to automate the process of removing unnecessary version blocks from YAML files used in an F-Droid build server environment. It replaces the manual "keep-latest" procedure by removing all versions that have been added by the "checkupdates" process, between the last built version and the last upstream version.

Script Overview:

1. The script reads information from the "stats/known_apks.txt" file to determine the last known version of each app.
2. It checks if a YAML file exists for each app in the "metadata" directory.
3. If a YAML file exists and the app doesn't have a "VercodeOperation" specified, it proceeds to remove unnecessary version blocks.
4. It reads the YAML file to get the list of version codes.
5. It identifies the last known version of the app from the "known_apks.txt" file and retrieves the last version code from the YAML file.
6. Based on the last known version and the last version code, it determines which version blocks need to be removed.
7. It removes the unnecessary version blocks from the YAML file.
8. Finally, it prints a message indicating the metadata update for each app and the versions that were removed.

Functions Overview:

• get_all_app_names(): Retrieves a list of all app names from the "known_apks.txt" file.
• get_last_known_version_from_known_apks(app_name): Retrieves the last known version of an app from the "known_apks.txt" file.
• has_vercode_operation(app_name): Checks if a YAML file for the app contains a "VercodeOperation".
• get_all_version_codes_in_yaml_file(app_name): Retrieves all version codes from the YAML file for an app.
• get_values_between(values, start, end): Filters values between two given endpoints from a list.
• remove_version_from_yaml_file(app_name, version_code_to_remove): Removes unnecessary version blocks from the YAML file for an app.
• main(): The main function that orchestrates the entire process.

---

This script streamlines the management of version blocks in the F-Droid build server environment, ensuring that only the necessary ("latest") versions are retained in the metadata files.

The script can be run like this:

tools/keep-latest.py

there is 1 pip requirement:

pip install pyyaml

changed the description

The apks in known_apks.txt are built apks. The versions which are going to be built in the current cycle is not included. Basically you shouldn't parse that file. You should check the git history in that commit.

Sorry I don't understand this. Which commit are you referring to, when you say "that commit" ?

I agree the versions of the current cycle is ignored and while the build cycle runs we don't know which apks it will update (because the build can fail). But why are these versions important ?

Are you presuming, this script will be run at a certain time? My presumption, was this script can be run at any time (just like the current manual procedure). But ideally someone would run this script directly before a new build cycle starts.

After the known apks are updated, a new cycle starts and all versions in the repo but not in the known apks file will be built in the cycle. The script shouldn't remove a version if it's going to be built in the current cycle. It should only remove a version if it's going to be built in the next cycle.

resolved all threads

added 1 commit

• 2575625f - Apply 1 suggestion(s) to 1 file(s)

Compare with previous version

added 1 commit

• 8e3a50c1 - add keep-latest-script

Compare with previous version

But ideally someone would run this script directly before a new build cycle starts.

One can't knew when that happens though.

I thought this will be part of cherkupdate and it will add a new version AND clean up old ones in one go.

By "that commit" I think @linsui means: the "commit of the current cycle"

You can see which one by looking at https://monitor.f-droid.org/builds/running -> fdroiddata version -> fdroiddata@THISISIT

You can get that commit by looking at the latest commit with the message "Update known apks" (which is exactly the one linked in monitor, eg. Today is https://gitlab.com/fdroid/fdroiddata/-/tree/39d322a6bc8574020d8d24ececc45843a47fa45d )

So my thinking is: checkout the tree at THAT COMMIT, then... do your magic.

Hmmm I was trying to mimic the manual procedure that you have to do all the time - so you can use this script as an assistant. However, reading through your comments, I would agree, that this would be best placed in "checkupdates" in the fdroidserver.

I thought this will be part of cherkupdate and it will add a new version AND clean up old ones in one go.

I think I can implement that - so the manual procedure will be redundant soon. However it will be a bigger change, since there is currently no means to doublecheck older commits of previous checkupdates-runs.

So my thinking is: checkout the tree at THAT COMMIT, then... do your magic.

Yes that makes sense. However I cannot simply checkout an old commit and then push a change to it. I still have to update the latest commit somehow.

The manual procedure goes like this, in my case, adapted for this cycle:

• git log --pretty=oneline 39d322a6bc8574020d8d24ececc45843a47fa45d..HEAD | cut -d " " -f 2-|sort|grep Update
• look at those listed twice
• we have some apps that have the same name but they are different (eg. Stable vs Nightly)
• this also lists "update currentversion" for apps that don't have AutoUpdateMode but have UpdateCheckMode, eg. I then manually test and update apps like Acode or PretixScan
• then edit, via a MR if too many, or each one
• edits go through the CI so many times we might see the checkupdate job for that commit fail because a new update was released, so we edit again
• global checkupdate job runs elsewhere (I didn't try to see it live) so our edits might have added a new version now and then checkupdate comes later and adds the same one again (this will fail both in CI and main server, duplicate versionCode)

I tried to list some of the stuff that we see and do around this "keep latest" thing.

Okay it looks like all of that can be resolved, by adding it to "checkupdates" and doing it right - except for the manual updates if there is no AutoUpdateMode .

I tried again to implement some strategies to use the old commit. i found, that I can't use the whole file of the old commit, because if I add something and commit and push I might overwrite some other metadata changes (eg. change of the author website URL). So what I will have to do is:

• get the last "Update known apks" commit
• get the "Builds" part of the yaml file
• checkout latest master again
• replace only the "Builds" part with the one I fetched 2 steps earlier
• run the normal "cheackupdates" procedure
• commit to latest master (if needed)

btw: your

git log --pretty=oneline 39d322a6bc8574020d8d24ececc45843a47fa45d..HEAD | cut -d " " -f 2-|sort|grep Update

can be rewritten to:

git log $(git log --grep="Update known apks" -1 --format=%H)..HEAD --grep="Update" --format=%s | sort

so it works all the time.

except for the manual updates if there is no AutoUpdateMode .

Ah, that was more of a FYI, not related to this :)

marked this merge request as draft

!15730 (merged)

closed