Draft: new standalone command: fetch_metadata, replacing update functionality.

When I try to add more search path for fastlane metadata, I take the chance to extract the whole function in to a sub command. I'm not sure if the performance will be affected. Before I polish this MR further I want to also get some feedback.

What I have done:

1. Break up copy_triple_t_store_metadata and insert_localized_app_metadata into small functions and classes.
2. Split the process into two parts. First the text is read and the image path is indexed. Then the images are copied.
3. Add more search path for fastlane.
4. Currenly both the mtime and size are compared between the images in repo and source. Because we also strip some images these images are always copied again and again. I removed the size check.
5. Add a new fetch_metadata subcommand (maybe there is a better name?) so that the metadata structure can be checked easily.

What I haven't done:

1. Add an item so that the search path can be specified in build metadata. Is this necessary? Should we support path out of the repo? E.g. another repo or a tarball?
2. Clean up leftover images from repo. Currently F-Droid also support adding images into repo directly. How should we decide if the image is leftover?
3. If there are images fetched from other path then all other images in repo should be remove.
4. Drop this function and ask users to host the images in build dir instead.
5. Output the result to a file.

What I want to drop/change:

1. Currently fastlane structure in fdroiddata is supported, i.e. short_description.txt, etc. However they are not used and only the custon style, i.e. summary.txt is used.
2. The author info from triple-t seems only used by few apps. They can be added to the build metadata directly anyway.
3. Description and changelog is not stripped. I don't know if there is any use case to keep the new line and space.

added 1 commit

• afdfa3e9 - fetch_metadata

Compare with previous version

@uniqx has been looking at fastlane stuff recently, so I'm requesting a review.

requested review from @uniqx

added 30 commits

• afdfa3e9...b222275d - 29 commits from branch fdroid:master
• e253be9b - fetch_metadata

Compare with previous version

changed milestone to %IPFS and .ipa publishing

added update label

changed title from fetch_metadata to new standalone command: fetch_metadata, replacing update functionality.

removed review request for @uniqx

marked this merge request as draft

I welcome improvements to the metadata parsing, especially involving profiling it. I think it takes up a substantial part of the time needed to run fdroid update.

Seems like a big change to rip this out of fdroid update. There are some poorly documented assumptions about the state of the apps' git repos that the whole cycle relies on, so making that kind of change. Is it necessary? If so, I would take that on as its own project and merge request, then make other changes elsewhere.

Also, FYI, description.txt is used in fdroiddata:

$ ls -1  metadata/*/en-US/description.txt 
metadata/click.dummer.funphonepuppet/en-US/description.txt
metadata/com.google.zxing.client.android/en-US/description.txt
metadata/com.hwloc.lstopo/en-US/description.txt
metadata/com.igisw.openmoneybox/en-US/description.txt
metadata/com.jakewharton.sdksearch/en-US/description.txt
metadata/com.jereksel.libresubstratum/en-US/description.txt
metadata/com.kolloware.wechange/en-US/description.txt
metadata/com.menny.android.anysoftkeyboard/en-US/description.txt
metadata/com.zfdang.touchhelper/en-US/description.txt
metadata/com.zfdang.zsmth_android/en-US/description.txt
metadata/cz.jirkovsky.lukas.chmupocasi/en-US/description.txt
metadata/de.kugihan.dictionaryformids.hmi_android/en-US/description.txt
metadata/de.schildbach.oeffi/en-US/description.txt
metadata/de.thecode.android.tazreader/en-US/description.txt
metadata/de.tobiasbielefeld.solitaire/en-US/description.txt
metadata/io.wookey.wallet/en-US/description.txt
metadata/jackpal.androidterm/en-US/description.txt
metadata/net.osmand.plus/en-US/description.txt
metadata/org.blokada.fem.fdroid/en-US/description.txt
metadata/org.fdroid.fdroid.ota/en-US/description.txt
metadata/org.fdroid.fdroid.privileged.ota/en-US/description.txt
metadata/org.kde.ktrip/en-US/description.txt
metadata/org.libre.agosto.p2play/en-US/description.txt
metadata/org.mozilla.klar/en-US/description.txt
metadata/org.safecoin.safeprice/en-US/description.txt
metadata/org.smssecure.smssecure/en-US/description.txt
metadata/org.wikipedia/en-US/description.txt
metadata/pc.javier.seguime/en-US/description.txt
metadata/pl.sanszo.pcis/en-US/description.txt

Also, FYI, description.txt is used in fdroiddata:

It's expected. But short_description.txt and full_description.txt shuold be droped.

Seems like a big change to rip this out of fdroid update. There are some poorly documented assumptions about the state of the apps' git repos that the whole cycle relies on, so making that kind of change. Is it necessary? If so, I would take that on as its own project and merge request, then make other changes elsewhere.

I'm not sure what you mean. Do you want to run this seperately? I thought this MR makes it possible to run it concurrently and incrementally though. But it's not part of this MR.

Smaller changes are much easier to get merged, but if you think it is important to rearchitect this piece, then I guess you should continue with this approach. But that'll probably mean it'll take a long time to get merged.

I agree we don't need to support using those files in fdroiddata but if it makes the parsing code simpler, I think it is fine to keep supporting them.

What kinds smaller changes do you want? If it can make review easier I can break up this MR.

I agree we don't need to support using those files in fdroiddata but if it makes the parsing code simpler, I think it is fine to keep supporting them.

In this MR I split different format into different classes so it's very easy to choose what to support.

But that'll probably mean it'll take a long time to get merged.

It's OK to take a long time to review this MR. But the current status is that I get no review.

Also, IMHO any field that Triple-T and/or Fastlane provides that maps to our fields should be supported. That makes it much easier for upstreams and means we do not have to document exceptions.

removed milestone %IPFS and .ipa publishing

added 509 commits

• e253be9b...e52a07d6 - 508 commits from branch fdroid:master
• a429408b - fetch_metadata

Compare with previous version

added 1 commit

• 2db8f139 - fetch_metadata

Compare with previous version

changed milestone to %buildbot

mentioned in merge request !1679 (merged)