Maintaining Patchmanager's web-catalog & sailversion

[Olf0]

Primarily branching this topic out of discussion #57

• As @CODeRUS provided https://github.com/cODeRUS/django-test (per Necessary steps for maintaining PM #57 (comment)), the extant code for running Patchmanager's web-catalog is available.
• [Further consideration] We will also have to evaluate, where to run this service:
  • Ask Basil (as the maintainer of Openrepos) to keep hosting it at Openrepos (which appears to be the easiest solution). E.g., by re-establishing this web-service at the Patchmanager account there.
  • Evaluate if some Github infrastructure allows for running this service?
• IMO, we should also consider to automatise updating the list of supported SailfishOS releases, by the help of the other, miniature web-service I would like to become re-established. Though this is an enhancement / optimisation, which can be carried out later.

[CODeRUS]

so, decide if you actually need direct access to server or not. if not we can setup github-actions to deploy server after merge to master with no problem.

[Olf0]

so, decide if you actually need direct access to server or not.

The current server instance at https://coderus.openrepos.net/pm2/ ?
That would be really helpful to get things up and running again quickly!
But not me personally, because I lack any Django know-how and have no practical experience in running a web-service; I do not even have a web-catalog account there.
@b100dian and / or @nephros, would one of you consider to take this offer?

if not we can setup github-actions to deploy server after merge to master with no problem.

Yes, but that opens up many questions again: deploy whereto? I am also new to GH-actions (looked at some, never deployed one by myself).
As Vlad already mentioned: "Step by step"

Actually re-establishing the simple "sailversion" crawler (without the manually maintained list of code-names) via these measures (GH-actions and a place to deploy the list) would be the first step, I would like to see happening (simply outputting lines of <SFOS release version>,<date in ISO format>, running daily, adding only a single line per run).
My issue with your Python code snippet is, that I do not speak Python fluent enough to easily write new code; my Python know-how is barely sufficient for understanding simple code (which that is) and small changes (like altering the output format). To construct a counting loop, which starts at the last found release version (pulled from an already deployed list at some web page), checks for a "200" HTTP-response code, extends the list and uploads it again, is likely a couple of days work for me (primarily researching the WWW and reading GH-actions documentation); I only know Bourne-Shell well (and most Bash extensions), and once knew C sufficiently (decades ago).
P.S.: This is how I think sailversion-v2.txt should look like (filled with the current data).

[nephros]

The current server instance at https://coderus.openrepos.net/pm2/ ?
That would be really helpful to get things up and running again quickly!
But not me personally, because I lack any Django know-how and have no practical experience in running a web-service; I do not even have a web-catalog account there.
@b100dian and / or @nephros, would one of you consider to take this offer?

That's a NACK from me.

I like developing stuff, but I do not like to be responsible for things where users will shout loudly if it goes away - like any kind of online service. Also, no experience with jango or any other web framework.

@CODeRUS, is hosting that service something you would like to get rid of?

Because if not, I vote for keeping it the main web catalog service, maintained if necessary by PRs from our side as proposed.
If it breaks, CODeRUS decides to not want to host it any more, or it goes away for some other reason we can still set up hosting (and we need a testing instance in the mean time anyway).

(... and judging from the other things coderus maintains there, going back to things for very ancient Nokias, I have sufficient trust that we can rely on it working for the time being.)

[CODeRUS]

nah, server will stay alive as long as you need :)

[b100dian]

Github actions to deploy to server after merge are good enough for now. We can leave this open to discussion maybe someone (or one of us) steps in later.

[Olf0]

Ack to all these statements.

I (obviously incorrectly) had the impression that @CODeRUS intends to hand over the web-catalog web-service, too.

[CODeRUS]

of course not :))

sailversiion as well as pm web catalog will remain as long as you use it :))

[Olf0]

Thank you!
And many thanks for this clarification.

It is nice, that this turned out to be just a misunderstanding on my side.

I now see, that you updated sailversion with "4.2.0.21 - Verla - 09 Sep 2021".
Shall I notify you per email to your Gmail account, if some version is missing in the future, as I did in the past?

P.S.: So I can revert Olf0/sfos-upgrade@fa41a54 now.

[CODeRUS]

need to check if script can be improved.
currently script only looking for 4.2.0.22 + 20, maybe can be improved to check also for 4.2.1.1 + 30 and 4.3.0.1 + 30

[Olf0]

... and 5.0.0.1 + 30

I.e., the highest recently released version is major.minor.micro.point (e.g., currently 4.2.0.21):

1. Iterate from major.minor.micro.(point+1) to major.minor.micro.(point+20) (actually, (point+10) is likely sufficient, nowadays: Jolla's release scheme has settled since the "early days").
   If a new version is found, go to 6.
2. Iterate from major.minor.(micro+1).1 to major.minor.(micro+1).30
   If a new version is found, go to 6.
3. Iterate from major.(minor+1).0.1 to major.(minor+1).0.30
   If a new version is found, go to 6.
4. Iterate from (major+1).0.0.1 to (major+1).0.0.30
   If a new version is found, go to 6.
5. Optionally: Output that no new version was found at this run.
   And go to 7.
6. Add the newly found version to the list.
7. Sleep for a day, then start over at 1.

That is at most 110 or 100 requests, which can be spread over 1440 minutes (a day): So sleeping up to 13 minutes (suitable prime number of seconds would be e.g., 727, 733, 739, 743, 751, 757, 761, 769, 773) after each request is fine in order to not hammer the SFOS-release server with suspicious requests.

That is the reason, why I suggested to put your complete sailversion script into some repository, so

• we can discuss it.
• we can review it.
• we can contribute ideas for it.
• we can contribute code to it.
• I can try to learn a little Python by trying to understand it.

If you are afraid to have this script public (I can understand that, because your day-job is also in that realm), you can create a private repository here and grant only members of the Github-"organisation" sailfishos-patches access (even selectively for specific people, if you want that).

[nephros]

One could also poll FSO for a "release notes" post, and increase polling if a recent one exists, relax polling otherwise.
Say, do a full run every three days usually, and if a EA/Release is announced, poll twice a day or so.

[CODeRUS]

actually, i can write a script pretends to be a device checking for updates without crawling anything :)

[nephros]

actually, i can write a script pretends to be a device checking for updates without crawling anything :)

Clever, sneaky, cool.

However being sneaky about this is probably not wise, we might cause Jolla to not like us, and make life hard as a consequence. Better to be open and communicate to them that we're crawling (User Agent and things), and not abusing their service.

[CODeRUS]

crawling cost much more for jolla

[Olf0]

actually, i can write a script pretends to be a device checking for updates without crawling anything :)

Clever, sneaky, cool.

Mmmh, for me just "clever & cool". 😉

However being sneaky about this is probably not wise, we might cause Jolla to not like us, and make life hard as a consequence. Better to be open and communicate to them that we're crawling (User Agent and things), and not abusing their service.

While I would generally agree with that mindset when communicating with peers (i.e., with other persons, personally and among equals / on a par), I have seen that well-intended strategy often failing when dealing with institutions (e.g., companies). Especially when they are usually opaque (= intransparent / not "open" at all). And Jolla is a perfect example of a company trying to be as opaque as possible. They never state anything not-positive, but that does not prevent them from performing such actions: They just keep their mouth shut, even when people start cry out loud. They also love to try to perform actions a bit stealthily, which may cause conflicts, e.g. the original posting announcing the git.sailfishos.org and OBS shutdown: The headline is just incredibly deceiving, IMO.
Another example is, that I do not really ever expect an answer or any other reaction to this request I posed: Actually something quite similar to what we are discussing here. I also considered, if people using sfos-upgrade may have triggered this measure, but that is unlikely with 100 to 200 users, which sure do not upgrade their SFOS installations at the same time (and sfos-upgrade performed a single download of that web-page per run). Side note: Mind, that sfos-upgrade allows users to upgrade devices with "SailfishOS free" ("trial version" from Jolla's perspective) installed (for which upgrading at the GUI is blocked by Jolla) much safer and easier than per "zypper dup" / "version dup".

IMO, for companies, which are not unconventionally open, the correct line of thinking is: If they can gain something by fulfilling a request, they may do it, if not, they sure will not.
Hence the right behaviour in the latter case is a "Don't ask, don't tell" policy.

That is why I think the better approach is trying hard to slip under their radar, in this specific case. So mimicking a device in EA mode looking for the next version to upgrade to sounds perfect for me, so that version is actually announced to that "device" and no crawling is necessary.

TL;DR: Jolla does not like us (we are just little paying beta-testers for them, causing a lot of effort; they only having a single, real customer from their perspective in the last couple of years) and makes our life hard, anyway (but basically just due to negligence / carelessness). Thus I cannot see a reason to play overly nice, although we should definitely stay away from doing something bad (a category, in which trying to download a couple of repo.xml files or asking for the next version to upgrade to does not fall, IMO).

[Olf0]

Well, per @CODeRUS' very kind and (at the same time) bold statement,

sailversion as well as pm web catalog will remain as long as you use it :))

, we can close this discussion, because it turned out to be unnecessary / superfluous.

[Olf0]

I intend to delete this discussion in a couple of days, if everybody is O.K. with that.
Hence please speak up, if you think something discussed in this thread is still unresolved or some information should be saved.

[nephros]

Object to deletion, because this documents coderus' commitment to maintaining/providing Web Catalog.
Lock it, just like the other discusions.

[Olf0]

O.K. & done.

[Olf0]

... plus adapted the title accordingly.

[nephros]

Reopening the discussion here:

[Further consideration] We will also have to evaluate, where to run this service:

• Ask Basil (as the maintainer of Openrepos) to keep hosting it at Openrepos (which appears to be the easiest solution). E.g., by re-establishing this web-service at the Patchmanager account there.

The recent developments have caused Openrepos do change hosters, but its future is unclear at the moment. In the wake of this @CODeRUS has called for someone to step up and host the Web Catalog independently.

So, anyone have a usable web service provider ready and wants to maintain it?

(... and by the way, establishing an Openrepos mirror would probably be a good idea as well...)

[nephros]

@b100dian @Olf0

• shall we/ should we bring the web catalog "home", i.e. under this github orgnization?
• are we up to hosting it (which will likely incur some costs: money and admin effort)

[nephros]

So, anyone have a usable web service provider ready and wants to maintain it?

AFAIK it could be hosted here at github...

[nephros]

update: I have succeeded to get a test version of the catalog running on my home lab server.

[nephros]

update: ... and dockerized it ,which may or may not improve maintainability

[nephros]

... and made some changes https://github.com/nephros/django-test

• rename the app proper from example/demo to pmwc/catalog
• create a Dockerfile
• some minor cleanups, like changing the "Patchmanger 2.0" title

[Olf0]

Side note, first: @nephros, I did not receive any notifications, but saw today that this discussion has been bumped to the top of the list. Hence I took a look at it and so became aware of these developments just recently.
I assume @b100dian also did not receive any notifications for your new messages at this discussion thread within the last three days.
As I believe to have received notifications for GH-discussions messages before, I switched the "lock conversation" off, because that is the only difference I can spontaneously think of. If both of you now receive a notification for this message and @b100dian also did not receive any notifications for @nephros earlier messages, this would constitute a bug I might report to GitHub: Hence please denote that in a reply.

For nephros' original question(s), I also would like to ask @poetaster to join this discussion, because at least the aspect of an OpenRepos mirror might be of interest for him, but maybe he can also contribute ideas to the hosting aspect of your question.

I also would like to assess how urgent this is. It sounds as if @CODeRUS contacted @nephros privately, so maybe @nephros can shed some light how pressing @CODeRUS perceives the situation or @CODeRUS can speak for himself here. @custodian ("basil") does sound as if he is executing a proper plan in his message at OpenRepos, but he is obviously quite terse (OTOH, he always tends to be terse).

AFAIK it could be hosted here at github...

This has to be evaluated. We can easily open a web-page at patchmanager.github.io, this is also allowed to contain some server-side JavaScript IIRC, but I have no idea if it is possible and allowed to deploy a permanently running django instance. AFAIR, GH-actions are not for continuously running instances, and have no idea, if they have access to a network interface with a route to the internet (likely not, or people would abuse them for various things). This means reading GH-documentation, maybe even contacting GitHub and asking (when we have something concise to ask), or maybe also trying things. OTOH, it looks as if only non-public repositories are limited in storage space and GH-actions minutes per month.

Another idea to evaluate would be, why mentaljam has chosen Netlify for hosting sailfishos-chum-web. Maybe they offer an appropriate "free plan" for FLOSS-projects.

[b100dian]

First let me confirm I did not receive notification for the posts above - safe for this one; maybe tagging does not work in replies.

I'll try to expand answers to the posts(s) above later today, but at first sight:

Indeed, we do need to evaluate hosting options, and I would add that it would be nice to have deployments done when PRs are merged (e.g. we could test with the version array update;). I would prefer not to use self hosting 'own hardware' and use a third-party service that can be configured for all of us to access.

Also, we need a strategy to migrate the DB then the URL in the patchmanager app, then some sort of redirection on the old instance.

[nephros]

It sounds as if @CODeRUS contacted @nephros privately, so maybe @nephros can shed some light how pressing @CODeRUS perceives the situation or @CODeRUS can speak for himself here.

Semi-privately.

Both him and Basil were in the Telegram group announcing the hosting move of Openrepos. As the situation of the future of OR is also unclear, he suggested I look for alternatives, to which I agreed (to the looking not necessarily the hosting). Hence the resurrection of this thread.

[nephros]

I'll try to expand answers to the posts(s) above later today, but at first sight:

Indeed, we do need to evaluate hosting options, and I would add that it would be nice to have deployments done when PRs are merged (e.g. we could test with the version array update;). I would prefer not to use self hosting 'own hardware' and use a third-party service that can be configured for all of us to access.

Also, we need a strategy to migrate the DB then the URL in the patchmanager app, then some sort of redirection on the old instance.

If we are agreed that we will at least take over the code base, we should make a fork in this organization here. I have already done some changes in my personal fork.

I think only you @b100dian have the necessary permission though (I have tried, I do not).

About hosting and migration: Fully agree a remote service with several accessors is the way to go.

I have as mentioned begun to familiarize myself with running that django program, and have asked coderus for a copy of the current database to play around with - no reply as of this time. I will then determine what s required to do a migration once we switch and hopefully can do it/assist there.

As far as I can see requirements are:

1. running django
2. usually that is connected to a frontend web server using WSGI, so that
3. ^^^ alternatively running as a web service directly from django - not recommended for "production" but probably good enough for us
4. possibility to map/serve the /media and /static URIs, and the pm2 root URI namespace
5. valid email account to be able to send account creation/password recovery emails
6. https certificates
7. resource usage (RAM/CPU) should be minimal, but we need a bit of local storage for all the patch.tar.gz files
8. bandwidth: no idea about how much it is utilized, doesn't need to be especially fast though

As mentioned, I have dockerized it so in case not all of the above can be fulfilled, the upper ones can be handled in Docker. Then we need a hoster which runs Docker containers, and still need the frontend, connectivity and SSL stuff.

As a side note: there are several domains available for "patchmanager", such as .app, .me, .net, .org.
Do we want a domain?

[nephros]

By the way, the version with my changes, dockerized, can be viewed at sailfishDOTnephrosDOTorgSLASHpmwc, port 80, NOT port 443!

[Olf0]

Oh, Django is all Python. Now I understand the need for a vServer or a Docker host.

Forking sound reasonable, if adapting for hosting the Web Catalog elsewhere is inevitable.

An own domain involves extra costs, I would assume that any service could be embedded at patchmanager.github.io, correct?
If a patchmanager domain is needed, my ranking sure is .org, then .net, then the others.

[poetaster]

Just to note that I'm going to stay out of the discussion, but have in mind the question if my infra at hetzner (one machine definitely not, one probably) could be serviceable. However, I avoid docker (use LXC or LinuxVserver), so it may just not be a good match anyway. + time. have little.

[Olf0]

Just to note that I'm going to stay out of the discussion,

That is fine; sorry, if I did not make enough, that all I was hoping for as a contribution from you in this thread was a hint or ideas how to resolve the question, "Where to host Patchmanager's Web Catalog?".

but have in mind the question if my infra at hetzner (one machine definitely not, one probably) could be serviceable.

This offer is much more than I expected. Thank you!

However, I avoid docker (use LXC or LinuxVserver), so it may just not be a good match anyway.

This is fine, as @nephros stated he only put the Django instance in a Docker image, because he expected that to be easier to deploy. Django primarily needs Python 3, even the WSGI conformant web-server can be omitted, see his list of requirements. Hence a LinuxVserver instance should be sufficient.

[poetaster]

This is fine, as @nephros stated he only put the Django instance in a Docker image, because he expected that to be easier to deploy. Django primarily needs Python 3, even the WSGI conformant web-server can be omitted, see his list of requirements. Hence a LinuxVserver instance should be sufficient.

I'm on the categorically deny the use of docker front :) But that doesn't preclude using another container type. And the requirements are 'standard'. By which I mean to say that python 3.x is standard on all my boxes. My python stuff is all in werkzeug/flash using python 3. Still, it may just make more sense for this effort to rent an inexpensive virtual which supports docker. It's a small application and will get relatively little traffic. And if the parties involved don't know python well, using django is at least disciplined if overkill.

One thing I would advise is running a proxy in front of django in any case. If @nephros wants to have some fun (in the Sysadmin sense of fun), I run a combination of ha-proxy and varnish (or nginx). it's much easier to manage in the long run since ha-proxy does things like 'load all the .pem files in directory x), just to name an example. and varnish (nginx) obviously hold all that static stuff in ram. ok. I'm in deeper than I planned already. One could just run apache as a proxy, but that's also more work, less benefits.... Dammit. back to work.