Sign in Join

Live chat for Free Pascal, Lazarus, Delphi and every Pascal programmer.

Mailing lists / lazarus / 2025-01

Online Package Manager Questions

9 messages · 5 people · –

Reply by email Subscribe to this list Your subscription

Read-only archive. Use j and k to move between messages.

Tony Whyman

I recently had a support question for IBX for Lazarus and for an issue that was fixed a couple of years ago. The response was inevitably a "please upgrade to the next version".

However, what surprised me was that the user came back pointing out that he was using the most recent version of IBX on OPM. This was a surprise to me as I had not uploaded IBX to the Online Package Manager nor do I recall a request to do so. On the other hand, an out-of-date version (2.5.0) of IBX is on OPM.

I don't object to IBX being on OPM - its probably a good idea, but it does raise a few questions:

1. Who makes the decision about what is uploaded to the OPM repository?

2. How is a package put there and what precautions are taken to ensure that the package is genuine and does not contain malicious code - especially when the upload was not done by the original author.?

3. How are the OPM repository maintainers told that a new version of a package is now available?

Tony Whyman

https://www.mwasoftware.co.uk

https://github.com/MWASoftware

Reply by email

Werner Pamler
to Tony Whyman

First of all: OPM was written and maintained by Balazs Szekely (aka GetMem). For a year, however, I have not heard from him any more - I really really hope that he is doing fine. For the moment OPM is maintained by myself.

Your questions:

Tony Whyman wrote 6 lines
1. Who makes the decision about what is uploaded to the OPM repository? 2. How is a package put there and what precautions are taken to ensure that the package is genuine and does not contain malicious code - especially when the upload was not done by the original author.? 3. How are the OPM repository maintainers told that a new version of a package is now available?

When the package owner has the feeling that a new version should be released to OPM he notifies the OPM maintainer by mail to containing a link to the new version or a zipped
file as attachment. I look over the code, compile it and run some of the sample projects provided.

Anybody can submit packages for inclusion in OPM.? Of course, I don't know any of the submitters personally, and I have no idea whether he/she is the original author. So, sorry that the your IBX files made it into OPM without your knowledge. But I don't see a way how to improve that without building up a huge beaurocracy.

Criteria to accept a package are for me (maybe they were different for Balasz):

* The library must be a package. Individual, isolated units cannot be handled by OPM.
* It must contain a brief description in the meta data and package file.
* It must contain a statement on the license. Commercial licenses are rejected. I also reject packages which "smell" like being pirated (for example, when there is the original Borland header in the units).
* The package and its files must be in English - this is an international community, and there is no other way to communicate.
* It must compile at least under the current releases of Lazarus/FPC. Of course, the more combinations are welcome. The working combinations should be specified in the meta-data (json), as well as the widgetset for which it works.
* There should be at least one sample project. Otherwise I have no way to test whether the code is working (I am not willing to write test projects for foreign code).
* Ideally there should be some documentation, either included as help files in the package, or as a separate wiki page, or similar.
* The package submitter must express his/her commitment to maintain the package if, for example, it does not compile any more due to compiler or widgetset changes. Unfortunately we have many unmaintained packages already now, and I tend to remove a non-functioning unmaintained package.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus-ide.org/pipermail/lazarus/attachments/20250111/72ef1415/attachment.htm>

Reply by email

Tony Whyman
to Werner Pamler

Werner,

I can certainly sympathise with with your workload for maintaining the OPM archive, and your reply was very useful. There are two threads that lead from this for me. The first is to get IBX on OPM up-to-date and that we should deal with by direct EMail. The second is really more general and to hopefully have a wider discussion on the how the OPM archive should be maintained in future.

Looking at your response, the issue that leapt out for me is "accountability". You seem to be taking on too much accountability when it comes to code published on the OPM archive and especially when it comes to checking that the code is at least compilable and is of an appropriate standard.

You said "There should be at least one sample project. Otherwise I have no way to test whether the code is working (I am not willing to write test projects for foreign code)."

This implies that you are going to compile, test and look at the results for every package submitted to OPM and potentially for each update (i.e. lots of work for you). In the case of IBX you do at least have 17 example projects to choose from plus a comprehensive regression test suite with 29 individual tests for the main package and a further 22 for the underlying Firebird Pascal Interface package (fbintf) - but you need to set up your own Firebird server to perform the tests/examples.

The regression tests make for a very stable and reliable package, but it seems to be asking a lot for you to check the results and validate the package as "good to go". My day job has been in safety critical aviation software and so I am well aware as to how much work goes into third party software certification.

Have you thought about requiring that submitted packages are signed by the author/submitter? The author can then take responsibility for at least the provenance of the code, correct licensing and that it compiles
- I would not go as far as "fitness for purpose" as open source software can only work if the user accepts that part of the deal.

That way you can limit your checks to when the package is first submitted and with updates the checking can be limited to signature verification with the occasional deeper check.

Do you also send out automated regular EMails (e.g. every 6 months after a package was first submitted asking authors to re-affirm that the package is up-to-date - and removing any where the author has gone AWOL)? If not, then that would be one way of avoiding stale packages on the archive.

Tony Whyman

Werner Pamler wrote 56 lines
First of all: OPM was written and maintained by Balazs Szekely (aka GetMem). For a year, however, I have not heard from him any more - I really really hope that he is doing fine. For the moment OPM is maintained by myself. Your questions: Am 11.01.2025 um 16:17 schrieb Tony Whyman via lazarus: > 1. Who makes the decision about what is uploaded to the OPM repository? > 2. How is a package put there and what precautions are taken to > ensure that the package is genuine and does not contain malicious > code - especially when the upload was not done by the original author.? > 3. How are the OPM repository maintainers told that a new version of > a package is now available? When the package owner has the feeling that a new version should be released to OPM he notifies the OPM maintainer by mail to containing a link to the new version or a zipped file as attachment. I look over the code, compile it and run some of the sample projects provided. Anybody can submit packages for inclusion in OPM.? Of course, I don't know any of the submitters personally, and I have no idea whether he/she is the original author. So, sorry that the your IBX files made it into OPM without your knowledge. But I don't see a way how to improve that without building up a huge beaurocracy. Criteria to accept a package are for me (maybe they were different for Balasz): * The library must be a package. Individual, isolated units cannot be handled by OPM. * It must contain a brief description in the meta data and package file. * It must contain a statement on the license. Commercial licenses are rejected. I also reject packages which "smell" like being pirated (for example, when there is the original Borland header in the units). * The package and its files must be in English - this is an international community, and there is no other way to communicate. * It must compile at least under the current releases of Lazarus/FPC. Of course, the more combinations are welcome. The working combinations should be specified in the meta-data (json), as well as the widgetset for which it works. * There should be at least one sample project. Otherwise I have no way to test whether the code is working (I am not willing to write test projects for foreign code). * Ideally there should be some documentation, either included as help files in the package, or as a separate wiki page, or similar. * The package submitter must express his/her commitment to maintain the package if, for example, it does not compile any more due to compiler or widgetset changes. Unfortunately we have many unmaintained packages already now, and I tend to remove a non-functioning unmaintained package.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lazarus-ide.org/pipermail/lazarus/attachments/20250112/e2c68947/attachment.htm>

Reply by email

Marc Weustink
to Werner Pamler: First of all: OPM was written and maintained by Balazs Szekely (aka GetMem). For a year,…

Please note that the opm email address mentioned should be sent to lazarus-ide.ORG and not to .COM

Marc

Werner Pamler wrote 37 lines
First of all: OPM was written and maintained by Balazs Szekely (aka GetMem). For a year, however, I have not heard from him any more - I really really hope that he is doing fine. For the moment OPM is maintained by myself. Your questions: Am 11.01.2025 um 16:17 schrieb Tony Whyman via lazarus: > 1. Who makes the decision about what is uploaded to the OPM repository? > 2. How is a package put there and what precautions are taken to ensure that the package is genuine and does not contain malicious code - especially when the upload was not done by the original author.? > 3. How are the OPM repository maintainers told that a new version of a package is now available? When the package owner has the feeling that a new version should be released to OPM he notifies the OPM maintainer by mail to containing a link to the new version or a zipped file as attachment. I look over the code, compile it and run some of the sample projects provided. Anybody can submit packages for inclusion in OPM.? Of course, I don't know any of the submitters personally, and I have no idea whether he/she is the original author. So, sorry that the your IBX files made it into OPM without your knowledge. But I don't see a way how to improve that without building up a huge beaurocracy. Criteria to accept a package are for me (maybe they were different for Balasz): * The library must be a package. Individual, isolated units cannot be handled by OPM. * It must contain a brief description in the meta data and package file. * It must contain a statement on the license. Commercial licenses are rejected. I also reject packages which "smell" like being pirated (for example, when there is the original Borland header in the units). * The package and its files must be in English - this is an international community, and there is no other way to communicate. * It must compile at least under the current releases of Lazarus/FPC. Of course, the more combinations are welcome. The working combinations should be specified in the meta-data (json), as well as the widgetset for which it works. * There should be at least one sample project. Otherwise I have no way to test whether the code is working (I am not willing to write test projects for foreign code). * Ideally there should be some documentation, either included as help files in the package, or as a separate wiki page, or similar. * The package submitter must express his/her commitment to maintain the package if, for example, it does not compile any more due to compiler or widgetset changes. Unfortunately we have many unmaintained packages already now, and I tend to remove a non-functioning unmaintained package.

Reply by email

Werner Pamler
to Tony Whyman: I can certainly sympathise with with your workload for maintaining the OPM archive, and…
Tony Whyman wrote 13 lines
You said "There should be at least one sample project. Otherwise I have no way to test whether the code is working (I am not willing to write test projects for foreign code)." This implies that you are going to compile, test and look at the results for every package submitted to OPM and potentially for each update (i.e. lots of work for you). In the case of IBX you do at least have 17 example projects to choose from plus a comprehensive regression test suite with 29 individual tests for the main package and a further 22 for the underlying Firebird Pascal Interface package (fbintf) - but you need to set up your own Firebird server to perform the tests/examples.

There is no general rule. I certainly will not run all sample projects, and in fact, before I uploaded your new version to OPM today, I ran only two. I knew that you were the author, the overall impression of the code was good. This makes me "trust" you. If the author would have been unknown, or assuming that there were massive preparations to test (installing third-party servers/clients etc), I would have asked the author to first present the package(s) in the forum to hear about the experience of others. And this, in fact, is the way it usually goes: People have an idea for a component/package and "exhibit" it in the forum first. When it gets good feedback they ask for adding to OPM.

The regression tests make for a very stable and reliable package, but it seems to be asking a lot for you to check the results and validate the package as "good to go".

Most of the OPM packages do not contain unit tests at all, I probably would not see even if I'd run the tests: what if there were fails? Would this be enough to reject the update? Not knowing the details of that software, I would still accept the submission and let the users decide.

Show quoted text 6 lines
Have you thought about requiring that submitted packages are signed by the author/submitter? The author can then take responsibility for at least the provenance of the code, correct licensing and that it compiles - I would not go as far as "fitness for purpose" as open source software can only work if the user accepts that part of the deal.

No beaurocarcy please. Requiring authors to sign their contributions would kill OPM.

Show quoted text 5 lines
Do you also send out automated regular EMails (e.g. every 6 months after a package was first submitted asking authors to re-affirm that the package is up-to-date - and removing any where the author has gone AWOL)? If not, then that would be one way of avoiding stale packages on the archive.

Again: too much beaurocracy, now for me - I don't even have e-mail addresses of what if the author/maintainer does not respond?

To be honest, once a package has made it into OPM it has a "safe life" there. There is no chance to re-test all existing
sometimes, usually after changes in FPC or in the LCL, users report in the forum that packages do not work any more. In this case, the user (or I) contacts the author or files a bug report in the author's repository, and the issue gets fixed. In other cases, the author is not active any more: when there is an easy fix I patch the component byself and replace the non-functional package. When this is not possible, the package is removed.

Reply by email

Kostas Michalopoulos
to Werner Pamler: There is no general rule. I certainly will not run all sample projects, and in fact,…
Werner Pamler wrote 9 lines
To be honest, once a package has made it into OPM it has a "safe life" there. There is no chance to re-test all existing sometimes, usually after changes in FPC or in the LCL, users report in the forum that packages do not work any more. In this case, the user (or I) contacts the author or files a bug report in the author's repository, and the issue gets fixed. In other cases, the author is not active any more: when there is an easy fix I patch the component byself and replace the non-functional package. When this is not possible, the package is removed.

Would it be possible to do some periodic 'smoke tests' where all packages are compiled against latest (main) Lazarus (preferably with latest/main FPC too, at least as an extra pass - i.e. main branch Lazarus compiled with main branch FPC) to find any packages that won't compile and even run automated tests for those packages that provide them?

Perhaps this could help not only to find early which packages might break but also - and most importantly IMO - will help find 'obvious' backward compatibility regressions in Lazarus (and FPC, if a pass for main FPC branch is used) before any releases are made.

Of course this wont help with packages that compile and have tests that pass (or wont have any tests at all) that still have their behavior affected negatively by changes in Lazarus / FPC, but it could still be useful to find and avoid some regressions.

Kostas

Reply by email

Werner Pamler
to Kostas Michalopoulos
Kostas Michalopoulos wrote 6 lines
Would it be possible to do some periodic 'smoke tests' where all packages are compiled against latest (main) Lazarus (preferably with latest/main FPC too, at least as an extra pass - i.e. main branch Lazarus compiled with main branch FPC) to find any packages that won't compile and even run automated tests for those packages that provide them?

Good idea but cannot be done manually (there are 280 repository packages with 350 lpk files at the moment). "Somebody" should write a script for that. On a local PC, it would probably require a dedicated test installation of Lazarus for this sole purpose because I don't know if Lazarus would still be usable with so many packages installed. Or maybe "github actions" or similar to run the job on a remove server? Experienced volonteers are welcome...

Reply by email

Martin Frb
to Werner Pamler
Werner Pamler wrote 14 lines
Am 13.01.2025 um 02:59 schrieb Kostas Michalopoulos via lazarus: > Would it be possible to do some periodic 'smoke tests' where all > packages are compiled against latest (main) Lazarus (preferably with > latest/main FPC too, at least as an extra pass - i.e. main branch > Lazarus compiled with main branch FPC) to find any packages that > won't compile and even run automated tests for those packages that > provide them? Good idea but cannot be done manually (there are 280 repository packages with 350 lpk files at the moment). "Somebody" should write a script for that. On a local PC, it would probably require a dedicated test installation of Lazarus for this sole purpose because I don't know if Lazarus would still be usable with so many packages installed. Or maybe "github actions" or similar to run the job on a remove server? Experienced volonteers are welcome...

You don't need to install them all.

Just compile each package against the lazarus directory (e.g. using lazbuild).

That checks that the package can be build with a given LCL + IdeIntf => meaning it would be compilable as part of the IDE. Of course package that are already tested as dependency of another don't need to be done by themself again.

Running it is of course a different beast.

I don't know how much time that would need to build them all. In terms of available CI minutes on gitlab => it is possible to have tasks that run only on request => but if it failed it would need several reruns...

Reply by email