Sign in Join

Live chat for Free Pascal, Lazarus, Delphi and every Pascal programmer.

Mailing lists / fpc-pascal / 2021-08

How to get fpc and lazarus sources when svn has shut down?

44 messages · 12 people · –

Reply by email Subscribe to this list Your subscription

Read-only archive. Use j and k to move between messages.

Bo Berglund

I have for many years been using an install script I wrote to install a complete freepascal/lazarus dev environment on Raspberry Pi devices.

The script uses subversion to get the sources for fpc and lazarus via svn based on the specific tags.
Then the script installs the needed dependencies and makes the complete installation on Raspberry Pi boxes.

Now I have this probles:
- The fpc svn server has been shut down so the source retrieval does not work

Please advice how to:
1) Find which is the latest release tag of fpc and lazarus
2) Retrieve the full sources for that tag on the command line from gitlab

(I do not need the sources to be "live", I can use an export that cannot be updated because I am not participating in developing either tool. But it must be something that can be incorporated in a script and run unattended.)

As soon as I have the sources I believe the rest of my script will work just fine...

I have used these lines in my script to accomplish the retrieval (with script variables holding target location, tag and version):

echo "Downloading version $FPCVER of FPC"
cd "$FPCDIR"
svn co https://svn.freepascal.org/svn/fpc/tags/$FPCTAG/ $FPCVER

echo "Downloading version $LAZVER of Lazarus"
cd "$LAZDIR"
svn co https://svn.freepascal.org/svn/lazarus/tags/$LAZTAG/ $LAZVER

What could I replace these svn co (or svn export) calls with?

All of my developing is versioned via subversion so I know next to nothing about git...

Bo Berglund Developer in Sweden
Michael Van Canneyt
to Bo Berglund
Bo Berglund wrote 30 lines
I have for many years been using an install script I wrote to install a complete freepascal/lazarus dev environment on Raspberry Pi devices. The script uses subversion to get the sources for fpc and lazarus via svn based on the specific tags. Then the script installs the needed dependencies and makes the complete installation on Raspberry Pi boxes. Now I have this probles: - The fpc svn server has been shut down so the source retrieval does not work Please advice how to: 1) Find which is the latest release tag of fpc and lazarus 2) Retrieve the full sources for that tag on the command line from gitlab (I do not need the sources to be "live", I can use an export that cannot be updated because I am not participating in developing either tool. But it must be something that can be incorporated in a script and run unattended.) As soon as I have the sources I believe the rest of my script will work just fine... I have used these lines in my script to accomplish the retrieval (with script variables holding target location, tag and version): echo "Downloading version $FPCVER of FPC" cd "$FPCDIR" svn co https://svn.freepascal.org/svn/fpc/tags/$FPCTAG/ $FPCVER

# Get sources
git clone https://gitlab.com/freepascal.org/fpc/source.git $FPCVER # switch to branch/tag
cd $FPCVER
git checkout $FPCTAG
cd ..

echo "Downloading version $LAZVER of Lazarus" cd "$LAZDIR" svn co https://svn.freepascal.org/svn/lazarus/tags/$LAZTAG/ $LAZVER

# Get sources
git clone https://gitlab.com/freepascal.org/lazarus/lazarus.git $LAZVER # switch to branch/tag
cd $LAZVER
git checkout $LAZTAG
cd ..

Michael.

Martin Frb
to Bo Berglund: I have for many years been using an install script I wrote to install a complete…
Bo Berglund wrote
Please advice how to: 1) Find which is the latest release tag of fpc and lazarus

Well I don't know your script, and I am not sure how exactly you do that in svn....

Also I do not know, if this is for a once up setup (so you only wont to download the one revision and nothing else), or if you don't mind. (I have not measured the overhead..>)

Most replies will be to clone all, and then look at tags. (or branches) But you can do ( -t => tags) https://git-scm.com/docs/git-ls-remote.html

    git ls-remote -t https://gitlab.com/freepascal.org/lazarus/lazarus.git

However, there is no "svn serial rev number", nor a date. So you need to parse the "release version" number, and compare them.

If you have a complete clone, you can afaik sort them by "commit date" (see "git for-each-ref")

2) Retrieve the full sources for that tag on the command line from gitlab

If you do not want to clone

git init .
git fetch --depth=1
https://gitlab.com/freepascal.org/lazarus/lazarus.git refs/tags/lazarus_2_2_0_RC1
git switch -c my-branch FETCH_HEAD

"my-branch" is your choice

*** the rest BELOW is completely UNTESTED / pure assumptions ***

I haven't tested how well updates work, if you go to other branches. Since you have no history fetched (depth =1), I expect a full download of any other version (unless it is a direct child / same branch, NOT diverged)

Git is written for getting the full history once. I *guess* (but might be totally wrong) that not doing so, will make you pay later. You could do a depth of 3000 or 10000.
That is 2.2.0 (with the ending 0) is likely less that 3000 commits divirged from main. Then that gives you common history, and 2.4 can base on that.

2.0.18 is probably more revisions from main.

If you only want to stay in the same fixes branch (and are ok with a new download for major releases) go with a depth of 500. It is unlikely that a release tag will be further away from its fixes branch (usual are 0 to 5 commits)

Bo Berglund
to Michael Van Canneyt: # Get sources git clone https://gitlab.com/freepascal.org/fpc/source.git $FPCVER #…
Michael Van Canneyt wrote 6 lines
# Get sources git clone https://gitlab.com/freepascal.org/fpc/source.git $FPCVER # switch to branch/tag cd $FPCVER git checkout $FPCTAG cd ..

How big a size is the git clone going to pull?

By the looks of it I get the impression that the git clone will download every single tag/branch/commit etc ever done and then the git checkout activates the actual tagged file set?
If this is correct then the size on disk will be too big.

Now as I said I have not used git and for my day use cannot switch because we use properties of svn that are simply not available in git...

I *really* do not want to revert to getting fpc/lazarus from the distro manager via apt and therefore be stuck with old code decided by someone else.

Bo Berglund Developer in Sweden
Unknown author
to Dmitry Boyarintsev: why not to use Github mirror? svn co https://github.com/fpc/FPCSource/trunk or svn co…

j have a problem with 3.2.2  write and writeln  halts the program

Earlier message 17 lines
-----Original Message----- From: Dmitry Boyarintsev via fpc-pascal To: FPC-Pascal users discussions Cc: Dmitry Boyarintsev ; Bo Berglund Sent: Wed, Aug 11, 2021 11:57 pm Subject: Re: [fpc-pascal] How to get fpc and lazarus sources when svn has shut down? why not to use Github mirror? svn co https://github.com/fpc/FPCSource/trunk  or svn co https://github.com/fpc/FPCSource/tags/$FPCTAG/  thanks, Dmitry_______________________________________________ fpc-pascal maillist  -  https://lists.freepascal.org/cgi-bin/mailman/listinfo/fpc-pascal
Dmitry Boyarintsev
to Michael Van Canneyt: Why not encourage people to go with the times ? Git is not some witchcraft or dark magic.…

On Wed, Aug 11, 2021 at 5:37 PM Michael Van Canneyt via fpc-pascal <

Show quoted text 5 lines
wrote: > why not to use Github mirror? Why not encourage people to go with the times ?

it might depend on the use of the version control. Maybe there's a certain automation involved behind the scenes. If it used to be configured for SVN then the only allowed kind of change is the URL address.

As a programmer, switching version systems should be a no-brainer.

amen

thanks,
Dmitry

Michael Van Canneyt
to Bart: On Wed, Aug 11, 2021 at 11:37 PM Michael Van Canneyt via fpc-pascal My brain begs to…
Bart wrote 6 lines
On Wed, Aug 11, 2021 at 11:37 PM Michael Van Canneyt via fpc-pascal wrote: > As a programmer, switching version systems should be a no-brainer. My brain begs to differ...

Well, sorry to say, but I don't understand this.

What's so hard about it ?

It's like switching car brands from a VW to an Audi or so. Some buttons are in different places, your key will maybe look different, but that's it. It's a car, it brings you from a to b.

A file version system manages versions of files.
The actual commands differ a little, but that's it.

Michael.

Dmitry Boyarintsev
to Michael Van Canneyt

On Thu, Aug 12, 2021 at 12:27 AM Michael Van Canneyt via fpc-pascal <

Show quoted text 6 lines
wrote: It's like switching car brands from a VW to an Audi or so. Some buttons are in different places, your key will maybe look different, but that's it. It's a car, it brings you from a to b.

It's not switching brands, it's switching the type of the car. from a passenger i.e. to a bus.
It's still a car, but the concept is a little different.

A file version system manages versions of files. The actual commands differ a little, but that's it.

The human psychology. There were no particular problems for anyone to use SVN.
It worked fine. So for the people it doesn't seem like an obvious reason for the change.
The change is not recognized by the brain as a needed change, as a cure of some sort.
Instead it's recognized as an unnecessary burden.
(unlike switching from CVS to SVN)

The same is happening with don't recognize the new knowledge as something useful.
Instead they still treat the school as a burden.
---
In the software we for some reason prefer to stick to "backwards compatibility"
No matter how much it affects a project. (it's considered that retaining backwards compatibility is a positive effect).
However the same concept doesn't apply to the infrastructure for some reason.

thanks,
Dmitry

LacaK
to Dmitry Boyarintsev
Show quoted text 21 lines
It's like switching car brands from a VW to an Audi or so. Some buttons are in different places, your key will maybe look different, but that's it. It's a car, it brings you from a to b. It's not switching brands, it's switching the type of the car. from a passenger i.e. to a bus. It's still a car, but the concept is a little different. A file version system manages versions of files. The actual commands differ a little, but that's it. The human psychology. There were no particular problems for anyone to use SVN. It worked fine. So for the people it doesn't seem like an obvious reason for the change. The change is not recognized by the brain as a needed change, as a cure of some sort. Instead it's recognized as an unnecessary burden.

I must say that I have similar feelings.
No matter ...
I have related question: in SVN was possible to checkout specific sub-directory (for example if I am interested in fcl-db package only I checkedout only this sub-directory).
Is it possible with gitlab? Or I must clone whole
https://gitlab.com/freepascal.org/fpc/source ?
Thanks ...
-Laco.

Dmitry Boyarintsev
to LacaK

On Thu, Aug 12, 2021 at 1:34 AM LacaK via fpc-pascal <

Show quoted text 8 lines
wrote: I have related question: in SVN was possible to checkout specific sub-directory (for example if I am interested in fcl-db package only I checkedout only this sub-directory). Is it possible with gitlab? Or I must clone whole https://gitlab.com/freepascal.org/fpc/source ?

No. This is not possible with Git.
You either have to deal with the entire project structure or you could use an SVN mirror (i.e. github based)

thanks,
Dmitry

Dmitry Boyarintsev
to Bo Berglund: I have for many years been using an install script I wrote to install a complete…

On Thu, Aug 12, 2021 at 2:00 AM Bo Berglund via fpc-pascal <

Show quoted text 5 lines
wrote: What could I replace these svn co (or svn export) calls with?

the current:
svn co https://svn.freepascal.org/svn/fpc/tags/$FPCTAG/ $FPCVER turns into:
svn co https://github.com/fpc/FPCSource/tags/$FPCTAG/$FPCVER

If you open https://github.com/fpc/ you can find all the mirrors are there. And this is the official Github mirror, so all the latest changes should be reflected there pretty quickly.

thanks,
Dmitry

Christo Crause
to Bo Berglund: On Wed, 11 Aug 2021 22:22:54 +0200 (CEST), Michael Van Canneyt via fpc-pascal How big a…

On Thu, Aug 12, 2021 at 12:50 AM Bo Berglund via fpc-pascal <

Show quoted text 26 lines
wrote: On Wed, 11 Aug 2021 22:22:54 +0200 (CEST), Michael Van Canneyt via fpc-pascal wrote: ># Get sources >git clone https://gitlab.com/freepascal.org/fpc/source.git $FPCVER ># switch to branch/tag >cd $FPCVER >git checkout $FPCTAG >cd .. How big a size is the git clone going to pull? By the looks of it I get the impression that the git clone will download every single tag/branch/commit etc ever done and then the git checkout activates the actual tagged file set? If this is correct then the size on disk will be too big. Now as I said I have not used git and for my day use cannot switch because we use properties of svn that are simply not available in git...

To retrieve a specific branch without history (I think this is approximately equivalent to the svn co command) specify options --branch (to specify which branch to check out), --single-branch (to not pull in other branches) and --depth (to limit the history that git pulls in):

git clone --depth=1 --branch fixes_3_2 --single-branch https://gitlab.com/freepascal.org/fpc/source.git fpc-fixes_3_2

This example downloads about 46 MiB of data, which expands to around 309 MiB on disk, of which 50 MiB is git information.

Michael Van Canneyt
to Dmitry Boyarintsev: On Thu, Aug 12, 2021 at 12:27 AM Michael Van Canneyt via fpc-pascal < It's not switching…
Dmitry Boyarintsev wrote 32 lines
On Thu, Aug 12, 2021 at 12:27 AM Michael Van Canneyt via fpc-pascal < > wrote: > It's like switching car brands from a VW to an Audi or so. > Some buttons are in different places, your key will maybe look different, > but that's it. It's a car, it brings you from a to b. > It's not switching brands, it's switching the type of the car. from a passenger i.e. to a bus. It's still a car, but the concept is a little different. > A file version system manages versions of files. > The actual commands differ a little, but that's it. > The human psychology. There were no particular problems for anyone to use SVN. It worked fine. So for the people it doesn't seem like an obvious reason for the change. The change is not recognized by the brain as a needed change, as a cure of some sort. Instead it's recognized as an unnecessary burden. (unlike switching from CVS to SVN) The same is happening with don't recognize the new knowledge as something useful. Instead they still treat the school as a burden. --- In the software we for some reason prefer to stick to "backwards compatibility" No matter how much it affects a project. (it's considered that retaining backwards compatibility is a positive effect).

Yes, because you break other people's code if you don't.

However the same concept doesn't apply to the infrastructure for some reason.

No, because the consequences are entirely different:

Changing infrastructure is normally only affecting the team working with it. If you change it, the effect is only for your team. The final product is not affected:

For users that download the installer of FPC or Lazarus, it doesn't matter whether we use git or SVN.

I can understand and up to a point relate to resistance against change, but when I speak of 'no-brainer' I simply mean that it's not difficult to learn or master another version system.

In the particular case of git, it's not like we switched to some obscure unknown system, totally undocumented... The internet overflows with info about git.

Michael.

Michael Van Canneyt
to Christo Crause: On Thu, Aug 12, 2021 at 12:50 AM Bo Berglund via fpc-pascal < To retrieve a specific…
Christo Crause wrote 15 lines
> Now as I said I have not used git and for my day use cannot switch because > we > use properties of svn that are simply not available in git... > To retrieve a specific branch without history (I think this is approximately equivalent to the svn co command) specify options --branch (to specify which branch to check out), --single-branch (to not pull in other branches) and --depth (to limit the history that git pulls in): git clone --depth=1 --branch fixes_3_2 --single-branch https://gitlab.com/freepascal.org/fpc/source.git fpc-fixes_3_2 This example downloads about 46 MiB of data, which expands to around 309 MiB on disk, of which 50 MiB is git information.

Nice, I didn't even know this one. We learn every day !

Thanks.

Michael.

Martin Frb
to Michael Van Canneyt: Well, sorry to say, but I don't understand this. What's so hard about it ? It's like…
Michael Van Canneyt wrote 11 lines
Well, sorry to say, but I don't understand this. What's so hard about it ? It's like switching car brands from a VW to an Audi or so. Some buttons are in different places, your key will maybe look different, but that's it. It's a car, it brings you from a to b. A file version system manages versions of files. The actual commands differ a little, but that's it.

That comparison is a bit to simple....
And the extend has also to be taken into account.

Lets say you switch from a European to a British car. In theory it is simple. The amount of time you try to find the hand/emergency break or gear stick and your hand goes down along the inside of the car's door.....

If all you need to do is use git for download, then well it is simple in terms, that indeed you just need a "translation".
If you have size restrictions, and can't do a full clone, you need a special translation (which you have to find, somewhere in a large manual). Still if you have time, that is indeed simple. "simple, but one off time consuming"

If you need to commit to, then its yet a bit more time consuming. And even if you read up, you will end up with a few surprises. Like you may end up making a commit while detached. Or resetting your local tracking branch onto another remote branch.
It's all recoverable.
And for a lot of people at some point it may be worth it, because they have new knowledge. Be that only to be put on the CV, or be it that they actually benefit from it.

If you need to learn a platform like GitLab with it. Well, we have already examples for some pitfalls that it does have. Example: In a merge-request, which the requestor rebased after making it (and before it got rebased) => there is a link, that shows the difference of the rebase. There is not much of an explanation there. It looks as if those commit between the 2 forkpoints (wrong name) are part of the merge request.

So every new set of features does add new pitfalls.

Still IMHO its more than worth it.

Martin Frb
to Dmitry Boyarintsev: On Thu, Aug 12, 2021 at 12:27 AM Michael Van Canneyt via fpc-pascal < It's not switching…
Dmitry Boyarintsev wrote 13 lines
On Thu, Aug 12, 2021 at 12:27 AM Michael Van Canneyt via fpc-pascal < > wrote: It's like switching car brands from a VW to an Audi or so. Some buttons are in different places, your key will maybe look different, but that's it. It's a car, it brings you from a to b. It's not switching brands, it's switching the type of the car. from a passenger i.e. to a bus. It's still a car, but the concept is a little different.

Ah, that is true for selected individuals, if looking at their own plate only...

There were issues for those who wanted to switch.

- the need to maintain the server on which it was hosted / to update server software.
- the absence of the feature the knew exist / and they knew would help them.

For those that "had to come along", well it's part of the group experience. Many of them will probably at some point pick up some of the new features, and then they can look back and be glad to have learned it. (just like the pupil in later live might come to benefit from what they learned in school).

For those who will not have the luck of benefiting themself ever, well they came along for the "greater good".

Martin Frb
to Bo Berglund: I have for many years been using an install script I wrote to install a complete…
Bo Berglund wrote
Please advice how to: 1) Find which is the latest release tag of fpc and lazarus

Well I don't know your script, and I am not sure how exactly you do that in svn....

Also I do not know, if this is for a once up setup (so you only wont to download the one revision and nothing else), or if you don't mind. (I have not measured the overhead..>)

Most replies will be to clone all, and then look at tags. (or branches) But you can do ( -t => tags) https://git-scm.com/docs/git-ls-remote.html

    git ls-remote -t https://gitlab.com/freepascal.org/lazarus/lazarus.git

However, there is no "svn serial rev number", nor a date. So you need to parse the "release version" number, and compare them.

If you have a complete clone, you can afaik sort them by "commit date" (see "git for-each-ref")

2) Retrieve the full sources for that tag on the command line from gitlab

If you do not want to clone

git init .
git fetch --depth=1
https://gitlab.com/freepascal.org/lazarus/lazarus.git refs/tags/lazarus_2_2_0_RC1
git switch -c my-branch FETCH_HEAD

"my-branch" is your choice

*** the rest BELOW is completely UNTESTED / pure assumptions ***

I haven't tested how well updates work, if you go to other branches. Since you have no history fetched (depth =1), I expect a full download of any other version (unless it is a direct child / same branch, NOT diverged)

Git is written for getting the full history once. I *guess* (but might be totally wrong) that not doing so, will make you pay later. You could do a depth of 3000 or 10000.
That is 2.2.0 (with the ending 0) is likely less that 3000 commits divirged from main. Then that gives you common history, and 2.4 can base on that.

2.0.18 is probably more revisions from main.

If you only want to stay in the same fixes branch (and are ok with a new download for major releases) go with a depth of 500. It is unlikely that a release tag will be further away from its fixes branch (usual are 0 to 5 commits)

Martin Frb
to Michael Van Canneyt: Well, sorry to say, but I don't understand this. What's so hard about it ? It's like…
Michael Van Canneyt wrote 11 lines
Well, sorry to say, but I don't understand this. What's so hard about it ? It's like switching car brands from a VW to an Audi or so. Some buttons are in different places, your key will maybe look different, but that's it. It's a car, it brings you from a to b. A file version system manages versions of files. The actual commands differ a little, but that's it.

That comparison is a bit to simple....
And the extend has also to be taken into account.

Lets say you switch from a European to a British car. In theory it is simple. The amount of time you try to find the hand/emergency break or gear stick and your hand goes down along the inside of the car's door.....

If all you need to do is use git for download, then well it is simple in terms, that indeed you just need a "translation".
If you have size restrictions, and can't do a full clone, you need a special translation (which you have to find, somewhere in a large manual). Still if you have time, that is indeed simple. "simple, but one off time consuming"

If you need to commit to, then its yet a bit more time consuming. And even if you read up, you will end up with a few surprises. Like you may end up making a commit while detached. Or resetting your local tracking branch onto another remote branch.
It's all recoverable.
And for a lot of people at some point it may be worth it, because they have new knowledge. Be that only to be put on the CV, or be it that they actually benefit from it.

If you need to learn a platform like GitLab with it. Well, we have already examples for some pitfalls that it does have. Example: In a merge-request, which the requestor rebased after making it (and before it got rebased) => there is a link, that shows the difference of the rebase. There is not much of an explanation there. It looks as if those commit between the 2 forkpoints (wrong name) are part of the merge request.

So every new set of features does add new pitfalls.

Still IMHO its more than worth it.

Michael Van Canneyt
Dennis Lee Bieber wrote 30 lines
On Wed, 11 Aug 2021 23:55:57 +0200, Bo Berglund via fpc-pascal declaimed the following: > How big a size is the git clone going to pull? > > By the looks of it I get the impression that the git clone will download every > single tag/branch/commit etc ever done and then the git checkout activates the > actual tagged file set? > If this is correct then the size on disk will be too big. > GIT and Mercurial are both "distributed" version control systems. That means that both rely upon creating a local repository from which one performs check-out/check-in operations, and periodically "pushes" the changes to the remote server repository. As I understand https://git-scm.com/docs/git-clone you get the "history" of branches/et al, but only get the most recent source for controlled files: """ Clones a repository into a newly created directory, creates remote-tracking branches for each branch in the cloned repository (visible using git branch --remotes), and creates and checks out an initial branch that is forked from the cloned repository’s currently active branch. """ But yes, you will have two copies of files if you check-out everything: the local repository, and the working file fetched from said repository.

The same was true for SVN, you always have the original and current version of every file.

Michael.

Bart
to Michael Van Canneyt: Well, sorry to say, but I don't understand this. What's so hard about it ? It's like…
AM Michael Van Canneyt wrote 9 lines
>> As a programmer, switching version systems should be a no-brainer. > > My brain begs to differ... Well, sorry to say, but I don't understand this. What's so hard about it ? It's like switching car brands from a VW to an Audi or so.

To me it feels more like switching from Pascal to C#. But maybe I'm just a little braindead.

Bart
Christo Crause
to LacaK: I must say that I have similar feelings. No matter ... I have related question: in SVN…

On Thu, Aug 12, 2021 at 7:34 AM LacaK via fpc-pascal <

Show quoted text 9 lines
wrote: I have related question: in SVN was possible to checkout specific sub-directory (for example if I am interested in fcl-db package only I checkedout only this sub-directory). Is it possible with gitlab? Or I must clone whole https://gitlab.com/freepascal.org/fpc/source ?

There is a relatively new git feature "sparse-checkout" that could potentially be used in this case (I haven't tested this). See e.g. https://stackoverflow.com/a/60729017/13781629

Martin Frb
Dennis Lee Bieber wrote 10 lines
And... the local repository will add files for each committed change... https://www.dulwich.io/docs/tutorial/file-format.html """ If you change a single line, another blob will be generated by Git each time you successfully run git add. This is how Git can fastly checkout any version in time. """

1) Afaik the OT is looking at only checking out. He isn't planing on making any (local) commits.

2) yes, for every changed file you commit, a new blob is added (and new tree objects too).
From a users point of view, each blob contains the entire file (changed and unchanged parts).
But git compresses those blobs.
https://git-scm.com/docs/git-repack#Documentation/git-repack.txt---windowltngt So in the end, the disk space used is not that bad. Still of course, it is significantly more, if you do a full clone, rather than just a shallow (depth=1).

I don't know how big your ".svn" folder is, if you checkout a current fpc/lazarus.
But as it was pointed out, cloning with depth=1 gives you about 50 MB of compressed data, for files worth over 300MB.

Martin Frb
to Christo Crause: On Thu, Aug 12, 2021 at 7:34 AM LacaK via fpc-pascal < There is a relatively new git…
Christo Crause wrote 19 lines
On Thu, Aug 12, 2021 at 7:34 AM LacaK via fpc-pascal < > wrote: I have related question: in SVN was possible to checkout specific sub-directory (for example if I am interested in fcl-db package only I checkedout only this sub-directory). Is it possible with gitlab? Or I must clone whole https://gitlab.com/freepascal.org/fpc/source <https://gitlab.com/freepascal.org/fpc/source> ? There is a relatively new git feature "sparse-checkout" that could potentially be used in this case (I haven't tested this).  See e.g. https://stackoverflow.com/a/60729017/13781629 <https://stackoverflow.com/a/60729017/13781629>

Afaik that is only checkout.
You still clone the entire set of files.

Michael Van Canneyt
to Christo Crause: On Thu, Aug 12, 2021 at 7:34 AM LacaK via fpc-pascal < There is a relatively new git…
Christo Crause wrote 14 lines
On Thu, Aug 12, 2021 at 7:34 AM LacaK via fpc-pascal < > wrote: > > I have related question: in SVN was possible to checkout specific > sub-directory (for example if I am interested in fcl-db package only I > checkedout only this sub-directory). > Is it possible with gitlab? Or I must clone whole > https://gitlab.com/freepascal.org/fpc/source ? > There is a relatively new git feature "sparse-checkout" that could potentially be used in this case (I haven't tested this). See e.g. https://stackoverflow.com/a/60729017/13781629

There is a good explanation here:

https://github.blog/2020-01-17-bring-your-monorepo-down-to-size-with-sparse-checkout/

Michael.

Unknown author
to Michael Van Canneyt: # Get sources git clone https://gitlab.com/freepascal.org/fpc/source.git $FPCVER #…
Michael Van Canneyt wrote 11 lines
On Wed, 11 Aug 2021, Bo Berglund via fpc-pascal wrote: > echo "Downloading version $FPCVER of FPC" > cd "$FPCDIR" > svn co https://svn.freepascal.org/svn/fpc/tags/$FPCTAG/ $FPCVER # Get sources git clone https://gitlab.com/freepascal.org/fpc/source.git  $FPCVER # switch to branch/tag cd $FPCVER git checkout $FPCTAG cd ..

FWIW: remember that "clone" is _only for the first time_, too... after that you would "fetch" and/or "pull" to bring in the changes...

to clarify:
"fetch" tells your local git to retrieve the latest meta-data info from the original repo but it doesn’t do any file transfers...

"pull" tells your local git to retrieve the latest meta-data info and to also do the actual transfers and apply them to your clone...

i use a bash script that contains the following routine which does the main work of *updating* my local repos... selection of the repo and changing into its directory are done outside this fairly generic routine...

----->8 snip 8<-----
function _gitUpdate(){

   if [ "$DOWNLOAD" != "y" ]; then
     return
   fi

local branch="$1"
set +e
git diff --exit-code 2>&1 > /dev/null

   if [[ $? != 1 ]]; then
     set -e
     git fetch origin
     git checkout --force "$branch"
     git pull --rebase
   else
     set -e
     git fetch origin
     git stash save --include-untracked --quiet
     git checkout --force "$branch"
     git pull --rebase
     git stash pop --quiet
   fi

}
----->8 snip 8<-----

this routine checks if there have been upstream updates... if you have any local changes, they will be stashed first, then the upstream updates will be pulled after which your stash will be applied so your local changes are available... if you have no local changes, then no stashing and popping are involved...

if there is a conflict when popping, it requires fairly standard conflict resolution and cleanup... i'll leave that to the reader to follow up and understand...

hopefully this helps and doesn't add to the confusion...

NOTE: No off-list assistance is given without prior approval. *Please keep mailing list traffic on the list where it belongs!*
Bo Berglund
to Martin Frb: 1) Afaik the OT is looking at only checking out. He isn't planing on making any (local)…
Martin Frb wrote
1) Afaik the OT is looking at only checking out. He isn't planing on making any (local) commits.

Exactly, I could have used svn export instead of svn co in my script and it would have worked fine. But at the time I was totally geared towards co...

What I am after is a way to retrieve the sources inside a script so no webpages to click buttons on etc, just a simple command to actually get the sources themselves is what is requested.

I am installing on multiple Raspberry Pi devices and it simplifies a lot to use the script.

Bo Berglund Developer in Sweden
Michael Van Canneyt
to Bo Berglund
Bo Berglund wrote 12 lines
On Thu, 12 Aug 2021 11:02:50 +0200, Martin Frb via fpc-pascal wrote: > 1) Afaik the OT is looking at only checking out. He isn't planing on > making any (local) commits. Exactly, I could have used svn export instead of svn co in my script and it would have worked fine. But at the time I was totally geared towards co... What I am after is a way to retrieve the sources inside a script so no webpages to click buttons on etc, just a simple command to actually get the sources themselves is what is requested.

Well, all mails so far gave you this info ?

The only thing what is not clear from your question is whether this script will also update sources, or whether the sources are cloned only once.

If you do an update of an existing directory, instead of

git clone http://theurl/ thesourcedirectory

you must do

cd thesourcedirectory
git pull
cd ..

This will reduce the download size considerably.

Michael.

Bo Berglund
to Martin Frb: Well I don't know your script, and I am not sure how exactly you do that in svn.... Also…
Martin Frb wrote
Well I don't know your script, and I am not sure how exactly you do that in svn....

It is not done in SVN, instead I visit the (now removed) page https://svn.freepascal.org/svn/lazarus/ (for lazarus) or
https://svn.freepascal.org/svn/fpc/ (for fpc)

and look at the tags list to find the most recent release tag. I then use that as input to my script.

Also I do not know, if this is for a once up setup (so you only wont to download the one revision and nothing else), or if you don't mind. (I have not measured the overhead..>)

It is for installation on a pristine RaspberryPi (or Ubuntu) machine and the script does everything including setting up the folder tree for development and adding the desktop shortcuts.
Of corse now it is unusable because it cannot retrieve the sources... :(

I want to download the complete source tree for the tagged revision I want to install. I do not care if it is with metadata or not, I only ever use the files as is and never modify them.
Of course I do not want to get excessive overhead like the complete history of all files for eternity...
No history at all would be fine.

But retrieval via a bash script in unattended operation (because it normally takes upwards of 20 min to install on an RPi) is what I want.

Bo Berglund Developer in Sweden
Martin Frb
to Michael Van Canneyt: Well, all mails so far gave you this info ? The only thing what is not clear from your…
Michael Van Canneyt wrote 12 lines
If you do an update of an existing directory, instead of git clone http://theurl/ thesourcedirectory you must do cd thesourcedirectory git pull cd .. This will reduce the download size considerably.

If he stays on the same branch (assuming he is on a branch). Or if his "--depth" was deep enough to have common history with the new target branch.

An svn server (afaik) calculates the diff between any 2 revisions, even on diff branches...
But (afaik, really not sure) git will look for the nearest common history, and download all changes from that point.

This makes sense for git, because "--depth" is the exception, and not what the design is made for. So git would assume you want the commits in the middle, and it would assume you have common history. (But again, not tested at all).

And if you "cloned" only a "tag" (depth 1), not a branch => then you may not have an upstream branch. So you can't "git pull".

Martin Frb
to Bo Berglund: On Wed, 11 Aug 2021 23:13:28 +0200, Martin Frb via fpc-pascal It is not done in SVN,…
Bo Berglund wrote 12 lines
On Wed, 11 Aug 2021 23:13:28 +0200, Martin Frb via fpc-pascal wrote: > Well I don't know your script, and I am not sure how exactly you do that > in svn.... It is not done in SVN, instead I visit the (now removed) page https://svn.freepascal.org/svn/lazarus/ (for lazarus) or https://svn.freepascal.org/svn/fpc/ (for fpc) and look at the tags list to find the most recent release tag. I then use that as input to my script.

ok, so you could run
  git ls-remote
in your shell, and look at that.

Show quoted text 12 lines
> Also I do not know, if this is for a once up setup (so you only wont to > download the one revision and nothing else), or if you don't mind. > (I have not measured the overhead..>) It is for installation on a pristine RaspberryPi (or Ubuntu) machine and the script does everything including setting up the folder tree for development and adding the desktop shortcuts. Of corse now it is unusable because it cannot retrieve the sources... :( I want to download the complete source tree for the tagged revision I want to install. I do not care if it is with metadata or not, I only ever use the files as is and never modify them.

Well

Christo Crause wrote
git clone --depth=1 --branch fixes_3_2 --single-branch https://gitlab.com/freepascal.org/fpc/source.git <https://gitlab.com/freepascal.org/fpc/source.git> fpc-fixes_3_2

You need to test, if it works with tags too (my 3 step solution did).

Of course I do not want to get excessive overhead like the complete history of all files for eternity... No history at all would be fine.

depth=1

Only when you update, then you do get history.

with my "fetch" solution

git init .
git fetch --depth=1
https://gitlab.com/freepascal.org/lazarus/lazarus.git refs/tags/lazarus_2_2_0_RC1
git switch -c my-branch FETCH_HEAD

you can then later do (may download entire 2.2.0, NOT just the diffs / if you are lucky it will reuse some of your local existing data / I don't know the internals well enough)

git fetch --depth=1
https://gitlab.com/freepascal.org/lazarus/lazarus.git refs/tags/lazarus_2_2_0
git switch -c my-NEW-branch FETCH_HEAD
git branch -d my-branch
git reflog expire --expire=now --all
git gc --prune=now --aggressive

And then you should have just the new branch, and all old data gone. But you temp have both.
(git switch takes -f to overwrite changed file / there also is a merge option instead)

Afaik You only can get rid of the old data, once you switched to the new. So for a brief moment you do have both.

Or you delete the entire dir, and start from scratch.

Michael Van Canneyt
to Martin Frb: If he stays on the same branch (assuming he is on a branch). Or if his "--depth" was deep…
Martin Frb wrote 29 lines
On 12/08/2021 15:04, Michael Van Canneyt via fpc-pascal wrote: > > If you do an update of an existing directory, instead of > > git clone http://theurl/ thesourcedirectory > > you must do > > cd thesourcedirectory > git pull > cd .. > > This will reduce the download size considerably. If he stays on the same branch (assuming he is on a branch). Or if his "--depth" was deep enough to have common history with the new target branch. An svn server (afaik) calculates the diff between any 2 revisions, even on diff branches... But (afaik, really not sure) git will look for the nearest common history, and download all changes from that point. This makes sense for git, because "--depth" is the exception, and not what the design is made for. So git would assume you want the commits in the middle, and it would assume you have common history. (But again, not tested at all). And if you "cloned" only a "tag" (depth 1), not a branch => then you may not have an upstream branch. So you can't "git pull".

If you look at the top of the mail, you'll see I assumed he did a full clone initially.

Less fuss.

Michael.

Bo Berglund
to Dmitry Boyarintsev: On Thu, Aug 12, 2021 at 2:00 AM Bo Berglund via fpc-pascal < the current: svn co…
Dmitry Boyarintsev wrote 10 lines
On Thu, Aug 12, 2021 at 2:00 AM Bo Berglund via fpc-pascal < > wrote: > > What could I replace these svn co (or svn export) calls with? > the current: svn co https://svn.freepascal.org/svn/fpc/tags/$FPCTAG/ $FPCVER turns into: svn co https://github.com/fpc/FPCSource/tags/$FPCTAG/$FPCVER
If you open https://github.com/fpc/ you can find all the mirrors are there. And this is the official Github mirror, so all the latest changes should be reflected there pretty quickly.

Thanks,
that would be a good solution by not involving git at all...

I will change my links in the script for fpc 3.2.2 and lazarus 2.0.12 like this:

svn co https://github.com/fpc/FPCSource/tags/release_3_2_2 3.2.2

svn co https://github.com/fpc/Lazarus/tags/lazarus_2_0_12 2.0.12

Tested these on an RPi4 with 4GB RAM on a 24 MBps mobile broadband connection:

checkout times:
fpc: 12m 35s
lazarus: 6m 37s

Checkout size:
fpc: 697 Mb, 352 Mb in .svn/
lazarus: 392 Mb, 192 Mb in .svn/

Also tried svn export rather than svn co:

svn export https://github.com/fpc/FPCSource/tags/release_3_2_2 3.2.2x

svn export https://github.com/fpc/Lazarus/tags/lazarus_2_0_12 2.0.12x

checkout times:
fpc: 4m 10s
lazarus: 1m 09s

Checkout size:
fpc: 345 Mb
lazarus: 199 Mb

Not verified by building fpc and lazarus on this machine yet. But it is clear that svn export is what should be in the script anyway to save both time and disk space...

Thanks again, this was what I needed to know! :)

Bo Berglund Developer in Sweden
Marco van de Voort
to Christo Crause: On Thu, Aug 12, 2021 at 12:50 AM Bo Berglund via fpc-pascal < To retrieve a specific…

Op 2021-08-12 om 08:12 schreef Christo Crause via fpc-pascal:

This example downloads about 46 MiB of data, which expands to around 309 MiB on disk, of which 50 MiB is git information.

 Note that a full clone is about 500MB. Still, it is a nice to have for e.g. machines that only do release packaging, and the only operation done on the VCS is updating and exporting.

There are many problems with GIT, but size is not one of them.

Vojtěch Čihák
to Bo Berglund: On Wed, 11 Aug 2021 22:22:54 +0200 (CEST), Michael Van Canneyt via fpc-pascal How big a…

Hi,
 
my numbers of Lazarus on Linux, Qt (incl. binaries and *.ppu and *.o files)  
SVN:
lazarus dir: 2.6 GB (40000 files, 1300 dirs)
.svn subdir: 1.6 GB (25000 files, 260 dirs)
 
GIT:
lazarus dir: 1 GB (15000 files, 950 dirs)
.git subdir: 150 MB (60 files, 40 dirs)
 
V.
 
______________________________________________________________

Show quoted text 5 lines
Od: "Bo Berglund via fpc-pascal" Komu: Datum: 11.08.2021 23:56 Předmět: Re: [fpc-pascal] How to get fpc and lazarus sources when svn has shut down?
Michael Van Canneyt wrote 6 lines
# Get sources git clone https://gitlab.com/freepascal.org/fpc/source.git <https://gitlab.com/freepascal.org/fpc/source.git>  $FPCVER # switch to branch/tag cd $FPCVER git checkout $FPCTAG cd ..

How big a size is the git clone going to pull?

By the looks of it I get the impression that the git clone will download every single tag/branch/commit etc ever done and then the git checkout activates the actual tagged file set?
If this is correct then the size on disk will be too big.

...

Bo Berglund Developer in Sweden
Martin Frb
to Vojtěch Čihák
Vojtěch Čihák wrote 19 lines
Hi, my numbers of Lazarus on Linux, Qt (incl. binaries and *.ppu and *.o files) SVN: lazarus dir: 2.6 GB (40000 files, 1300 dirs) .svn subdir: 1.6 GB (25000 files, 260 dirs) GIT: lazarus dir: 1 GB (15000 files, 950 dirs) .git subdir: 150 MB (60 files, 40 dirs)

The size of the .svn / .git dir will vary, dependent how often you "clean" it.

That is it will grow, when you work with the repro (svn update / git pull or commit ....)

So you can compare those either
1) immediately after your first checkout/clone

2) after cleaning
IIRC
in svn "svn cleanup  --vacuum-pristines " not sure.... in git "git gc" (or git gc --aggressive)

IIRC, svn will end up a little bit smaller. But you lose that as soon as you stop running maintenance.

Not to say if you need to often switch between trunk and fixes..... (In git, I have 2 work dirs, one for main, one for fixes => but only ONE .git dir => so that saves space)

Kevin Lyda
to Bo Berglund: On Wed, 11 Aug 2021 22:22:54 +0200 (CEST), Michael Van Canneyt via fpc-pascal How big a…
PM Bo Berglund wrote
How big a size is the git clone going to pull?

: ok;: Fri 13 9:13; cd /tmp;:
: on pts/7; time git clone
https://gitlab.com/freepascal.org/fpc/source.git fpc.git > fpc.git.out 2>&1
git clone https://gitlab.com/freepascal.org/fpc/source.git fpc.git > 2>&1 98.46s user 3.52s system 199% cpu 51.065 total : ok;: Fri 13 9:14; cd /tmp;:
: on pts/7; time svn co
https://github.com/fpc/FPCSource/trunk fpc.svn > fpc.svn.out 2>&1 svn co https://github.com/fpc/FPCSource/trunk fpc.svn > fpc.svn.out 2>&1 9.27s user 7.62s system 48% cpu 34.517 total : ok;: Fri 13 9:15; cd /tmp;:
: on pts/7; du -sh fpc.*
551M fpc.git
4.0K fpc.git.out
739M fpc.svn
972K fpc.svn.out

By the looks of it I get the impression that the git clone will download every single tag/branch/commit etc ever done and then the git checkout activates the actual tagged file set? If this is correct then the size on disk will be too big.

Full git repositories are normally smaller than svn working copies - and that's the
case here. By using git you'll be saving a bit under 200M. If you're curious why,
it boils down to git storing the repo very efficiently and svn needing two copies
of every file in a working copy.

In addition if you're really concerned about disk space you can always go into the git
repo, change to the branch you want and then delete the .git directory. But if you
were keeping the .svn directory I don't see the point since...

: ok;: Fri 13 9:17; cd /tmp;:
: on pts/7; du -sh fpc.*/.{svn,git}
373M fpc.svn/.svn
185M fpc.git/.git

Now as I said I have not used git and for my day use cannot switch because we use properties of svn that are simply not available in git...

People think that and in my experience they are usually wrong. You have opinions.
And that's fine. Enjoy them. As noted above, a checked out git repo is smaller. Subsequent fetches will be far faster. In this case, as the data above shows, your
workflow will improve.

Kevin

Bo Berglund
Dennis Lee Bieber wrote 15 lines
On Thu, 12 Aug 2021 16:39:56 +0200, Bo Berglund via fpc-pascal declaimed the following: > >Not verified by building fpc and lazarus on this machine yet. >But it is clear that svn export is what should be in the script anyway to save >both time and disk space... > Based upon google searches; export is used to obtain the source file tree but without maintaining version control on the fetched files. OTOH, co creates a local SVN repository with a version control subdirectory, metadata, etc., and working directory above it.

I know this and that was the reason that I now tested both above.

Note that I started version control life with CVS back in 2001 and there are no metadata in CVS except for a file listing the versioned files in the checkout including the revision.

Export is the same as if one first does a checkout and then erase the hidden CVS directory in each folder. So the size difference between checkout and export in CVS is tiny, whereas both svn and git keeps huge data in parallel.

If you never intend to apply version control to the files -- and download a complete new set of files for each change of interest then

For the case I am asking there is ABSOLUTELY NO WISH to keep any metadata or possibility to update to newer revisions.
Just the complete source file trees are required.

Bo Berglund Developer in Sweden
Michael Van Canneyt
to Bo Berglund
Bo Berglund wrote
For the case I am asking there is ABSOLUTELY NO WISH to keep any metadata or possibility to update to newer revisions. Just the complete source file trees are required.

In that case by far the easiest is simply to download and extract the archives that gitlab creates for you on request, as Martin Friebe showed in his mail.

You absolutely don't need svn or git, just wget or curl will do, followed by an unzip or tar.

Michael.