Sign in Join
Wiki copytemporary

git migration: what changed

Revision 1 to 2. Removed lines have a minus sign; added lines have a plus sign.

OldNewLine
11> **Note:** The migration is finished, so this page is kept mainly for historic reference
22
33This page is about migrating FPC from SVN to git
44
55## Why?
66
77- More flexible and modern SCM (normally faster than svn, FPC development workflow might improve)
88- Some of the developers (known: Florian, Jonas) already use git svn regularily, for a lot of reasons (easier testing on different machines, recording of commits before pushing, parallel work on different features easily possible, before/after testing before pushing easily possible).
99- Third party contributors might have an easier life.
1010- Stitching of the svn repository with the old cvs repositories possible: very complete history of FPC available
1111
1212## Why not ?
1313
1414- Subversion allows inexperienced users to be given a simple "fixed from revision 12345 onwards" or "revert to r12340" statement.
1515- Some of the developers don't use git (at least Marco, Nikolay), so it is an extra burden.
1616- Conversion time, retraining.
1717- Client configuration needed after install ? SVN client does not need much configuration, if any.
1818- See list "misc"
1919- Simple operations get more complicated. Complications also affect users that don't benefit from git. (double so if every commit must be done in a branch to emulate current merging model)
2020- Uncertain merge model.
2121- The third party contribution (pull request and the like) is very informal, and many of the new, inexperienced contributors routinely run formatters and the like. Practicality might be limited. In the Linux kernel these go through a layer of contributors and then branch lieutenants, we don't have such tiered structure for vetting, and require some discretion on the hand of the submitter.
2222- Can users perform (accidental) irreversible damage to the central repository by pushing certain conflicts? GIT seems to allow much deeper internal meddling in the repo, and if these can be accidentally pushed......
2323
2424## Concerns/Questions
2525
2626### What part of SVN to migrate ?
2727
2828- More is better.\
2929 Jonas has a very complete git mirror of the SVN+CVS part.\
3030 (care needs to be taken: there used to be a time when copyrighted code was checked in)\
3131 A first test conversion by Florian using subgit was attempted: completed in 5 hours 1, crash. Looks OK.
3232- Does git have some form of obliterate option (remove revs based on copyrighted sources) ? Would solve some of the importing old history problems.\
3333
3434
3535
3636Yes, it seems it does. [See this StackOverflow topic](https://stackoverflow.com/questions/18010048/how-to-permanently-delete-a-commit-from-gits-history) --Graeme ([[User talk:Ggeldenhuys|talk]]) 14:42, 19 December 2017 (CET)
3737
3838
3939The SO topic does not help much, but after some experimenting I got `git filter-branch --index-filter 'git merge-base --is-ancestor <hash> $GIT_COMMIT ; if [ $? -eq 1 ] ; then git rm --cached --ignore-unmatch <path in repo/file> ; fi' HEAD` which removes all revisions of \<path in repo/file\> which are not ancestor of \<hash\>. So this removes old versions of files which were cleaned by the commit identified by \<hash\>. This is reasonable fast (~ 1 hour with my currently converted repository). Afterwards, the file appears to be committed by \<hash\>. Drawbacks: All history before of the file is lost, also of clean parts. All revisions afterwards get new hashes, so this means, all clones done before must be removed. --FPK ([[User talk:FPK|talk]]) 19:59, 19 December 2017 (CET)
4040
4141- In order to save on diskspace, find ways to tell user how to clone only a part.\
4242
4343
4444
4545Use `git clone --depth 1` to get only the latest revision (with fetch, later on more revisions can be fetched if needed)
4646
4747Is this really a problem? A git clone of FPC trunk with full history is 50% smaller (disk space usage) than a HEAD revision 'svn co' checkout using SubVersion. --Graeme ([[User talk:Ggeldenhuys|talk]]) 14:36, 19 December 2017 (CET)
4848
4949
5050It is not: my test respositry is around 500 MB on linux after gc, sources itself are around 350 MB
5151
5252--FPK ([[User talk:FPK|talk]]) 20:18, 19 December 2017 (CET)
5353
5454
5555
5656Just checked, only .svn directory of trunk checkout is 800MB If git clone is 500MB it is about 50% smaller. --AlexVinS ([[User talk:AlexVinS|talk]]) 00:35, 20 December 2017 (CET)
5757
5858
5959But after **svn cleanup** it gets reduced to 327MB. --Nickysn ([[User talk:Nickysn|talk]]) 01:19, 20 December 2017 (CET)
6060
6161With aggressive gc .git gets 425 MB --FPK ([[User talk:FPK|talk]]) 21:32, 21 December 2017 (CET)
6262
6363And do not forget: using git worktrees, you need only one .git dir for "trunk" and "fixes" --FPK ([[User talk:FPK|talk]]) 21:35, 21 December 2017 (CET)
6464
6565And I worry mainly about fpcbuild with its non differential history, since my most major disk constraints are VMs for release building Marcov ([[User talk:Marcov|talk]]) 15:01, 21 December 2017 (CET)
6666
6767
6868`--depth 1` works with submodules too--AlexVinS ([[User talk:AlexVinS|talk]]) 15:55, 21 December 2017 (CET)
6969
7070Don't worry too much: fpcbuild trunk .svn: 57.2 MB; full fpcbuild .git: 40.4 MB. So if you are still worried about disk space, we should switch to git asap --FPK ([[User talk:FPK|talk]]) 21:32, 21 December 2017 (CET)
7171
7272
7373That would resolve my scenario with the biggest (feared) size problems. Marcov ([[User talk:Marcov|talk]]) 12:01, 22 December 2017 (CET)
7474
7575### Build repository
7676
7777The fpc build repository uses <svn:external> references.
7878
7979Git has modules, which is in essence the same. This needs to be properly set up.
8080
8181### Branching model ?
8282
8383#### Merging model
8484
8585Merge branches using fast-forward or not ?
8686
8787The 2 models mentioned below use a different merging technique, fast-forward using merge commit. A hybrid could enforce fast-forward on some branches (notably, the releases branch), see [track all remote git branches as local branches](https://stackoverflow.com/questions/379081/track-all-remote-git-branches-as-local-branches)
8888
8989In particular, the command to force fast-forward:
9090
91-`git config --add branch.master.mergeoptions --ff-only`
91+```text
92+git config --add branch.master.mergeoptions --ff-only
93+```
9294
9395#### How to handle the fixes branch
9496
9597FPC uses trunk for development. Release are generated from the fixes branch. The fixes branch consists of cherry picked commits from trunk. After a patch proved to be good in trunk, it is cherry picked and committed to the fixes branch. This is a development model which works very well with subversion and is used by FPC for almost 20 years. However, to make this development work well, it requires that the SCM supports cherry pick tracking, something git does not have. So this workflow does not work well with git. Changing the workflow of FPC is not an option at this point, it proved to work, furthermore there are good reasons to use it: When making a patch, it is often hard to know if it is good enough for fixes or not, normally, it is decided after some time or even short before the next release, if a patch is good and safe enough to make it into fixes. This might not happen because it required more changes and those changes got too invasive or because it e.g. turned out that caused incompatibilites which make the patch a candidate for the next minor release instead. So this is probably the key question for a full git migration: can git be used with the workflow fpc uses?
9698
9799So far, various workflow models exist:
98100
99101#### [A successful Git branching model](http://nvie.com/posts/a-successful-git-branching-model/)
100102
101103Also known as "git-flow" model.
102104
103105Advantages
104106
105107- A simple and logical workflow that plays to the strengths of Git - branching and merging.
106108- 'master', 'develop' and 'release' are good branch names which immediately makes it obvious what they are for. Many developers clone a repo and want or expect a stable version. This workflow allows just that - the 'master' branch is the default branch, and is always the latest stable release of the product.
107109- Features or multi-commit (complex) bug fixes show a clear commit history or related commits, that is easily tracked, viewed or even rolled-back in the commit history using a tool like gitk.
108110
109111Disadvantages
110112
111113- It works only well if the the whole release policy of FPC is changed. This is something very likely not going to happen.
112114- cluttered and basically unreadable history
113115- doubled regression testing effort during development of new stuff
114116 - test branch before merge
115117 - test branch after merge, as merging could have broken things
116118- ~~micro management of branches: every single bug fixes requires and results in a new branch~~ This is simply not the case. single bug fixes can be committed into the 'develop' branch as a linear history. Only more complex (multi-commit) fixes or features will result in a parallel history or development, but quickly merged back into 'develop'. --Graeme ([[User talk:Ggeldenhuys|talk]]) 18:50, 17 December 2017 (CET)
117119
118120
119121 You do not see the problem: if it is in develop, it will never make it into master without merging whole develop into master (i.e. in terms of FPC releases this would mean a new minor release x.y+1.0). So each bug fix has to go into an own hotfixes branch (e.g. hotfix_bug12345) which is branched from master and merged first into develop. If it works, it will be merged back into master. --FPK ([[User talk:FPK|talk]]) 22:58, 17 December 2017 (CET)
120122- One has to know before pushing how invasive a change is because it has to go in the right branch. For compiler development this is very cumbersome.
121123- develop and release have no clearly described function it seems.
122124
123125#### [The cactus model](https://barro.github.io/2016/02/a-succesful-git-branching-model-considered-harmful/)
124126
125127Also known as the anti-"A successful Git branching Model"
126128
127129Advantages
128130
129131- clear and straight history
130132- no time consuming micro management of branches
131133- history and logs not cluttered with merge commits
132134- no breakage by wrong conflict resolving during merge commits by less experienced users
133135
134136Disadvantages
135137
136138- Git was designed to work well with branches and handle merging as a common task. This workflow suggests merging as a bad idea, which is crazy.
137139- This model recommends that a human must now keep track of which commits to cherry-pick into other branches. This is bounds to fail in the long run and vital commits will be missed at some point. Merging two or more branches on the contrary will automatically handle all commits in a branch seamlessly, so why not use what Git was designed to do well.
138140- This model suggests that you can also cherry-pick from an unstable branch back into a stable release branch. This is just the wrong way round.
139141- This model sees some of its own flaws and suggests a 3rd party tool to help keep track of things like cherry-picking. This simply isn't needed if you use a better workflow.
140142- It suggests that 'rebase' get used often. This has always been frowned upon in the Git community, and it is widely known to use rebase sparingly as it rewrites history and doesn't result in a logical evolution of the code history.
141143- This worklfow suggests that a merge commit is a bad thing. A merge commit is a commit like any other. It can be a simple commit (with no conflicts), or can be a commit that resolves conflicts (thus code changes were required - which this commit records). Nothing bad about that.
142144
143145#### [See the Git project itself](https://git-scm.com/docs/gitworkflows)
144146
145147Advantages
146148
147149- The git project itself gets an extreme amount of contributions via pull requests and emailed patches. Their workflow model clearly works well and uses branching and merging extensively - even though the commit history looks quite scary. :-)
148150
149151Disadvantages
150152
151153- No real description of the process.
152154
153155### User management ?
154156
155157Git has no concept of users. To manage permissions on a server, a separate program is needed.
156158
157159- Correction: A separate program makes some functionality a bit easier but not mandatory. Simple UNIX style group and file permissions with SSH public keys work very well too. --Graeme ([[User talk:Ggeldenhuys|talk]]) 14:32, 2 January 2018 (CET)
158160
159161#### gitorious
160162
161163- Advantages
162164 - uses git repo for administration
163165 - No server binary
164166- Disadvantages
165167 - No web interface
166168 - administration needs ssh key, only ssh possible.
167169 - Web integration ?
168170
169171### gitea
170172
171173- Advantages
172174 - Web based
173175 - Fine tuning possible
174176- Disadvantages
175177 - Separate config
176178 - Requires running binary all the time, on a separate port.
177179
178180Git differentiates between the person that does a commit, and the author of a change/patch. The latter is always available and doesn't need any repository permissions. Somebody with write permissions can commit patches (or merge pull requests) and keep both the Committer and Author information intact. SubVersion doesn't track the author of a change, only the person that did the commit.
179181
180182### What about Lazarus ?
181183
182184The Lazarus team (Martin Friebe) is currently maintaining a copy of the SVN repo on gitea.
183185
184186- <http://svn.freepascal.org:8080/Lazarus/lazarus> (does not work)
185187- <https://github.com/graemeg/lazarus>
186188
187189### Dependencies on Unix-world tools / ports ?
188190
189191There seem to be dependencies on quite a few tools common on Unix platforms. Unfortunately, ports of these tools may not be up-to-date and/or working well on non-Unix platforms. As an example, port of GIT 2.13.3 to OS/2 lists dependencies on a Unix shell, Python, Perl and CURL among others. Using Unix shell for launching other programs on a non-Unix platform may have various unwanted effects (remember cygwin ports which we tried to avoid on MS Windows as much as possible for similar reasons). The list of dependencies seems to suggest that quite a few operations require external tools (similarly to merging with early SVN versions). It may be difficult to test all such cases potentially important for certain workflows in advance before the final switch.
190192
191193
192194
193195OS/2 being the first commercial operating system to ship with Java built-in. So why not simply use the Java version of Git. The Eclipse and IntelliJ IDEA project do on a daily basis. --Graeme ([[User talk:Ggeldenhuys|talk]]) 14:39, 18 December 2017 (CET)
194196
195197History of OS/2 and Java are one thing, practical usability is another. First, I'm not sure that I want to run Java runtime to commit a change to one file, get a log of recent changes, etc. Second, which JRE version would be necessary for that? Are such JRE versions available for OS/2? --User:Xhajt03
196198
197199Sorry, another addition - the MS Windows installation of GIT contains a subdirectory with a mingw64 tree (containing a bunch of stuff like bzip2.exe, gettext.exe, antiword.exe, openssl.exe, odt2txt.exe, sqlite3_analyzer.sh or tclsh.exe), plus another subdirectory containing another copy of bzip2.exe plus e.g. gawk.exe, gzip.exe, iconv.exe, nano.exe, perl.exe, sed.exe, ssh.exe, xargs.exe, i.e. a _lot_ of 3rd party tools from various sources, not even mentioning the main bin directory containing bash.exe and sh.exe in addition to git.exe). In total, the Git installation there amounts to 500 MB!!! SVN installation on the same machine is 17 MB, and even the TortoiseSVN (i.e. the GUI addon) is 47 MB. Don't get me wrong, it isn't the size that scares me, but the amount of dependencies. Although many of these tools are probably ported to OS/2 too, they may not be compatible to each other (e.g. requiring different versions of the same DLLs, etc.). :-( I'd need to understand which of the tools are really strictly necessary for regular working with Git (see the section with scenarios on this page).
198200
199201
200202There is supposedly an rpm package for OS/2 that installs everything you need (for command line git): <http://www.edm2.com/index.php/Using_Git_under_eComStation>
201203
202204This is indeed annoying on windows if files like bash or "sh.exe" are in path due to this. Marcov ([[User talk:Marcov|talk]]) 13:33, 1 February 2020 (CET)
203205
204206### Misc.
205207
206208Collected from mails of Marco which is not handled in other sections:
207209
208210- I really dislike losing global revs.
209211- Will we need to store anything in one repo (build and fpc?) Would be huge with all histo, and problematic for release building VMs etc . What are the options for partial checkouts/externals ?
210212- Can branches be in a not usable (multi-head) state? I heard some complaints about that, where users avoided work till it was fixed, and had flashbacks of locking VCSes all over again. Should not be possible.
211213- git requiring sequences of commands for simple operations (local then global push etc)
212214- lineendings, should be entirely server dictated. In general, most scenarios should be simple singular commands not several with bunches of parameters.
213215- from the "branching" paragraph: readable logs. Marco didn't add it because he never would have guessed it would be a problem.
214216- I heard github mentioned a few times, did sb check GH licensing?
215217
216218
217219Answer from Chain\|Q:
218220
219221
220222<https://help.github.com/articles/github-terms-of-service/>
221223
222224Point "D" caused an uproar a while ago, because some thought that it implied that you're granting GitHub rights to use your content as they see fit. This has been clarified since, I think.
223225
224226### Scenarios to document
225227
226228#### SVN scenarios that need an equivalent: (note that equivalency means that changes should be visible on the central server)
227229
228230- install on new machine, mandatory configuration (preferably: none mandatory other than URL) for at least linux and windows supported client (tortoise?). Things to setup/configure(server dictated?)
229231 - crlf?
230232 - The defaults which git supplies (Linux, Mac and Windows) should be just fine. Repo internal storage is LF and checked out code is the native line endings of your system. The ".gitattributes" file could control line endings of specific file types.
231233 - this file is in the repo, iow server dictated, making it zero conf?
232234 - Configure user name and email address for commit messages
233235 - git config user.name "John Doe"
234236 - git config user.email "johndoe@example.com"
235237 - Can add "--global" after "config" if you want to set this as the default for all repositories
236238- Checkout a working repo
237239 - command: git clone \<url\>
238240- Changing from trunk to a fixes branch (and vice versa)
239241 - command: git checkout \[\<branch\>\]
240242 - Two different checkouts are usually used with SVN, but if it's done differently with git, it should be documented here as well
241243- update/sync
242244 - Get latest changes via command: git fetch \[remote_server\]
243245 - This only downloads the changes, without applying them. In order to apply them, you have to do a 'git rebase', which will reapply your local commits on top of the remote changes. If there are conflicts, you will have to resolve them (TODO: "git add" as equivalent for "svn resolved", and "git rebase --continue" to continue after adding all resolving conflicting files)
244246 - Push latest changes via command: git push
245247- check for modifications (svn status)
246248 - command: git status
247249- see history log (svn log -v)
248250 - command (console only): git log
249251 - or command (gui showing all branches): gitk --all
250252- commit a simple modification.
251253 - A two/three step process:
252254 - git add \<file_that_changed_or_that_is_new\>
253255 - Trap for Subversion users: If you apply further changes to the file, you need to 'git add' again the same file, otherwise git won't include them in the commit (i.e. 'git add' takes a snapshot of the file at the moment you enter the command).
254256 - To undo 'git add \<file\>', use 'git reset \<file\>'.
255257 - git commit -m "commit message"
256258 - git push (to send all local commits to the server)
257259 - This will refuse to work if there have been any remote changes (even if they are in different files). In that case, first "git fetch" and "git rebase" before pushing again. For comparison, Subversion will ask you to update and merge the remote version, only if there are any changes in the files you're trying to commit.
258260- get the list of eligibles
259261- create a branch/tag (fixes/release/rc)
260262 - All-in-one command: git checkout -b \<new_branch_name\> \[starting_point\]
261263- merge a revision to a different branch (trunk-\> fixes or fixes -\>release/rc) (added later:I assumed this was tracked. The answer below might not reflect that)
262264 - Very easy via the GUI gitk tool. Right-click and select "cherry pick"
263265 - This works only in simple priyatna.orgcases. In general I think
264266
265-` git cherry-pick -x -n <commit-id>`
267+```text
268+ git cherry-pick -x -n <commit-id>
269+```
266270
267271\<if needed, fix conflicts\>
268272
269-` git commit -F .git/MERGE_MSG `
273+```text
274+ git commit -F .git/MERGE_MSG
275+```
270276
271277is the way to go. This tracks the merged revision in the commit message and it can be search by
272278
273-` git log --grep <commit-id>`
279+```text
280+ git log --grep <commit-id>
281+```
274282
275283if a certain revision is cherry picked.
276284
277285- edit commit message
278286 - Generic, safe way
279287 - git rebase -i \<commit-hash\>^ (note the extra ^ at the end)
280288 - in the window that appears, change "pick" into "r" or "reword" for the commits whose commit message you wish to change
281289 - save the file, and git will open a new editor for each commit message you selected for rewording
282290 - Shortcut in case you want to change the commit message of HEAD (in case you did not yet push it to the remote server)
283291 - git commit --amend
284292 - Note: if you added any changes to the index (= used "git add" to select changes to commit), these changes will also be added to that same commit. Use "git reset" to remove them from the index again, if you do not want this)
285293
286294- revert local changes (svn cleanup)
287295 - git reset --hard HEAD
288296- show annotations for a particular file (svn blame)
289297 - git blame
290298- show diffs for a particular revision of a particular file or of all files included in that revision (comparing it to the previous revision or to some other explicitly specified revision)
291299 - git show -p \<commit-hash\>
292300- exporting 'clean' sources (without repository metadata) for release building (svn export)
293301 - git checkout-index --prefix=destination/directory/ -a
294302
295303#### (New) git specific scenarios
296304
297305- Perform a change not (yet) visible on the central server, but still recognized as a specific change with a specific commit message in the local repository
298306- Restructure local changes from the previous scenario to a different set of commits
299307
300308## Tool for migration
301309
302310### subgit
303311
304312Advantages
305313
306314- very flexible
307315- very fast (remote cloning of the whole FPC repository takes 5-6 hours)
308316
309317Disadvantages
310318
311319- external java-based tool, so some dependencies
312320
313321### git-svn
314322
315323Advantages
316324
317325- included in git it self
318326
319327Disadvantages
320328
321329- less flexible
322330
323331## Work to do
324332
325333### Migrate SVN repo.
326334
327335- 2017-12-16: A first test conversion by Florian using subgit was attempted: completed in 5 hours, 1 crash. Looks OK
328336
329337#### Branch mapping for subgit
330338
331339Proposal for branch mapping
332340
333341 trunk = trunk:refs/heads/master
334342 branches = branches/merged/*:refs/heads/merged/*
335343 branches = branches/joost/*:refs/heads/joost/*
336344 branches = branches/laksen/*:refs/heads/laksen/*
337345 branches = branches/maciej/*:refs/heads/maciej/*
338346 branches = branches/olivier/*:refs/heads/olivier/*
339347 branches = branches/paul/*:refs/heads/paul/*
340348 branches = branches/*:refs/heads/svn/*
341349 branches = branches/svenbarth/*:refs/heads/svenbarth/*
342350 branches = branches/tg74/*:refs/heads/tg74/*
343351 tags = tags/*:refs/tags/*
344352 shelves = shelves/*:refs/shelves/*
345353 excludeBranches = branches/aspect
346354 excludeBranches = branches/avr32
347355 excludeBranches = branches/blaise
348356 excludeBranches = branches/cpstr
349357 excludeBranches = branches/cpstrnew
350358 excludeBranches = branches/ctypes
351359 excludeBranches = branches/dodi
352360 excludeBranches = branches/florian
353361 excludeBranches = branches/foxsen
354362 excludeBranches = branches/fcl-web_joost
355363 excludeBranches = branches/generics
356364 excludeBranches = branches/genfunc
357365 excludeBranches = branches/janbruns
358366 excludeBranches = branches/linker
359367 excludeBranches = branches/merged/avr
360368 excludeBranches = branches/merged/generics
361369 excludeBranches = branches/merged/nodeopt
362370 excludeBranches = branches/newthreading
363371 excludeBranches = branches/peterjan
364372 excludeBranches = branches/ssa
365373 excludeBranches = branches/tg74/rtl
366374 excludeBranches = branches/tg74/tests
367375 excludeBranches = branches/tg74/utils
368376 excludeBranches = branches/unitrw
369377 excludeBranches = branches/wkrenn
370378 excludeBranches = branches/FIXES_2_2
371379 excludePath = /fixes_2_0
372380 excludePath = /fixes_2_4
373381 excludePath = /trunk
374382
375383### Set up user management and permissions
376384
377385### Set up and automate github mirror
378386
379387- I host the unofficial Git mirror at (http://github.com/graemeg/freepascal.git). I can share my cron jobs and script if anybody is interested. --Graeme ([[User talk:Ggeldenhuys|talk]]) 14:25, 2 January 2018 (CET)
380388
381389---
382390*From the Free Pascal wiki, page "git migration", as it stood on 2022-01-13 after a change by Trev, out of the wiki's dump of 2024-05-26. Copyright stays with the author who wrote it, according to the old wiki's [[Licensing]] page. If you have concerns about this material, [message the admin](/wiki-contact).*
383391Category: FPC development
384392Category: FPC internals

Back to history