Sign in Join

Live chat for Free Pascal, Lazarus, Delphi and every Pascal programmer.

Mailing lists / fpc-pascal / 2014-07

Migrate mantis bugtracker

10 messages · 4 people · –

Reply by email Subscribe to this list Your subscription

Read-only archive. Use j and k to move between messages.

Michael Van Canneyt

Hi,

In case someone is interested:

I created a FPC program to migrate a mantis bugtracker database from MySQL to Postgres.

The reason for creating such a beast is that MySQL is inferior to Postgres in many ways, and so we're switching the FPC bugtracker from MySQL to Postgres. (the recent bugtracker downtime is no stranger to this decision)

The program takes care of several conversion pitfalls: Different data types and erroneous data (MySQL allows wrong UTF8 in the tables, postgres rightly complains about it).

If someone has interest in such a program : let me know, and I'll send you the sources.

Michael.

silvioprog
to Michael Van Canneyt

On Thu, Jul 31, 2014 at 1:48 PM, Michael Van Canneyt <

Show quoted text 22 lines
wrote: Hi, In case someone is interested: I created a FPC program to migrate a mantis bugtracker database from MySQL to Postgres. The reason for creating such a beast is that MySQL is inferior to Postgres in many ways, and so we're switching the FPC bugtracker from MySQL to Postgres. (the recent bugtracker downtime is no stranger to this decision) The program takes care of several conversion pitfalls: Different data types and erroneous data (MySQL allows wrong UTF8 in the tables, postgres rightly complains about it). If someone has interest in such a program : let me know, and I'll send you the sources. Michael.

Hello buddy!

I'm interested! Please send it to Github, that way everyone will benefit. :-)

Silvio Clécio My public projects - github.com/silvioprog
Graeme Geldenhuys
to Michael Van Canneyt: In case someone is interested: I created a FPC program to migrate a mantis bugtracker…
Michael Van Canneyt wrote
I created a FPC program to migrate a mantis bugtracker database from MySQL to Postgres.

And soon there will be one from MySQL to Firebird too. :-) Seems great minds think alike. My tiDataPump tool (yes, tiOPF based) already managed to convert a 80GB PostgreSQL database to Firebird. MySQL was my next target.

I would be interested in your code though, so I can improve the MySQL and PostgreSQL mapping in tiOPF. So if you could sent that email, it would be much appreciated.

Regards,
- Graeme -

fpGUI Toolkit - a cross-platform GUI toolkit using Free Pascal http://fpgui.sourceforge.net/
Michael Van Canneyt
to silvioprog: On Thu, Jul 31, 2014 at 1:48 PM, Michael Van Canneyt < Hello buddy! I'm interested!…
silvioprog wrote 20 lines
On Thu, Jul 31, 2014 at 1:48 PM, Michael Van Canneyt wrote: Hi, In case someone is interested: I created a FPC program to migrate a mantis bugtracker database from MySQL to Postgres. The reason for creating such a beast is that MySQL is inferior to Postgres in many ways, and so we're switching the FPC bugtracker from MySQL to Postgres. (the recent bugtracker downtime is no stranger to this decision) The program takes care of several conversion pitfalls: Different data types and erroneous data (MySQL allows wrong UTF8 in the tables, postgres rightly complains about it). If someone has interest in such a program : let me know, and I'll send you the sources. Michael. Hello buddy! I'm interested! Please send it to Github, that way everyone will benefit. :-)

Well, FPC also has a repository for this kind of side-projects:

Just check out

http://svn.freepascal.org/svn/fpcprojects/convertmantis

Michael.

Michael Van Canneyt
to Saunders, Rich: I'm interested! I'd love to see your approach because I may have to do something similar…
Rich wrote 10 lines
On 2014-07-31 12:48, Michael Van Canneyt wrote: > I created a FPC program to migrate a mantis bugtracker database from > MySQL to Postgres. > > If someone has interest in such a program : let me know, and I'll send > you the sources. I'm interested! I'd love to see your approach because I may have to do something similar between two other databases.

Well, the code is nothing special. Pretty straightforward use of SQLDB.

See:
http://svn.freepascal.org/svn/fpcprojects/convertmantis

If you don't have subversion, let me know and I'll send it to you in a private mail.

Michael.

Michael Van Canneyt
to Graeme Geldenhuys: And soon there will be one from MySQL to Firebird too. :-) Seems great minds think…
Graeme Geldenhuys wrote 13 lines
On 2014-07-31 17:48, Michael Van Canneyt wrote: > > I created a FPC program to migrate a mantis bugtracker database from MySQL to Postgres. And soon there will be one from MySQL to Firebird too. :-) Seems great minds think alike. My tiDataPump tool (yes, tiOPF based) already managed to convert a 80GB PostgreSQL database to Firebird. MySQL was my next target. I would be interested in your code though, so I can improve the MySQL and PostgreSQL mapping in tiOPF. So if you could sent that email, it would be much appreciated.

See SVN :)

But the code is pretty simple, nothing special.
It just takes some care of bugs I encountered in the MySQL database, and it converts int to boolean through some casting tricks in SQL. (It seems Mantis creates different databases for different engines)

The hard work is creating all the SQL statements.

Michael.

silvioprog
to Michael Van Canneyt: Well, the code is nothing special. Pretty straightforward use of SQLDB. See:…
Michael Van Canneyt wrote 24 lines
On Thu, 31 Jul 2014, Saunders, Rich wrote: > On 2014-07-31 12:48, Michael Van Canneyt wrote: > >> I created a FPC program to migrate a mantis bugtracker database from >> MySQL to Postgres. >> >> If someone has interest in such a program : let me know, and I'll send >> you the sources. >> > > > I'm interested! I'd love to see your approach because I may have to do > something similar between two other databases. > Well, the code is nothing special. Pretty straightforward use of SQLDB. See: http://svn.freepascal.org/svn/fpcprojects/convertmantis If you don't have subversion, let me know and I'll send it to you in a private mail.

Thanks a lot sir! :-)

Silvio Clécio My public projects - github.com/silvioprog
Michael Van Canneyt
to silvioprog
silvioprog wrote 23 lines
On Fri, Aug 1, 2014 at 4:43 AM, Michael Van Canneyt wrote: On Thu, 31 Jul 2014, Saunders, Rich wrote: On 2014-07-31 12:48, Michael Van Canneyt wrote: I created a FPC program to migrate a mantis bugtracker database from MySQL to Postgres. If someone has interest in such a program : let me know, and I'll send you the sources. I'm interested! I'd love to see your approach because I may have to do something similar between two other databases. Well, the code is nothing special. Pretty straightforward use of SQLDB. See: http://svn.freepascal.org/svn/fpcprojects/convertmantis If you don't have subversion, let me know and I'll send it to you in a private mail. Thanks a lot sir! :-)

You're welcome.
I have meanwhile enhanced the code so the sequences for the ID field are also updated at the end of a table conversion, but that is not yet committed. I will commit that tonight.

I've also had to implement some changes to limit memory usage; The same base code is used to convert the FPC testsuite from MySQL to PG. Since 500 million records do not fit in memory, I had to use some more advanced mechanism to limit memory usage.

Michael.