I've been nibbling at data migration system for many years. It's gone through various transformations, and it's latest addition is mostly working. The original purpose of the program I forget, but it's main use for awhile has been to extract the EVE Online database data from the SQL Server database dump that CCP kindly provides. Each EVE revision, I take the CCP dump, spin up a Windows server in the cloud, import the database and extract what I need to port it into PostgreSQL, which is my system of choice.
Over the years, JDBC has improved, and technologies have moved along. In the beginning I wrote Hermes-DB, a simple ORM that was very much not type safe, but coped with many of the auto-probing of table information that comes along with a more dynamic style ORM. One can argue that this isn't really ORM at all, and at this point, I'm inclined to agree.
Having said that, the auto-probing capabilities turned out to be very very useful in extracting data. Because the system was predicated on the idea that learning about the database should be the job of the framework, not the developer, it had a reasonably well formed concept of representing tables and columns as objects. With a bit of tweaking, adding a new metadata class along the way, the package can represent a table definition fairly well now.
What this allows me to do today, is create both a solid database dump, and the DDL to build the table structure. Theoretically this system could be modified to pull from any datastore and generate for any other datastore. The system was built in a way that was hopefully designed to facilitate that.
2012 rolls around, and things have changed. The landscape for web development has been shifting over the last decade as people struggle to find way to get the tools out of the developer's way, and enable them to do their job more and fight with code less. The most recent evolution in that sequence that I've been working with, is Scala and Play. As I work with two tools, I'm increasingly finding it easier to build systems that are stable, and take much less code to write.
Hermes-DB was originally designed just to output DDL, but when I started working with JPA, a system that requires a whole lot of scaffolding, it made sense to have one of the output "DDLs" be Java classes with JPA annotations. Over the last few days, I've been making a new variety of output, Scala case classes designed to work with Play and therefore Anorm. Anorm is very powerful, and gives you tools that "get out of your way", but doesn't have a lot when it comes to scaffolding. I've poked around a bit, and it seems there was a scaffolding plugin for Play 1, but none exists for Play 2. This little utility, is helping fill that gap for me. It outputs Scala class and companion object definitions based on the database schema.
The EVE Online database comes out of the box with about 75 tables. 75 tables that I'd rather not have to manually create mappings for for model classes. This little utility made my life much easier. A bit cheer for code generation tools!
It is open source of course, and can be found on gitorious with the git URL: git@gitorious.org:export4pg/export4pg.git
Please note that some of this code is very very old, and it's worked for probably close to a decade so some of it is a bit ancient in both understanding and coding style. It is however, very useful, and possibly one of the pieces of code I've written that's still in usage and not broken from constant tinkering!
No comments:
Post a Comment