Software industrialization for Internet PDF Print E-mail
Written by Christophe Fondacci   
Monday, 27 July 2009 11:36

Today's challenge for IT organizations is defined by these 3 words : "time-to-market". This time defines the delay between the idea and its general availability for users. Organizations need to : 1) have the best ideas 2) be the first to release them 3) provide them with the highest level of quality and performance.


Efforts to reduce the time between the idea and its implementation leads to 2 major principles :

  • Every manual, repetitive task of the realization process must be identify and automated
  • Every development must be tested immediatly after it has been made


Software production cycle - Time to market

The software industrialization era was born. For now it mostly consists in tools, robots, scripts, schedulers which can assemble and deploy all the software pieces together to build the whole application and then run unit, technical and functionnal tests against it. This is a software factory which can build and test every piece of code each time it changes. We can now deploy an application by clicking one single button...


But, we're missing a big part... Almost every application relies on a database to store / retreive its data. Even though databases generally represents less development efforts than the whole application on top of it, it still defines the structure and the data required by the application in order to work. So if we have a software factory for the application, we need a database factory to assemble the database developments as well. Without, we would test our application against the development database, or we would mock the entire database access layer to only test the application code preventing end-to-end functionnal tests before delivering.


The purpose of industrialization is to automate things and to detect problems early so that they can be fixed quicker... In the previous use case we might face some troubles after delivering the software, that means later. Another problem is that we have a fully automated package & delivery process for the application code, but we still need to create SQL scripts by hand for structural upgrades of the database. Many will tell us that they've made their own custom delivery system and that it works well...


Our point of view is that it might work far. Several issues came to us during our past experience with different organizations :

  • Nobody is capitalizing the packaging and delivery mechanisms: each organization has its own way of doing it. It could be hard to get into those scripts for newcomers, it generally implies to "teach" how to make a database change. Even worse: you may multiply the tools you are using depending on the database vendor you are developing for within a single company.
  • Database is crucial: if an upgrade fails, your whole data may get corrupted and the system will not work at all.
  • Today's applications are modular: they are made of components or services which are assembled together to constitute the application. This facilitate code capitalization, code reuse and provides a more flexible way of architecturing things. How do you handle this modular approach with your data models ? Will you regenerate all SQL scripts ? Once you get 2 different applications based on a same data module, how do you handle maintenance of this shared component (new features, bug fixes) ?
  • Developments will generally be made in paralllel: you always have several releases to maintain. At least you will have the production release which you may need to patch and the next development release. Production patches will need to be merged with your current development to avoid regression when the development release will go live.


Because we faced all these issues, we started to build the neXtep designer platform... The idea is to provide an IDE that addresses those needs in a transparent way. The targetted objective is to let all database developers concentrate on their expertise : building efficient and consistent data models.


First, we started to think on how we could handle the packaging and delivering automation. The singularity of database deployment resides in its incremental aspect. You do not install  the new release, you transform the existing model into the new one by sending the appropriate SQL commands to the database engine. Because of this incremental aspect, versioning is the cornerstone of the neXtep designer IDE. Thanks to version history, we can compare any version with another, retreiving any differency, thus allowing us to automatically produce corresponding SQL scripts.


This is also the reason why we answer to the package & delivery automation with a development IDE: we need history of changes from the development stage in order to automate the incremental packaging and deployment stages. Once you will get used to it, you will start to consider SQL DDL (Data Definition Language) as a way of delivering changes to your data structures.


We considered building a new database IDE was relevant, because we think we are lacking of good and modern integrated database development environment. But we needed to do more than what database IDE do today. We need to gather all development features that a database developer need into one single environment. That means graphical modelization, database client, offline database development, version control, compiler, content assist for SQL edition, debugger (for vendors which can support it like Oracle) and so on. That's why the Eclipse RCP platform perfectly suits our needs with the modular pluggable architecture of OSGi.


Thanks to the extension mechanisms offered by the Eclipse platform, we progressively add more and more features to the environment. Our roadmap is full of great enhancements, such as :

  • Continuous integration server capable of upgrading any target data structure to a given one, providing a web frontend (through the Eclipse Rich Ajax Platform project) to monitor nightly builds & tests.
  • Support for additional database vendors : PostGreSQL and DB2 are strong candidates
  • Integrated debugger for Oracle PL/SQL
  • Domains definition
  • HTML documentation generator
  • Code templates to handle more complex use case (like partitioning / unpartitioning an existing table, data migration, temporary load tables)
  • neXtep installer API to embed your database deliveries within your application
  • Integrate, link, automate more and more features together
  • Support more and more vendor-specific features


We need to finance all these features, and we want to keep a pragmatic approach: we only build features for which we have a demand. That's why we set up our Collaborative Open Source strategy which allow anyone to contact us and to drive the development of a feature he needs.


Last Updated on Saturday, 01 August 2009 19:02
Goto Top