Address Object Storage Before You Start Your ECM Project

We’re going to add one more item to our pre-ECM project checklist: Where should we store our content?

An Enormous Drop in Storage Costs Have Changed Storage Practices

Database storage used to be expensive.  In the 1950’s, the cost per megabyte of storage exceeded $10,000/MB.  Today, the cost has dropped to a few cents.  Not only have storage costs dropped, so have memory costs as they have followed the same price drop as storage.  Taking advantage of lower costs; most DB manufacturers have begun offering high performance in memory databases (IMDB – In-Memory Database).

From an ECM perspective; because of the higher database costs, content storage solutions were designed to use databases to store only the metadata or the index values associated with content, and the actual files and documents were stored on cheaper file storage devices.   While lowering costs, this approach meant that ECM solutions were forced with managing, synchronizing, backing up, and designing applications where index values were one place and the actual documents, audio/video files were somewhere else.

Database manufacturers have taken advantage of the lower costs and introduced file-types that allow for storage of large objects.  Oracle for example offers a number of ‘large object’ datatypes such as BLOB, CLOB, NCLOB, etc.

Back-ups Can Challenge Huge In-Memory DBs

Newer ECM products such as ILINX Content Store have taken advantage of the lower costs and allow for the storage of all content in the database.  This has made the management of ECM solutions much simpler.
So, what’s the downside?

Recently, we were contacted by a customer that was utilizing BLOB storage for a content database that had grown to well over 1TB.  The database backups were taking longer and longer and were approaching the maximum time window available.  The most cost-effective solution was to remove the content from the database and store it on network storage.

Check out John Linehan’s post to see how we did this.

Lesson Learned—Planning Upfront Will Avoid Fixes Down the Road

The project itself was not terribly difficult – however it was very, very time consuming.  Our recommendation going forward is to make the storage decision upfront.  Decide how big your content application will become, and make and document your decision regarding where best to store that content.

Steve Kissinger
Project Manager
ImageSource, Inc.