Category Archives for "Document Management"

Migrating 85 million documents from Oracle 10g with no data loss—a process, not an anomaly

ImageSource has been successfully executing migrations, large and small, to and from content repositories since 1994. Recently, we completed a mammoth project for a global financial information services company in which we migrated 85 million documents from a deprecated Oracle 10g IPM system to ILINX Content Store in 9 months with 99.99995% accuracy. That’s zero loss of data, except a few hundred docs that were already corrupt in the source system!
The speed and accuracy of this project can be attributed to 3 important factors that made the content migration successful:

  1. We assembled a strong technical team.
    The team for this project consisted of an Oracle 10g technical subject matter expert, a seasoned PMP Project Manager with years of experience managing Oracle 10g migrations and a SQL database expert. This team provided all the expertise required to plan, execute and audit this size of migration.
  2. We used time-tested methodology from years of experience performing similar data migrations.
    Given the mission critical and high transaction volume nature of the legacy Oracle 10g system, maintaining business continuity was critical. A detailed Migration Plan was developed addressing all content to be migrated, associated requirements (e.g. retention of annotations, excluding content that had or would meet document retention\destruction requirements during the duration of the migration, etc.), and auditing requirements. A detailed audit and migration report was performed and delivered to provide an accounting for every single document in the source system.
  3. We utilized powerful, configurable technology.
    We leveraged multiple tools and techniques in the execution model (e.g. temporary migration environment that included replicated data, multiple instances of ILINX Export, etc.) to be able to complete the migration. By creating a temporary migration environment, we were able to leverage the power of ILINX Export, ILINX Import and a multi-instance model to perform the migration in record time. The numbers speak for themselves: 85M docs, 9 months, zero data loss.

We’ve used a similar services formula for migration execution large and small—your migration doesn’t have to be massive like the one mentioned above to give you great ROI. That, and the right software mix, will eliminate headaches from the equation the next time you have to migrate content.
If schedules or other factors outside your control don’t allow time for a full migration prior to going live with your new system, we have a feature in ILINX Content Store that will allow customers to turn off their normal end user access to your legacy system on day one of using the new system and still pull content from the legacy system through the ILINX interface, read more about that here. This enables significant benefits that include smarter resource allocation within your operating constraints and migration flexibility. Contact us for more information.

Gene Eckhart, PMP
Project Manager
ImageSource, Inc

6 Automation Improvements to Help You Grow Consumer Lending Without Increasing Supporting Staff

We recently helped the largest credit union in Alaska (17th in the nation) automate their consumer loan processing operations. With a growing number of their 77 locations looking to processing auto dealership loan requests, Alaska USA Federal Credit Union needed to replace their manual, paper-based process to meet current and future demands.

As is the case with many lenders who haven’t yet gone electronic, paper loan requests packets were being submitted via multiple avenues from dealerships. These packets consisted of anywhere from 20 to 25 pages per packet with a variety of page sizes, file sizes and quality. Some documents were even printed from DOT Matrix printers at some of the dealerships. Packets came in primarily through fax and email, and local dealers in Anchorage would even hand deliver loan packets. Once received, faxes and emails were printed out and paper documents were used to process the loan request. The documents had to go through various people for review and approval creating a lot of manual movement of paper. Having these documents in paper format also created a need for document storage in file cabinets consuming significant floor space in the building.

In order to rise above the limitations that paper processing placed on loan approval volume, Alaska USA went digital, with 6 major automation improvements to shave valuable time off the process:

  1. Fax Ingestion – When faxes come in they are dumped into a folder, which is monitored by a product called ILINX® Import. As soon as a fax lands in the folder, it is immediately injected into a robust capture product—ILINX Capture
  2. Email Ingestion – Similar to faxes, ILINX Import monitors email folders and pushes the documents (attachments) into ILINX Capture
  3. Scanning Paper – Paper is scanned using prebuilt batch profiles in ILINX Capture for quick indexing in. This component is very flexible, in that scanning can be done with a variety of devices, from small desktop scanners to MFPs to dedicated, high-volume scanners
  4. Image Clean-up – Documents are automatically improved and made more readable through image repair: de-speckle, deskew, auto-rotate, advanced binarization and more
  5. Auto-Indexing – Once in ILINX Capture, database lookups from supporting systems are performed to auto populate many of the index values to alleviate manual entry of data already available
  6. Workflow – The documents, along with pre-populated metadata are then released into Alaska USA’s existing backend software, EMC ApplicationXtender, to travel through workflow to various queues for further processing

Having automated their Consumer Loan process, Alaska USA has greatly improved their service to auto dealers and can be more competitive on winning new business. On the cost savings side, they can process more loans without increasing headcount and have eliminated hard copy document storage.

Al Senzamici

Account Executive

ImageSource, Inc.

Migrating from Stellent UCM & IBPM – A little foresight can alleviate a lot of trouble

Migrations from systems like IBPM to ILINX can be fraught with issues that can bite the unwary in very bad places. However, if you are aware of such problems, you can plan ways to mitigate them and have a successful migration in the end.

One issue we run into is documents that have a page or two with corrupt images. Perhaps when the page was first contributed to IBPM, a system or other type of issue caused the image to be corrupt or cease to exist. Either physical hardware or a software bug can be the culprit. The product we use for migration, ILINX Export, will flag this document as an error, skip it and move on to the next document in RECID order. Once the export is completed, these flagged documents have to be re-visited. Once a determination is made that an image is indeed corrupt, and the chance to recover it from backups is extremely remote, the document can be deleted or manually exported from IBPM without the corrupt image.

Another matter we’ve dealt with is related to non-tiff images. This category is “universal” type images, and includes PDF, DOC, XLS, MSG and a host of other file types that IBPM supports. There are options within the ILINX Export tool that will allow the export of these files types in their native format through the IBPM SDK. Or the export can be done through database manipulation that can directly access the image file and then “unzip” the universal file into its native format.

The issue that can be encountered here is twofold, and manifests itself when migrating to another repository. One, IBPM stores the native file zipped up with another file that contains metadata and has no file extension. When the document is unzipped there are two files, one with a valid file type and one without. Typically, backend repositories require file extensions, which are useful for performance, like displaying file type icon on the user interface, and a variety of other reasons. During the migration, importing to the backend may be impeded due to a lack of extensions on the metadata files. Secondly, if the extension of the universal file has been altered or damaged in storage, the file type may not be a standard that the new repository will accept. In any case, having your migration come to a screeching halt is something to avoid.

Awareness is the key. By proactively incorporating a response into your migration plan, you can eliminate much heartburn and anxiety. That is where the expertise and knowledge of a seasoned Optika / Stellent / Oracle integrator, like ImageSource, comes into play. We have helped many customers build migration plans that take these and other items into account, so the migrations are as smooth and worry-free as possible.

Chris Hillenburg
Sr. Systems Engineer
ImageSource, Inc.

Oracle IPM 10g and Imaging 11g Migration: Part 2 Blog

A couple weeks ago I wrote a post about ECM migrations, with a focus specifically on moving content from Oracle IPM/Imaging to other destination systems—projects we’ve been performing a lot of lately. Our tools of choice for migrations are ILINX Export and ILINX Import, but if the destination ECM system isn’t supported by ILINX Import, there are other options. Almost every ECM system has mechanisms to do bulk or mass imports. ILINX Export provides many options to format the data so sometimes it is a matter of configuring the output to be in a format supported by the third party import application. Other times, utilizing these third party import applications may require a little development. Regardless of what’s necessary, we’ve never run into a destination system that we couldn’t work with.

There are multiple reasons we split the migration operations into two parts—export and import—flexibility being the biggest one. There are a lot more options when splitting the migration into two separate operations. Since we don’t modify the data on export from the source system, a snapshot can be taken for long term archival. Then on import, or pre-import, we can massage the data, perform file conversions, or augment the data by pulling additional data from an external source. Even though we split the migration up into two operations, they can be run in tandem so there is little effect on the overall duration of the migration.

One of the biggest concerns surrounding these migrations is the amount of time it will take. Performing tests in the actual environment is required because of how many variables go into the throughput of a migration. If the migration is estimated to take too long after initial testing, there are options to address that scenario, including:

  • Create a migration environment with instances of the source ECM system software on newer, more powerful servers, and restore the production data to these new servers in order to execute the migration from there. This has the additional benefit of removing any potential performance impact to the legacy production system for the duration of the migration.
  • Spin up additional instances of ILINX Export and/or ILNX Import to increase throughput. There will be a point when additional instances of the export or import process will not increase throughput—generally when when a bottleneck restricts the maximum throughput that the source or destination system can achieve.

Recently, I had a customer that had set a hard go-live date that was just 60 days after project initiation for their new system. We had no problem meeting this requirement from a technology deployment standpoint, but our migration testing indicated that we wouldn’t be able to move all of their 25+ million documents in that time frame. In order to make the new system go-live date, we migrated the three previous years’ content first, then resumed with the older, remaining content. Since the vast majority of content to be retrieved would be from the previous year, the fact that the migration wasn’t 100% complete at go-live was a non-issue. This is an approach we’ve followed numerous times.

Once a migration is in full swing, auditing can be the most time consuming part of the process. ILINX Export and ILINX Import have very complete auditing capabilities, so while the migration is occurring, issues are immediately identified and can be addressed. We generally audit a couple different ways to confirm success. If only using ILINX Export, what is exported can be compared with what is in the source system to ensure all content was pulled out. When performing a complete migration, what is imported into the destination system is compared with the source system. Any migration can only be considered a success when it is proven that all the content was migrated, which is why we practice multi-step auditing during the migration.

By following our standard methodology for migrations and utilizing the technology we’ve developed over the years, we consistently perform reliably successful migrations. To read more about migrations, review my previous blog posts Oracle IPM 10g and Imaging 11g Migration and Steps for a successful ECM migration using ILINX Export.

If you have any questions about my blogs, or would like to discuss the possibilities for migration within your organization, please reach out to me or your contact at ImageSource to start the conversation.

John Linehan
Sr. Systems Engineer
ImageSource, Inc.

Oracle IPM 10g and Imaging 11g Migration

One of the things we’ve done a lot of here at ImageSource are migrations. It’s definitely one our core competencies. For a little more information on our approach to migrations, you can review my earlier post here. Lately, migrations have been more focused because the majority of them involve moving content out of the Oracle IPM 10g or Oracle Imaging 11g products. Oracle IPM 10g has reached end-of-life, and Oracle Imaging 11g was the terminal release of the product, so essentially that product line is dead. The IPM 10g product was something we worked with for many years, so we have a wealth of knowledge on its “ins and outs.” IPM was a feature-rich, but older, product stack, and it was in need of a rewrite. However, when Oracle rewrote the product as Imaging 11g, there were a lot of key and important features that didn’t make the cut.

Because of everything I’ve mentioned, businesses running on these particular Oracle ECM platforms have had to make decisions for their long term ECM vision or roadmap. I have worked with a number of clients on technology evaluations and like to help determine their roadmap, but that’s a blog post for another time. One of the key pieces to any ECM roadmap for a company performing  these ECM solution changes is the migration of their content from the systems that they are replacing. Luckily we have ILINX Export and ILINX Import to make these migrations straightforward as possible.

There are a number of options with ILINX Export, but in short we use it to export all content and metadata out of a source system for migration into any destination system. By default, ILINX Export retrieves the content from the source system in exactly the same format it was in when it was added to the original system. By exporting the content out in its native format, a customer can keep a copy of the original data, and any data manipulation or file conversions can be done downstream. ILINX Export does have the ability to convert files to PDF, but we generally recommend image conversion when importing the content into the destination system. Utilizing our knowledge of Oracle ECM, there are plenty of options when extracting the content from these products. For example:

  • Only migrate certain applications.
  • Only migrate content created after, or before, a certain date.
  • Only migrate the content that falls within certain criteria: i.e., specific business unit, a set of document types, or virtually any criteria that can be identified with the content metadata.
  • Split the content up so content that meets certain criteria goes to one destination, and content meeting other criteria goes elsewhere.
  • Retain the IPM, or Imaging, annotations. These can be flattened into the documents, but I only recommended that in certain instances. If the client is migrating to ILINX Content Store, we can migrate the annotations as an overlay into the new ILINX system.
  • There are many options for formatting the data when it its exported from the source system. ILINX Export can output the metadata to text or XML files with complete control of the format, delimiter, field order, layout, and size of those files. That flexibility allows for the creation of input files in a format that can work for just about any destination system.
  • The metadata can also be written directly to SQL to support long term storage or manipulation if necessary.
  • Schedule the export to run during off-hours to keep the load off the servers while clients are using the old system.
  • Detailed auditing of the entire process to help with reporting, compliance and troubleshooting.
  • Many more.

Once you’ve defined all the rules surrounding the migration and started execution, the next step is importing that content and metadata to the destination ECM system. For that, we use our ILINX Import product which I’ll cover in a later post. If you have any questions about ILINX Export, reach out to us for a demo or discussion.

John Linehan
Sr. Systems Engineer
ImageSource, Inc.

1 2 3 11