Speeding up development tests with cached data

Speeding up development tests with cached data

One of the most exciting things about working with FME is creating new workspaces, but this process is not without its challenges. Don’t worry, though—FME is about to spare you the pain of several long, repetitive tasks you would otherwise have to do yourself.

Testing workspaces can be time consuming. You have to adjust parameters, add/delete certain transformers, and run the process over and over until you achieve the desired result. Sometimes the data is very bulky, and sometimes data processing is complex or just takes a long time to run. Either way, it’s pretty tedious to have to test the whole workspace every time you tweak a parameter.

What if you could test only the section you’ve modified? FME Desktop 2018 introduced a new functionality: Feature Caching. Feature Caching uses cached data generated by FME when running a workspace to:

1)    View the data at every step of creating a workspace
2)    Do partial runs of the process

This feature greatly speeds up workspace development.

To enable it, click on the magnifying glass icon on the toolbar.

1. View the data at every step of creating a workspace

The first step in using Feature Caching functionalities is to run a workspace once through. Once the workspace runs to completion, little green magnifying glasses appear next to each transformer and reader output port:

Simply click a green magnifying class to “inspect” the data at any time.

This feature saves time by eliminating the need to add an Inspector everywhere you want to view data, as was necessary in previous versions.

2. Do partial runs of the process

Recording data in cached memory also means you can run only certain parts of a workspace (partial runs), which is a big time saver: up until the 2018 version of FME, users had to run workspaces from the very beginning to test each change they made.

So, for example, if you made a change to AreaCalculator in previous versions of FME, you had to disable the next transformer and add an Inspector to AreaCalculator to see the result of your process.

Now, with Feature Caching, if you make a change to AreaCalculator, you have the option of running only the transformation generated by this transformer by selecting “Run To This.” First, choose the transformer and the option will pop up above it. Once a change has been made, the green magnifying glass turns yellow, which indicates that the cached data no longer reflects the result of the transformer’s parameters.

Choosing “Run To This” runs only the part of the process highlighted in green (see above), which is the section to be completed since the last time the data was cached. If a change had also been made to AttributeManager, the green section would be longer.

Next, you can view the result by clicking on the green magnifying glass. In this case, the processing time fell from 0.9 seconds for a full run to 0.2 seconds for just AreaCalculator—not a huge difference in this example because the beginning of the process is fairly light to run, but this option can often slash processing time significantly.

Another option is available right next to “Run To This”: a complementary feature called “Run From This.”

“Run From This” lets you run the part of the process that immediately follows the selected transformer. Once again, the part of the process affected by the operation is highlighted in green.

To summarize, here are the best uses for Feature Caching:

• In the development phase only. Once you hit the production phase and the workspace is up and running, there’s no use keeping all the cached data.
• When the workspace takes a long time to run and you’re testing changes made toward the middle or end of the workspace. This way, you’ll save all the time you need to complete the first steps of the workspace.
• Web requests (for example, through HTTPCaller) are good opportunities to use Feature Caching in the development phase. In previous versions, you had to redo the HTTP request each time; now, with Feature Caching, you can test and complete the rest of the workspace without running the same request each time, which can save you a few valuable minutes here and there.

With Feature Caching, avoid: 

Oversized datasets. Accumulating thousands of features in memory will slow down the workspace’s performance. If the datasets are large, you’re going to want to limit the number of features to be read at the development phase (the “Max Features To Read” reader parameter in the Navigator).