Topic: performance

Hi Tim Zimmerman,

There's different performance characteristics based on which components you add to your job. Especially which analyzers you use can have a big impact.

Expect very-near-linear time on analyzers such as String analyzer, Number analyzer, Date/time analyzer and Boolean analyzer.

Depending on the data, something like the Pattern finder can either be near-linear in good cases (if most values have the same pattern) or be quite slow in bad cases (if there are lots and lots of patterns).

Most transformers and filters are linear-time, since they are completely stateless. A few exceptions exist though, like the Table lookup which has a cache and therefore it depends on the amount of repetition.

Which analyzers are you using?

Usually I am running with DC's default configuration (30 threads) on my 8gb ram / Intel core i5 machine.

PS: I assume by "MM" you mean millions? Usually I would only write "M" though, so I hope I am completely oblivious to what you're talking about :P

Aha, yes the value distribution is also a pretty tricky one. In deed it is not linear time! But there are tricks you can apply...

Basically the reason it is not linear is that we need to save the temporary results on disk. This is because we want DC out-of-the-box to not crash due to outofmemory problems.

BUT, if you have a lot of memory you can change the strategy! The trick is to open conf.xml (in the DC dir) and to replace this element:

<storage-provider>
  <combined>
    ...
  </combined>
</storage-provider>

With this:

<storage-provider>
  <in-memory max-rows-threshold="1000"/>
</storage-provider>

Let me explain: The value distribution needs somewhere to hold it's state, like "I've so far seen value X 5 times and value Y 7 times". But if a dataset contains eg. ONLY UNIQUE values and you have millions of records, then this approach would quickly run out of memory! So as a fail-safe precaution, we save that data in an embedded database as we go. But of course a database like that will get slower and slower, thus the non-linearness.

The trick I described will instead of the database use an in-memory approach. This is vulnerable to out-of-memory issues if you're short on memory, but if you do expect to see repeated values (even say 10's of thousands of distinct values, but not millions of distinct values) then it will work fine! Additionally you can trim the JVM to assign more memory, so if you have a big machine, almost everything is doable in that fashion.

Regarding threads: Don't increase it, since I think with your usage 30 is quite enough. Having too many threads can also harm performance since they might just all be waiting (and thereby wasting performance because of the thread management overhead) for access to the same resource (the state of the analyzer).

PS: Thanks for the kind words :)