We've just finished merging back a number of changes to OysterKit following the work done on performance and simplification. We've even had a chance to tweak the demo project so that you can now fully define your own tokenisation and colouring scheme and watch the impact live on some text you provide.
We are doing some more research before we write this up, but the whole effort proved very interesting. Essentially we have changed the way the tokenizer works in order to make it much simpler to customise and extend the behaviour. In doing this we have also been able to improve the ptimised performance dramatically (it's 98.7% faster)... unoptimised performance remains unchanged!
Here's the change log
- Extended OK script to support named re-usable states
- Added new tokenisation states to make it easier to deal with complex situations that would otherwise have required back-tracking. The API for those simply using OysterKit is unchanged, just build against the new version.
- Started preparing for features to make it very easy to add syntax highlighting to any NSTextView, we also intended to build in OK Script tokenizers for many common languages. We think this is the most common use-case.
- Performance (when built with optimised settings) is transformed, although as we build a re-usable syntax highlighter we will take huge files into account (push tasks to background threads, tokenise what's visible/changed first).
- Updated the Tokenizer demo target to let you completely define your own syntax highlighter.
We hope you find this a useful update, and do let us know what you think. Find out more about OysterKit here