Leonardo is getting closer and closer to the point where you can use it for real work, but there are still a few things missing:

1. HUD-notifications (worked a long time ago but the code has decayed)
2. Custom brush-stamps as opposed to just ellipses.
3. Add, remove and rearrange brush-preset list.
4. Merging and changing Z-order of layers.
5. Better UI for layers.
6. Drag-n-drop from the export bar.

All user tests were conducted in full-screen mode which meant that no user could access the menu bar (this should be changed until next user test so that the menu bar “auto hides” in full-screen mode).

1. All participants requested layers (we have layers but no UI for it, yet).
2. Although no participants explicitly said so, all of them would have enjoyed a brush-preview when they where adjusting the brush settings.
3. All of the participants instantly understood how to use the new color picker (wheel + triangle) and two of the participants loved it more then any other color picker they have ever used.
4. All participants seamed to like the new default eraser (its own context plus pressure sensitive diameter).
5. Two of the participants complained that the diameter slider was “too sensitive to get the exact diameter you want” and one of them requested being able to enter the diameter numerically from the keyboard (current diameter slider: 207 pixels wide, from 0.5 to 1500 with gamma 2.5).
6. Two of the participants did not know what the difference between Flow and Opacity is and could not figure it out even after some testing.
7. All of the participants used the Hardness parameter quite a lot.
8. Two of the participants requested being able to change the background color (currently possible, but only from the console).
9. Two of the participants asked if it would be possible to select and move a portion of the canvas (not currently possible).
10. Two of the participants tried to hit the Z key when they were told to zoom (doesn’t work).
11. One participant tried to “double click” the space-bar when told to snap the view (doesn’t work).
12. None of the participants noticed the “zoom box” at the lower left corner even after they were asked to zoom.
13. All participants seamed to like the rotate-canvas-feature and understood how to move the pivot point.
14. One of the participants asked “what is the current DPI?” (he knew he was at 100% zoom but he didn’t realize that’s how you are suppose to know the resolution you are at).
15. All of the participants loved the draw-straight-line feature but requested more advanced functionality. One of the participants (Kalle) requested being able to draw curves in a similar fashion and told us it’s very important to be able to offset the curve, without changing it, to draw repeating patterns.
16. Two of the participants requested being able to get some sort of “A4-paper reference” that just hovered above the canvas.
17. None of the participants realized that you could move around the color picker and brush settings window.
18. We forgot to test if the user realized that it was possible to pan around the canvas using the rulers.
19. One of the participants thought the F key would be more logical for flip-horizontal then the H key.

1. Martin loved being able to rotate the canvas but didn’t like the shortcut Space+Shift. Martin usually rotated the canvas in a way that made it possible to move the hand in a back-and-forth (as opposed to left-right) direction while painting.
2. Martin loved the draw straight line tool and was able to find the shortcut himself (Shift).
3. Martin used the eraser a lot, both as a painting effect and a way of undoing mistakes. Martin likes to work with an eraser with a large diameter as well a smaller one.
4. Martins brush setting of choice for sketching was a hard round 24 pixel wide brush at 36% zoom with a light blue color.
5. Martin liked to outline his sketch with a hard black brush with a diameter that depends on pressure (not possible on the current version).
6. Martin requested the “multiply blending mode” between layers.

Yesterday, Daniel & Max was over and we did a user test on Daniel. Here is a list on things that probably should get addressed:

1. Daniels initial reaction after lunching the application was something along the line: “What tool, brush and color do I currently have and how do I change it?” Daniel eventually figured all of this out, but it was not obvious from the start.
2. When Max was experimenting around in Leonardo he zoomed out to ~15% and continue painting without realizing that he was at ~15% zoom. This was never a problem for Daniel though.
3. Daniel wanted to draw a straight line but never figured out how to do it (it is obviously impossible for the user to know to hold down Q). Once I told him how to do it he instantly loved the way the “draw straight line tool” worked.
4. When Daniel got the task: “convert your current painting to gray-scale” he did not realize that the command “Luminosity” under Filters does this.
5. Both Daniel and Max wanted to create a new Canvas and start searching for “New…” under the File menu, but is not there…
6. By mistake Max eyedropped white, he then resumed painting but did not realize he was painting with white and asked why nothing was happening.
7. When Daniel was switching to Finite Canvas he thought it was strange that the canvas boundary did not get aligned with his current view.
8. When Daniel should flip the image he flipped it vertically when he was supposed to flip it horizontally (an icon should probably make this much more obvious).

User Interface
This is how Xade Leonardo currently looks:

(The Swedish flag should not be there in the final version ;-) )

Since I want to minimize UI clutter I have thought long and hard over what should be visible at all time. I have come up with the following things: Current tool, tool help, current color, file size, zoom level, currently selected layer and if Leonardo is currently doing some background work. You don’t need to show the current brush and radius since the cursor already contains that information.

Although the Leonardo engine can handle Layers, the UI doesn’t show you any information about them right now. So far, my best idea for this is to put small tabs on the right hand side of the canvas where the Layer name is written vertically to save space. I am planing on making layers a “first class citizen” so that it will be possible to drag-n-drop layers between tabs, drop files in and out as layers including recently exported files from the Export bar (not shown in picture above).

One thing that hit me recently is that you want to avoid having sliders on the left and upper part of the screen. This becomes apparent when you use a tablet with a built-in display. Your arm will then cover most of the screen while you are adjusting the slider something you obviously want to avoid.

Color Spaces and Gamma
A couple of weeks back I spent some time teaching myself about Color spaces and Gamma correction. I had some prior knowledge of this but if you would have asked me: “Why doesn’t a standard HSV-Hue shift preserve luminosity?” I would have no good answer. Now I know the answer and I am planning on addressing it in Leonardo among a whole host of other issues. What is amazing to me is that not even Photoshop manage to do all this correctly. I guess they know all these at Adobe but they are stuck with what they got because of backward compatibility issues.

My current thinking is having Leonardo work in an absolute color space like sRGB or AdobeRGB as opposed to just “random”-RGB and storing pixels in linear-space as opposed to Gamma-space. I also hope to be able to do some of the pixel operations in LUV-1976 space (like Hue shifts) which I find a really nice color space although there might be some problems with out-of-gamut colors.

Everybody is familiar with a tone-histogram (the one you get in a digital camera or under Photoshop Levels) which mostly is used for setting the black- and white-point of an image. A couple of days ago I had a crazy idea on taking this to the next level with a density or contour plot of the chromaticity of an image. I think this will be an awesome visualization for Hue/Saturation and Color-correction style adjustments and it will make it obvious to a novice user why a Hue-shift is a modular adjustment.

Destructive vs. Non-destructive editing
Over the past 5 months I must have spent over 60 hours just thinking about destructive vs. non-destructive editing (my favorite occupation while taking a walk along the lake). Now, Leonardo is primarily a destructive image editor but since you want some form of synthesis between different layers the question is how far you go down the path of non-destructive editing? Do you allow blend modes? Do you allow procedurally generated layers?  Do you allow vector layers? Do you allow non-destructive adjustment layers? Do you allow non-destructive warps? Do you allow visibility masks? Do you allow “layer styles”? All of these are still open questions…

Another problem related to this is my personal disgust about “blending modes”, I understand they are extremely powerful for the expert user, but even me, with a strong mathematical background, can’t use them intuitively! On the other hand, I haven’t come up with a good alternative :-(

Node recursion
In a previous blog post I talked about switching to a fixed root system, well, I switched back! I found a way to solve the problem and still keeping a non-fixed-root which I am really satisfied with.

While we are on the topic of node recursions, this is one of the most beautiful things I have ever written:

struct node_s {
    unsigned int cb : 4;
    unsigned int id : 28;
    struct node_s *childs[0];
};

node = node->childs[ bitcount[ node->cb & ((1<<c) - 1) ] ];

(node->bash is a compact child pointer list, node->cb is child-bits and c is the child number you want to get to)

The node data have a very small footprint (notice the struct-hack), it’s super fast and yet the whole thing is relatively simple. Storing you node meta data in this way only takes a fraction of the space it otherwise would! Unfortunately I use quite big nodes these days (128×128 pixels) so this doesn’t really matter that much anymore :-(

I have defined 3 primary user cases for Leonardo:

1. Lean Forward – the user sits in front of his PC or Mac with his right hand on the Wacom and the left hand on the keyboard. The user might drag-n-drop tabs and files, using the built in console and other “advanced” stuff.
2. Lean Backward – the user sits in a sofa with a wireless Wacom in his lap while doodling on his big screen TV. The user might not have “easy” access to the keyboard and the TV might be far away which makes small things harder to see.
3. Doodling in a public place – the user is doing some rough sketching on his iPad on the subway, in a cafe or public park. When the user gets back home he wants to continue on his PC or Mac with minimum fuzz.

I believe the second user case really is the future of digital doodling and less hardcore painting. I realized this after seeing two of my friends throwing out their stationary PCs and now is doing everything in front of their TVs with a wireless keyboard and mouse.

This has convinced me to buy a big screen TV, a Mac Book Air, an iPad and a wireless Intuos.

So the problem is disk vs. main-memory, which one should you optimize for? or can you find a workaround?

After spending a couple of days trying to come up with a good solution for minimizing disk IO, while maintaining small nodes (32×32 or 64×64), I finally gave up since all my solutions was immensely complex and the disk IO was still was pretty bad. Instead I decided to set the node size to whatever is optimal for disk and then try to work around the problems that would arise in memory. I knew the optimal node size for disk was either 128×128 or 256×256. What was surprising though was that 256×256 only generated about 60% of the IO requests compared to 128×128 where you naively would have expected ~25% and since the bandwidth and file-size goes up with 256×256 I decided against it and set the final node size to 128×128.

To solve the memory footprint problem I introduced nodes in different pixel precision. Doing it this way means that nodes that should only be displayed can stay in the same precision as on disk while nodes that should be edited can be converted to RGBA_FP (the main reason I use RGBA_FP is to spare myself all the fixed-point math headaches during development). If you start running low on memory you can start converting nodes back to file precision before you finally clean and unmap them.

A side effect of using large node sizes is that the “TOC” (Table Of Content) for each file gets a lot smaller which allows you to store hash values in them (somewhat similar to a .torrent file). This turns out the be extremely convenient if the user stores his data in the cloud since everything can then be fetched from a local cache file eliminating 99% of all downstream traffic, and since Amazon don’t charge you for upstream traffic you have effectively eliminated 99% of the bandwidth cost of using Amazon S3! (of course if the user change host, all data must be downloaded again but that should be quite rare).

In the last couple of weeks it have started to crystallize for me exactly what I have been doing for over a year now: I have researched and developed a kick-ass quad-tree/octree engine and soon it is time to tap the potential of this engine by starting a company and develop a product around it. The engine can be used for at least two different products with at least three different platforms on each product:

1. Xade Leonardo – pixel painting and sketching for the PC, Mac and iPad.
2. Xade Michelangelo - voxel sculpting and painting for the PC, Mac and iPad.

On top of this, the engine allows for very efficient streaming of data between client and server so a cloud based storage solution would be possible with a revenue model similar to DropBox. Now of course I will not build all the above permutations at once. The Leonardo-PC version have the highest priority mainly because pixels are easier then voxels and I am more familiar with the PC compared to the Mac and iPad.

But the core value of Xade Software is in the engine technology, which, to my knowledge, is unique in the following areas (at least compared to Photoshop):

1. Infinite canvas — just pan the view in any direction and keep painting.
2. ‘zero’ load/save time — even if the file is thousands of MB it loads and saves under a second.
3. zero lag for brushes/filters — even if you use a complex brush that is hundreds of pixels in diameter there is no hick-ups or lag.
4. Cloud storage — My streaming data technology means you can work over a relatively slow network connection (ex. 3G) without any lag which means storing your data in the cloud would be possible.
5. Flexible canvas history – You don’t loose the history when you close a file and the history is branched instead of just linear.

On top of this there is of course tonnes of minor features, and I am also working hard on the non-engine stuff…

So, if a technical person ask, what is Xade Software? I will answer:

“We are experts in quad-tree/octree technology which we use to build software that handle large data sets of raster-based graphics. Our first product is Xade Leonardo which is painting application that focus on the experience of digital painting“

And if a non-technical investor/banker ask, I will answer:

“We have developed a technology that is really revolutionary which we use to build an image editor, streamlined for painting and sketching, that is better then anything else that is out there, including Photoshop”

And if a customer ask, I will answer:

“We are the creator of Xade Leonardo which is a revolutionary new painting and sketching application for the PC, Mac and iPad. Our primary focus for Leonardo is the experience using it as opposed to a gazillion features. Just try it for 5 minutes, it is super intuitive, you will never want to go back!”

1. Fixed the bug where future snapshot node references got screwed during node spawn.

2. Swapped to a fixed root system.

The first breakthrough was not resolved until I finally drew a 1D version (a binary tree) of the problem on paper and walked trough each step that needed to happen.

The second breakthrough allowed me to use Morton keys to identify nodes which have tonnes of benefits when we need to locate all nodes that share a particular place in space. Last spring, during my master thesis on Octrees, I did try the fixed-root/Morton-key implementation but decided against it. This time around it looks much more promising for the following reasons:

1. Each node stores a grid of pixels/voxels so the quadtree/octree never have to map down to individual pixels/voxels.

2. We don’t use any ray tracing now which was the biggest reason for keeping the height of the quadtree/octree small.

3. With a quadtree and 64^2 pixels per node we can fit a Morton key inside a single 32-bit word.

One small drawback of using a fixed root system is that the canvas is not technically ”infinite” anymore. My current implementation clamp the canvas size to 2 million by 2 million pixels which would be enough for most users (going higher then this would, among other things, require me to switch to doubles instead of floats inside the engine).

There is at least another week of engine work and then maybe 2~3 weeks of client work before I am finally done with my Leonardo prototype! Stay tuned!

This is the current look of Xade Leonardo:

(the edges around the xade drawing look bad because of the current flawed implementation of alpha compositing)

Here are some of the features not visible on the above picture:

1. You can toggle a Quake style console between the tab strip and the menu.
2. Below the status bar you can toggle a Chrome like “download bar” where files that recently have been exported appear which then can be dragged-n-dropped to other applications.
3. You can toggle a pie menu that appear centered around the cursor with the most basic features for the current tool.

The UI is, among others, inspired by Chrome, Silo, Quake, Nuke, Mari, Google Docs & Facebook.

The engine code starts to come together for the most basic functionality, but there are probably many more weeks before the more complex stuff is working.

Right now there is only one tool: the paint brush! In the next couple of weeks I will add a few more and my current plan is to add the tools/filters which is the most likely to break the current engine architecture first and save the “easy” ones for last. I will probably implement them in the following order:

1. Export tool (should work as a background process for large images).
2. Eyedropper (pretty easy, but one of the few tools that needs a round trip to the server).
3. Selection tool (very different from all other tools).
4. Gaussian blur filter (a good way to stress test the engine in adaptivity and speed).
5. Clone tool (probably needs reading and writing to the same snapshot which might be tricky with SMP).
6. Smudge tool (needs some kind of sampling from the current snapshot)
7. Liquify tool (very different from other tools since it works on polygons instead of pixels).