User Interface
This is how Xade Leonardo currently looks:

(The Swedish flag should not be there in the final version ;-) )

Since I want to minimize UI clutter I have thought long and hard over what should be visible at all time. I have come up with the following things: Current tool, tool help, current color, file size, zoom level, currently selected layer and if Leonardo is currently doing some background work. You don’t need to show the current brush and radius since the cursor already contains that information.

Although the Leonardo engine can handle Layers, the UI doesn’t show you any information about them right now. So far, my best idea for this is to put small tabs on the right hand side of the canvas where the Layer name is written vertically to save space. I am planing on making layers a “first class citizen” so that it will be possible to drag-n-drop layers between tabs, drop files in and out as layers including recently exported files from the Export bar (not shown in picture above).

One thing that hit me recently is that you want to avoid having sliders on the left and upper part of the screen. This becomes apparent when you use a tablet with a built-in display. Your arm will then cover most of the screen while you are adjusting the slider something you obviously want to avoid.

Color Spaces and Gamma
A couple of weeks back I spent some time teaching myself about Color spaces and Gamma correction. I had some prior knowledge of this but if you would have asked me: “Why doesn’t a standard HSV-Hue shift preserve luminosity?” I would have no good answer. Now I know the answer and I am planning on addressing it in Leonardo among a whole host of other issues. What is amazing to me is that not even Photoshop manage to do all this correctly. I guess they know all these at Adobe but they are stuck with what they got because of backward compatibility issues.

My current thinking is having Leonardo work in an absolute color space like sRGB or AdobeRGB as opposed to just “random”-RGB and storing pixels in linear-space as opposed to Gamma-space. I also hope to be able to do some of the pixel operations in LUV-1976 space (like Hue shifts) which I find a really nice color space although there might be some problems with out-of-gamut colors.

Everybody is familiar with a tone-histogram (the one you get in a digital camera or under Photoshop Levels) which mostly is used for setting the black- and white-point of an image. A couple of days ago I had a crazy idea on taking this to the next level with a density or contour plot of the chromaticity of an image. I think this will be an awesome visualization for Hue/Saturation and Color-correction style adjustments and it will make it obvious to a novice user why a Hue-shift is a modular adjustment.

Destructive vs. Non-destructive editing
Over the past 5 months I must have spent over 60 hours just thinking about destructive vs. non-destructive editing (my favorite occupation while taking a walk along the lake). Now, Leonardo is primarily a destructive image editor but since you want some form of synthesis between different layers the question is how far you go down the path of non-destructive editing? Do you allow blend modes? Do you allow procedurally generated layers?  Do you allow vector layers? Do you allow non-destructive adjustment layers? Do you allow non-destructive warps? Do you allow visibility masks? Do you allow “layer styles”? All of these are still open questions…

Another problem related to this is my personal disgust about “blending modes”, I understand they are extremely powerful for the expert user, but even me, with a strong mathematical background, can’t use them intuitively! On the other hand, I haven’t come up with a good alternative :-(

Node recursion
In a previous blog post I talked about switching to a fixed root system, well, I switched back! I found a way to solve the problem and still keeping a non-fixed-root which I am really satisfied with.

While we are on the topic of node recursions, this is one of the most beautiful things I have ever written:

struct node_s {
    unsigned int cb : 4;
    unsigned int id : 28;
    struct node_s *childs[0];
};

node = node->childs[ bitcount[ node->cb & ((1<<c) - 1) ] ];

(node->bash is a compact child pointer list, node->cb is child-bits and c is the child number you want to get to)

The node data have a very small footprint (notice the struct-hack), it’s super fast and yet the whole thing is relatively simple. Storing you node meta data in this way only takes a fraction of the space it otherwise would! Unfortunately I use quite big nodes these days (128×128 pixels) so this doesn’t really matter that much anymore :-(

So the problem is disk vs. main-memory, which one should you optimize for? or can you find a workaround?

After spending a couple of days trying to come up with a good solution for minimizing disk IO, while maintaining small nodes (32×32 or 64×64), I finally gave up since all my solutions was immensely complex and the disk IO was still was pretty bad. Instead I decided to set the node size to whatever is optimal for disk and then try to work around the problems that would arise in memory. I knew the optimal node size for disk was either 128×128 or 256×256. What was surprising though was that 256×256 only generated about 60% of the IO requests compared to 128×128 where you naively would have expected ~25% and since the bandwidth and file-size goes up with 256×256 I decided against it and set the final node size to 128×128.

To solve the memory footprint problem I introduced nodes in different pixel precision. Doing it this way means that nodes that should only be displayed can stay in the same precision as on disk while nodes that should be edited can be converted to RGBA_FP (the main reason I use RGBA_FP is to spare myself all the fixed-point math headaches during development). If you start running low on memory you can start converting nodes back to file precision before you finally clean and unmap them.

A side effect of using large node sizes is that the “TOC” (Table Of Content) for each file gets a lot smaller which allows you to store hash values in them (somewhat similar to a .torrent file). This turns out the be extremely convenient if the user stores his data in the cloud since everything can then be fetched from a local cache file eliminating 99% of all downstream traffic, and since Amazon don’t charge you for upstream traffic you have effectively eliminated 99% of the bandwidth cost of using Amazon S3! (of course if the user change host, all data must be downloaded again but that should be quite rare).

1. Fixed the bug where future snapshot node references got screwed during node spawn.

2. Swapped to a fixed root system.

The first breakthrough was not resolved until I finally drew a 1D version (a binary tree) of the problem on paper and walked trough each step that needed to happen.

The second breakthrough allowed me to use Morton keys to identify nodes which have tonnes of benefits when we need to locate all nodes that share a particular place in space. Last spring, during my master thesis on Octrees, I did try the fixed-root/Morton-key implementation but decided against it. This time around it looks much more promising for the following reasons:

1. Each node stores a grid of pixels/voxels so the quadtree/octree never have to map down to individual pixels/voxels.

2. We don’t use any ray tracing now which was the biggest reason for keeping the height of the quadtree/octree small.

3. With a quadtree and 64^2 pixels per node we can fit a Morton key inside a single 32-bit word.

One small drawback of using a fixed root system is that the canvas is not technically ”infinite” anymore. My current implementation clamp the canvas size to 2 million by 2 million pixels which would be enough for most users (going higher then this would, among other things, require me to switch to doubles instead of floats inside the engine).

There is at least another week of engine work and then maybe 2~3 weeks of client work before I am finally done with my Leonardo prototype! Stay tuned!

This is the current look of Xade Leonardo:

(the edges around the xade drawing look bad because of the current flawed implementation of alpha compositing)

Here are some of the features not visible on the above picture:

1. You can toggle a Quake style console between the tab strip and the menu.
2. Below the status bar you can toggle a Chrome like “download bar” where files that recently have been exported appear which then can be dragged-n-dropped to other applications.
3. You can toggle a pie menu that appear centered around the cursor with the most basic features for the current tool.

The UI is, among others, inspired by Chrome, Silo, Quake, Nuke, Mari, Google Docs & Facebook.

The engine code starts to come together for the most basic functionality, but there are probably many more weeks before the more complex stuff is working.

Right now there is only one tool: the paint brush! In the next couple of weeks I will add a few more and my current plan is to add the tools/filters which is the most likely to break the current engine architecture first and save the “easy” ones for last. I will probably implement them in the following order:

1. Export tool (should work as a background process for large images).
2. Eyedropper (pretty easy, but one of the few tools that needs a round trip to the server).
3. Selection tool (very different from all other tools).
4. Gaussian blur filter (a good way to stress test the engine in adaptivity and speed).
5. Clone tool (probably needs reading and writing to the same snapshot which might be tricky with SMP).
6. Smudge tool (needs some kind of sampling from the current snapshot)
7. Liquify tool (very different from other tools since it works on polygons instead of pixels).

Voxel Engine
I have build a voxel engine around the concept of Sparse Octrees where each leaf store 16^3 grid of voxels. Each voxel is represented by a 16-bit  ”iso”-value and then 8-bits each for R G B and specular. This works great and by storing an array of voxels in each leaf we avoid the pressure of minimizing overhead in the node data structure.

Right now the voxel engine is split on a Server and a Client side which run on separate threads. The server side is responsible for disk read/write, voxel edits and tessellation. The client run in the same thread as the rest of the application and handles rendering and ray tracing. The main reason for this split is to avoid hick-ups in the thread that runs the graphics but have also some really nice side benefits like parallelization.

Disk storage
I have already rewritten this part of the code 3 times. Right now I use a solution based on Sqlite which gives me atomic commits. This is really a must when we handle files of several 100′s of MB. I don’t really use any of the RDBS stuff that comes with SQL, I just want to store tens of thousandths records of varying size with an unique 64-bit identifier in a single file with transactions. I tried to write this kind of system myself first and although it’s easy to get it to work, it’s very difficult to guarantee that data never gets corrupt on a crash. It is really convenient to use a 64-bit node identifier over a 32-bit since we then don’t have to reuse any identifiers.

Rendering
One of the main things I have researched the last 9 weeks is the rendering. I have come to the some what boring conclusion that Marching Cubes probably is the most efficient and artifact free solution. Voxel ray tracing looks promising at first but using ray marching directly against the voxels is waaaaay to slow and converting the voxels to a discrete SVO gives us the ugly Lego-artifact. Building a renderer based on splats is possible but a Marching Cube based solution is not much more expensive and looks a lot better. Since we use an Octree to store the nodes it should still be possible to ray trace against our marched cube triangles in real-time using GPGPU.

Right now I use vertex buffers objects and store all the indexes in CPU memory. This minimize the amount of draw calls we have to make but increase the data traffic over the PCI. This gives us about 0.7 million polys at 60 Hz. When I get back I will try to push this higher by experimenting with index buffers and glDrawMultiElements. Since I’m planing to relay only on vertex colors for texturing we want to be able to render more triangles then there are pixels on screen.

One big advantage with raster based data over vector data is down sampling. This works really well but when you do this adaptively with Marching Cubes you get pixel cracks between different resolutions. Hopefully this can be fixed by introducing “skirts” on all cubes but I haven’t tested this yet.

To calculate normals you can either use the voxel gradient or calculate it from the triangles. I have tried both and come to the conclusion that calculating it from the triangles probably is better but I’m 100% sure yet.

Widgets
I have written a widget system for UI rendering that works amazingly well. One of the core features is how I handle the XYWH coordinates for all the widgets. I basically works like this: All coordinates are relative to the widget parent. There are 2 types of coordinates, one for window area and one for client area. This coordinates can be mounted to the left or right edge. Finally we store all the coordinates on a file and allow them to be changed through the UI which allows you to try different UI layouts without ever recompiling or restarting the application.

Future:
Below is a list of what remains to be researched when I get back home:

• Procedural texturing.
• Automatic UV-mapping in tessellation code.
• Prioritize tessellation front-to-back.
• Marching Cube skirts.
• Use odd number of voxels with redundancy per node?
• Parallelization of tessellation.
• Locally change voxel resolution.
• Optimize away solid and empty spaces.
• Parallelization of voxel edits.
• GPGPU voxel edits.
• Adaptive voxel edits.
• The Level Set Method for deformation.
• More then one voxel object per file.
• Solidbits.

Using triangles for rendering is kind of boring compared to ray tracing but for the moment it seems to work out a lot better. Ray casting directly against the voxel array is slow and using SVO is kind of ugly if you don’t use post process blurring.

This means that Goal 1 of the this project has to change a little. Here is a updated list of the features the future prototype should have:

1. A Storage Engine that can handle think surfaces and normal calculations well.
2. Basic tools for voxalizing solids such as boxes and spheres.
3. Basic carve, sculpt and smooth brushes.
4. A Renderer build on Marching cubes with no visual artifacts.
5. A development framework: console, command prompt and basic UI.

This prototype is still a couple of weeks away. Hopefully I will make it before I travel to Nepal (middle of November).

The storage engine is still build around an octree, where each node store a 8^3 voxel grid. This gives tones of benefits: mipmapping, recursive culling, efficient space mapping, fast ray tracing etc.