May 1, 2008

Type safe / refactor friendly configuration for Db4o

As of today, Db4o uses strings to reference fields in some scenarios such:
  • SODA queries
  • Configuration

See the sample bellow:

IConfiguration cfg = Db4oFactory.NewConfiguration();
cfg.ObjectClass(typeof(Test)).ObjectField("_age").Indexed = true;
IMHO, using strings in these situations is sub-optimal as:
  • Such solution relies in internal class details (field names)
  • Compiler can't check whenever the field actually exists.
  • We don't take advantage of modern IDE's refactorings so, chances are that these entities and references to them get out of sync breaking your code; it's even worst because you will figure it out only at runtime :(; it may even get unnoticed as Db4o is completely happy with some invalid field names.
With the introduction of C# 3.0 and its automatic properties we have another issue: we don't even know the field's name used to back the property! Sure, we can cheat by using some tool like reflector or ildasm to figure it out but this would be a fragile solution as C# compiler may choose whatever name it wants, so it may choose different names schemas in the future and we don't want to get our code breaking when new C# compilers come out, do we?

I do believe that this approach was chosen due to technical limitations at the time the code was written, but now, with the debut of C# 3.0 and its new features like extension methods, lambda expressions, expression trees, etc. I bet we can do better :).

To be fair, we've already been improving (removing) some string usages, for instance, Linq / Native Queries uses strong typing instead of strings to reference fields (under the hood they generate SODA queries that still use strings, but the crux here is that these "names" will always be in sync with its entities).

One area that, in my opinion, can be improved is configuration. For instance, in order to setup indexes, deletion behavior (when to cascade delete), etc. we still use strings to identify the field we want to apply the configuration. Suppose we have the following code to configure indexes:

IConfiguration cfg = Db4oFactory.NewConfiguration();
cfg.ObjectClass(typeof(Test)).ObjectField("_age").Indexed = true;
We could introduce new methods to IConfiguration interface so one would be able to do something like this:
IConfiguration cfg = Db4oFactory.NewConfiguration();
cfg.ObjectField((Test s) => s.Age).Indexed = true;

Wow! Using this syntax we no longer have issues with refactorings, abusive internal class knowledge and we get compiler time support for free :) (for instance, if we mistype "s.Afe" C# compiler will complain).

To be fair, we have some issues with refactorings. If we change field/class's name we need to inform Db4o about this changes as explained here (I do have some ideas on how to express these calls also but this is a little bit more complex topic that I won't cover here).

What do you think?

In a following post I'll discuss how we could implement this new configuration method. Go to part II of this post. Adriano

10 years ago....

------------------------------ Warning: Personal post :) ------------------------------
Well, ten years ago (may, 1, 1998) I got married to Gislene, my wife :) 


Since then, she has been helping me, providing me advice, being patient with me and loving me! 
Gislene, thanks for all these years! I hope (and will do my best) to be at your side for the next 10 years, then for the next 10 and so on. 


I couldn't finish this without saying.. Gislene, I love you! 


Adriano

Apr 24, 2008

Redirecting standard error output in cmd.exe

Hi. Suppose you have the following C++ program (in a Windows box):
#include "stdafx.h"
#include "string.h"


void write(char *str, FILE *target)
{
fwrite(str, strlen(str), 1, target);
}

void _tmain(int argc, _TCHAR* argv[])
{
write("Test:stderr\r\n", stderr);
write("Test:stdout\r\n", stdout);
}
and you want to redirect its output to a file. We could try something like in Windows console:
test.exe > myfile.txt

Unfortunately the naive approach don't work because there are data being outputted to stderr and we are redirecting only the standard output (stdout). In order to redirect stderr we can use the following syntax:
command handler# > target

let's try a sample:
test.exe 1>myfile.txt

This will redirect stdout (handle 1) to myfile.txt To redirect stderr (handle 2) we can use:
test.exe 2>myfile.txt

Ok. Now we can try:
test.exe > myfile.txt 2>myfile.txt

but, again, this doesn't work; instead we get an error message:
D:\temp\cpp\Test\Debug>Test > myfile.txt 2>myfile.txt
The process cannot access the file because it is being used by another process.
Ok, there's one last trick we can try (duplicate stderr to stdout):
test.exe > myfile.txt 2>&1

Great! Now it works as expected! Last, but not least,
you can find more information here. Hope this can be useful :) Adriano

Apr 22, 2008

Db4o news!

Hi, Db4o just announced a new release! Check it out here and let us know what you think about these new (IMO) cool features! Adriano

Apr 9, 2008

A Linq playground for you and your objects

In this post we'll implement a very basic Linq Console application that supports "ad hoc" queries allowing users to enter Linq queries and run them against a db4o database. Our main goal is to show how such mechanism can be implemented on top of Linq/Db4o.

That said, let's play with the application for a while and them dig a little bit in its code.

How to use it ?

First, download the code here and compile it (any version of Visual Studio 2008 can be used).

Now populate a db4o database (with data) and copy the an assembly containing the types stored in that database to the LinqConsole.exe folder.


You will need to copy your model assembly to
LinqConsole executable directory, otherwise it
will not be able to find it.


Start the application and, once presented the prompt (
>), enter commands (prefixed with ":") or a query (finished with ";"). The very first command you may want to try is :help to see a list of valid commands (type :help and press ):
Basically, we want to run a (Linq) query against a db4o database, so we need to specify:
  • the database file to query (through database command)

  • the assembly containing the types stored in the database; we call this assembly, model assembly because usually, it will hold your business objects (the model in MVC pattern). Unfortunately, this assembly must be in the same directory as the application, otherwise the application will not be it able to find it (technically one can configure the application using Linq.Console.exe.config file to allow assemblies to be stored in alternative directories).
With these parameters defined you can type Linq queries as you would do pragmatically, for instance:

from Type t in database select t;

and press to see the results:


As the application tries to run the query only when it finds a semicolon ";" it's also possible to enter a query in multiple lines. You can see a short demo of the application in bellow.








Before leaving this topic, I'd like to call your attention that in order to Linq for db4o to work you need to specify a type in the query (that's the entry point for our custom "Linq Provider"), otherwise, you will get an error like the one bellow:


Note that in this case I've omitted Person type in the query, i.e, I've typed:

from p in database select p;

instead of:

from Person p in database select p;


How it works ?

The basic idea behind this application is pretty simple: Given a model assembly (specified through
load command), a db4o database (specified through database command) and a Linq query, find all types and namespaces defined in such assembly them emit code to run the query against the database.

In
CompileQuery() method we emit a new type (TheQuery) using the specified values (database, namespace and query). This type implements IInteractiveQuery interface, so, after instantiating it we call its Run() method to execute the query.

In order to dynamically compile this type (
TheQuery) we use classes defined in System.CodeDom.Compiler and Microsoft.CSharp namespaces; any compilation errors will be rendered in red on yellow background text.

Improvements

Well, I must admit: in its current state, this program has some limitations (but most of them are there to allow you to play with it ;)

From the top of my head, I can came up with a list (of coarse, not an exhaustive one) of possible improvements:
  • Support model assemblies in other directories
    Currently in order to use a model assembly, users are required to copy it to the application folder.


  • Check if the types stored in the database can be found in the model assembly.

  • Better error handling in general

  • Warn users about invalid commands Currently nothing happens for invalid commands (no message).

  • List types inside an assembly A new command (for instance, types) could be added to allow users to list the types contained in an assembly.

  • Add support to query Db4o in server mode.

  • Auto-completion This probably would require major changes in the input handling, but it would be nice to add auto-completion to list types, assemblies, databases, etc. in the commands.

  • You name it! The sky is the limit :)
Final Words!

I really expect that you enjoyed this application; if you have questions/suggestions please, fell free to drop me a line :)

Thoughts?

Adriano

Dec 10, 2007

Activation in depth

In this post I'll discuss a key concept of db4o and how it improved in our latest versions: object activation. I'll focus mainly on the .Net version but these concepts apply equally well to the Java version.

In order to access some resources listed in this post you will be required to register (for free) in http://developer.db4o.com.

Activation in depth

Activation in db4o parlance is the process of loading object data from the database to memory. You can see this concept in practice by asking db4o to not activate objects at all and having a look at them. Objects that are not activated will have only its object identity loaded from the database (In order to see the identifier for an object stored in db4o you can
 

IObjectContainer.Ext().GetId(obj);)

After building the graph of candidates (objects that satisfies a query's conditions), db4o needs to know how deep in this graph it must go activating (i.e, reading the object's data into memory). At this point it may seems that we have two contradictory objectives: i) we want to make db4o as transparent as possible to developers (making it easier to be used); ii) we do want to make sure that db4o will use computational resources (in this case memory) as efficiently as possible.

The simple approach to achieve i is to activate the whole graph but this clearly contrasts to ii because it may exhaust all free memory if the query returned a lot of objects. By the other hand, the easiest way to achieve ii (from the perspective of memory utilization) is to read only the identity of the objects in the graph and
leave to the developer the responsibility to activate them when needed. Again, this strategy goes against the other one (i.e, simplicity of use). db4o latest versions includes a new functionality (conceptually called Transparent Activation, or simple TA for brevity) aimed to achieve both goals (i and ii) without sacrificing each other.

Consider the following classes:

public class Person : SupportObjectId
{
    private Address address;

    public Address Address
    {
        get { return address; }
        set { address = value; }
    }

    private string name;

    public string Name
   {
        get { return name; }
        set { name = value; }
   }

   private int age;

   public int Age
  {
      get { return age; }
     set { age = value; }
  }

   private Person parent;

   public Person Parent
   {
       get { return parent; }
       set { parent = value; }
   }

   public Person(string name, int age, Address address, Person parent)
   {
        this.name = name;
        this.address = address;
        this.age = age;
        this.parent = parent;
    }

   public override string ToString()
   {
       return PrintInfo(0);
   }

   private string PrintInfo(int i)
   {
       string tab = new string('\t', i);
      return string.Format("{0} {4} {1}{5}\r\n{4}{4}Address: {2}\r\n{4}{4}Parent: {3}",
                                        (name ?? "null"),
                                        age,
                                        (address != null ? address.ToString() : "null"),
                                        (parent != null ? parent.PrintInfo(i+1) : "null"),
                                       tab,
                                      base.ToString());
    }
}

public class Address : SupportObjectId
{
    private string street;
    public string Street
    {
         get { return street; }
         set { street = value; }
    }

    private int number;
    public int Number
    {
        get { return number; }
        set { number = value; }
    }

    private string city;
    public string City
   {
        get { return city; }
        set { city = value; }
   }

    public Address(string city, string street, int number)
   {
        this.city = city;
        this.street = street;
       this.number = number;
   }

   public override string ToString()
   {
        return "[" + (street ?? "**") + "(" + number + ") - " + (city ?? "**") + base.ToString() + "]" ;
   }
}


and the following objects in the database:


private static void InsertData()
{
     using (IObjectContainer db = Db4oFactory.OpenFile(DATABASE_FILE))
    {
         Person grandParent = new Person("Joseph", 65, new Address("Descalvado", "JV street", 160), null);
         Person father = new Person("Adrian", 36, new Address("São Paulo", "C. L. E. M. street", 250), grandParent);
         Person child = new Person("Caroline", 4, new Address("São Paulo", "C. L. E. M. street", 250), father);

         db.Set(child);
    }
}

The following image shows the result when querying for "Car
oline" at different activation depths (with Transparent Activation disabled):


Note that at depth 0 (zero) only the object identifier for "Caroline" (1336) was loaded from the database; at depth 1 Caroline's value types were fully loaded and reference fields were initialized (to null) but not activated (see Caroline's address and parent fields). At depth 2 Caroline's parent object was also loaded (and activated) but its reference fields (for instance Caroline's grand father) were not activated yet.

Related Concepts

The same trade-of (simplicity/performance) applies when updating objects in a db4o database; I mean, usually you'll not want to update the whole set of objects reachable from the one you are currently updating because this would probably end up updating a lot of objects needlessly. To control this behavior you may configure the update depth through IConfiguration.UpdateDepth() method which takes an integer meaning how deep db4o will go when updating objects (keep in mind that when inserting, db4o will follow all references and insert all of then). This configuration may be globally (i.e, to all classes) or selectively applied (i.e, to specific classes). As of the current version db4o sets the default update depth to 0 that means that only value types and strings fields will be updated when an object is updated (i.e, reference fields will not be updated). Optionally it's possible to propagate updates to all objects reachable from the object being updated by configuring cascade on update for the specific classes you want to propagate updates.

IConfiguration.ObjectClass(typeof(your_type)).CascadeOnUpdate(true);

When configured this way, db4o will go down, updating all objects reachable from your_type until it reaches an object not configured for cascade updating (note that this may impact performance severely if this object references a lot of objects also configured for cascade updating).

Transparent Activation

Transparent Activation is a functionality that enables developers to handle objects returned by queries as if they were fully activated without incurring the cost of activating a whole graph of objects. In other words, when TA is enabled, developers may code as if activation depth was set to infinity without having to worry about the whole set of objects being loaded into memory. Basically, Transparent Activation transfers the responsibility of object activation from developers to db4o allowing them to focus on their business logic.

Configuring your db4o projects to use TA is 2 steps process:

  • Configure TA support (prior to opening the database):

    IConfiguration.Add(new TransparentActivationSupport());

  • implement IActivatable interface in classes you want to be TA aware.

    You can implement it by yourself or let us (ok, our tools) to do this work for you through Db4oTool (the old Db4oAdmin) application.

Keep in mind that when transparent activation support is enabled IConfiguration.ActivationDepth() is ignored; also, all objects that are note TA aware (i.e, that doesn't implement IActivatable interface) will be fully activated; so, we don't recommend to enable TA if you have lots of objects that doesn't implement this interface. While activating objects db4o will stop following references as soon as it reaches an TA aware object (after all it will be activated when needed).

More technical details about TA can be found here and a full sample along with details about it can be found here.

Putting it all together

Here you can checkout the sample application that shows the points discussed above. It inserts a simple graph of objects into a database and then queries for the root of the objects using different activation depth configurations. Also, you can ask the application to pretend that it's objects supports TA through the -ta command line option.

I recommend you to download it and do the following:

  • Poke around, looking into the code :)
  • Give it a try, i.e, run it without any parameter and see what you get;
  • Try to run it again, but this time passing -ta parameter, and, again, see what you get;
  • Make its classes TA aware (see here the easy way) and try to run it with and without -ta parameter.

If you have any question don't hesitate to ask them here!

Adriano

Nov 20, 2007

Old news from the front!

Visual Studio 2008 has been released (Yep, I'm downloading it right now :) Niiice.