Did you know (Part II) File / Registry virtualization on Windows 7 / Vista

Hi

In this post I discussed some issues I faced regarding software running on Vista 64 bits. Well it ended up that all my troubles was happening because my program (64 bits) was launching IE 64 bits (instead of the 32 bits) and I had not installed Silverlight plugin on it! Solution: just switch my program to 32 bits and it worked as expected!

In this post I want to discuss some pitfalls when running software on x64 bit versions of Windows (most of the discussion applies equally well to Vista x64 ,Windows 7 x64 and probably to newer versions also - haven't tried it with Windows  8 but I bet that the behavior is the same).

Keep in mind that when talking about the reasons that lead Microsoft to add a specific feature I am just speculating, so take every word with a grain of salt :)

On previous Windows versions, to many (almost every single?) developer used to login as admistrator into their workstations¹ so there were way to many applications that relied on being able to change system resources (such as to create files/fonders under "Program Files"), after all they always succeed doing that since developers had the rights to do so (remember, administrator...).

At Vista launch time MS had a tough issue to deal with: how raise the bar for system security / stability without breaking such ("legacy") applications?

AFAIK a compromise was settled so Windows development team decided that for some 32 bit applications such requests would not fail, but instead, Windows would lie to the calling process and would just create the file somewhere else (the same principle apply to registry access): this feature is known as file virtualization (the rules that governs whether file/registry will be enabled or not for a specific process can be found here)

Note that this feature is meant to allow legacy applications² to run on the new platform (at that time, Windows Vista) so Microsoft plans to remove it (as soon as your application gets compiled as 64 bits, virtualization will be disabled anyway).

If you are running the 64 bits version of Windows 7 (or Vista for that matter) you can try it right now by downloading the test application source³ and running it with something like:

FileSystemVirtualizationTest.exe -e %windir%\test.txt "adriano's test"

For now, just run it with the arguments shown above and take a closer look in the produced output:

Virtualization ENABLED.

Virtualization is ON

14 bytes writen to C:\Windows\test.txt

It seams that the application succeed in creating a file under %windir% (which it should obviously not be allowed to) but how can we confirm that? Easy, just run the application again now providing only the file name (the application will try to read the file and dump its contents):

FileSystemVirtualizationTest.exe %windir%\test.txt

Virtualization is ON

adriano's test

and there it is! But wait! Let's open Windows Explorer and try to locate the file. Hey! I can't find the file anymore! Windows search can't find it either:

But the application does succeeded opening the file! how can that be? Launch Windows Explorer again on %windir% folder but this time paying attention to the little button: "Compatibility Files":

Ok, now that you noticed this button, press it and Windows Explorer will open a folder that looks like the one bellow:

Wow! Windows just created the file under %localappdata%\VirtualStore\Windows folder (which is a per user folder). 

This way Windows let applications believe they had successfully updated files / folders / registry keys when actually they didn't (preventing possible global system corruption). Clearly avoiding system corruption is a fair reason to support this functionality, but now, developers (and IT admins) must pay attention to scenarios with mixed 32 / 64 bits applications interacting in such ways that they need to observe changes from each other (since the first ones may be writing to the virtual store and the last ones will not see the virtual store).

Last, but not least :), make sure your application do not depend on registry/file virtualization; these mechanisms were introduced sonly to raise the likelihood of legacy applications to run without problems on Windows versions with stricter security settings (Vista, 7, etc). So if you have an application that does writes to protected file system areas, please do your users a favor: update it to write to a more appropriated location (%appdata%, %localappdata%, etc).

Bellow you can find some links if you want to read more about file system virtualization:

• http://blogs.sepago.de/helge/2008/04/20/windows-x64-all-the-same-yet-very-different-part-7/ 
  
• http://msdn.microsoft.com/en-us/library/dd464643(VS.85).aspx
• Best practices for WOW64
• UAC file / registry virtualization

Have fun!

Adriano

[1] I am guiltily of doing the same thing :(
[2] Basically all applications developed for previous Windows OSs are handled as legacy by Vista
[3] In the first drafts of this post I used Notepad++ to write to %windir% but it looks like they added a manifest in the latest versions so it will not work for our demonstration purposes. Even in my sample application I was required to remove the manifest explicitly (VS 10 included one that disabled virtualization)

Little handy presentation tool

Hi


Yesterday I stumbled over a really nice tool for producing screen casts (called zoomit) that allows you to zoom, draw and write directly on your desktop. If you do screen casts (or any kind of demonstration with a computer connected to a projector) I highly recommend  it (BTW I highly recommend all the Sysinternals stuff :).


Hope you find that useful.

Updating to git 1.7.11 on a Windows box

Hi


If you just updated to git version 1.7.11 and started to get the following error when you try to run git svn: 

Can't locate Git/SVN/Editor.pm in @INC (@INC contains: /lib /usr/lib/perl5/5.8.8/msys /usr/lib/perl5/5.8.8 /usr/lib/perl5/site_perl/5.8.8/msys /usr/lib/perl5/site_perl/5.8.8 /usr/lib/perl5/site_perl .) at C:\Program Files (x86)\Git/libexec/git-core\git-svn line 81.
BEGIN failed--compilation aborted at C:\Program Files (x86)\Git/libexec/git-core\git-svn line 81.

you can check the comments in this issue; in my case it fixed the problem.


See you!

byte code instrumentation in java...

Hi!

Are you a "low level",  developer? Have you ever disassembled a class byte code just to see what's under the hood? If not, I urge you to give it a try; IMO having at least basic knowledge about how the platform running your app works at low level may help us (developers) to write better code.
Since I joined db4o team I had the opportunity to play a lot with Mono.Cecil, a great .Net byte code manipulation lib, but I've been kind of lazy when it comes to the Java side (sure, we do employ byte code manipulation on Java code but, to be sincere, I've never touched that area of the code).
This has been slowly changing since I have joined another project at Versant (the company behind db4o) and this lead me to this post: last weekend I was debugging some Java class library code that I had no source code for¹ and I wanted to check some parameter values.
Since I was alone in my house (wife and kids went to do some shopping with my mother in law :) I decided to try 2 related technologies in the Java world: Java agents and ASM library (you can read more about Java agents here and here)².
Simply put, Java agents is a technology used to change class bytecode when it enters into the system, meaning, instead of doing the manipulation in the bytecode and saving it to the disk using java agents one can change the in memory representation of the class (without touching the actual bytes in the disk) when the JVM tries to resolve the class. If you use Sun JVM you can simply type the following command in the command line:

java -javaagent:jar-containing-agent-code.jar[=agent-args] ...

ASM is a library that allows developers to manipulate Java bytecode in a flexible, yet relatively easy way so it is a great match to use with Java agents (if you want more details I recommend to read this ASM tutorial).

Bellow you can find the code that I wrote for the agent (you can download a zipped eclipse project from here); it expects the class name and method which should be instrumented in the form className#methodName (if you omit the method name all methods will be instrumented):

package com.thinkingsoftware.diagnostics;

import java.io.*;
import java.lang.instrument.*;
import java.security.*;
import java.util.*;

import org.objectweb.asm.*;

public class DumperClassTransformer implements ClassFileTransformer {

	private final String className;
	private final String methodName;

	public DumperClassTransformer(String... args) {
		this.className = args.length > 0 ? args[0] : null;
		this.methodName = args.length > 1 ? args[1] : null;
	}
	
	private static Map> wrapperMapping = new HashMap>();
	
	{
		wrapperMapping.put(Type.INT_TYPE.getDescriptor(), Integer.class);
		wrapperMapping.put(Type.FLOAT_TYPE.getDescriptor(), Float.class);
		wrapperMapping.put(Type.DOUBLE_TYPE.getDescriptor(), Double.class);
		wrapperMapping.put(Type.BOOLEAN_TYPE.getDescriptor(), Boolean.class);
	}

	@Override
	public byte[] transform(ClassLoader loader, final String className, Class classBeingRedefined, ProtectionDomain protectionDomain, byte[] classfileBuffer) throws IllegalClassFormatException {

		if (className.startsWith("java") || className.startsWith("sun")) {
			return classfileBuffer;
		}
		
		if (this.className == null || className.equals(this.className)) {
			
			ClassReader cr = new ClassReader(classfileBuffer);
			ClassWriter cw = new ClassWriter(ClassWriter.COMPUTE_MAXS);
			
			ClassVisitor cv = new ClassVisitor(0, cw) {
				
				@Override
				public MethodVisitor visitMethod(int access, String name, String desc, String signature, String[] exceptions) {
					MethodVisitor mv = super.visitMethod(access, name, desc, signature, exceptions);
					
					if (methodName != null && !name.equals(methodName)) {
						return mv;
					}
					
					try {
						String printlnStringMethodDesc = Type.getMethodDescriptor(System.err.getClass().getMethod("println", String.class));
						
						mv.visitFieldInsn(Opcodes.GETSTATIC, Type.getInternalName(System.class), "err", Type.getDescriptor(System.err.getClass()));
						mv.visitLdcInsn(name + " :");
						mv.visitMethodInsn(Opcodes.INVOKEVIRTUAL, Type.getInternalName(System.err.getClass()), "println", printlnStringMethodDesc);
						
						Type[] args = Type.getArgumentTypes(desc);						
						for(int i = 0; i < args.length; i++) {
							
							if (args[i].getSort() == Type.ARRAY) { // not supported... :(
								continue;
							}
							
							mv.visitFieldInsn(Opcodes.GETSTATIC, Type.getInternalName(System.class), "err", Type.getDescriptor(System.err.getClass()));
							mv.visitInsn(Opcodes.DUP);
							
							mv.visitInsn(Opcodes.DUP);
							mv.visitLdcInsn("\t");
							mv.visitMethodInsn(Opcodes.INVOKEVIRTUAL, Type.getInternalName(System.err.getClass()), "print", printlnStringMethodDesc);
							
							mv.visitIntInsn(Opcodes.BIPUSH, i);
							mv.visitMethodInsn(Opcodes.INVOKEVIRTUAL, Type.getInternalName(System.err.getClass()), "print", "(I)V");
							
							mv.visitInsn(Opcodes.DUP);
							mv.visitLdcInsn(" = ");
							mv.visitMethodInsn(Opcodes.INVOKEVIRTUAL, Type.getInternalName(System.err.getClass()), "print", printlnStringMethodDesc);							
							
							Class wrapperClass = wrapperMapping.get(args[i].getDescriptor());
							
							if (wrapperClass != null) {
								
								mv.visitVarInsn(args[i].getOpcode(Opcodes.ILOAD), i + 1);

								Type.getMethodDescriptor(Type.getType(wrapperClass), args[i]);
								mv.visitMethodInsn(
										Opcodes.INVOKESTATIC, 
										Type.getDescriptor(wrapperClass), 
										"valueOf", 
										Type.getMethodDescriptor(Type.getType(wrapperClass), args[i]));
							} else {
								mv.visitVarInsn(Opcodes.ALOAD, i + 1);
							}
							
							mv.visitMethodInsn(Opcodes.INVOKEVIRTUAL, Type.getType(Object.class).getDescriptor(), "toString", Type.getMethodDescriptor(Object.class.getMethod("toString")));
							mv.visitMethodInsn(Opcodes.INVOKEVIRTUAL, Type.getInternalName(System.err.getClass()), "println", printlnStringMethodDesc);
						}						
					} catch (Exception ex) {
						ex.printStackTrace();
					}
					
					return mv;
				}
			};
			
			cr.accept(cv, 0);
				
			
			byte[] byteArray = cw.toByteArray();
			
			writeFile(className, byteArray);
			
			return byteArray;			
		}
		
		return classfileBuffer;
	}

	private void writeFile(String className, byte[] byteArray) {
		FileOutputStream f;
		try {
			String tempDir = System.getProperty("java.io.tmpdir");
			String outputPath = tempDir + className + "Inst.class";
			
			System.err.println();
			System.err.println("Instrumented class stored at: " + outputPath);
			System.err.println();
			System.err.println("You can look into it actuals bytecode with the following: ");
			System.err.println("javap -v -private -classpath " + tempDir + " " + className);
			System.err.println();
			
			f = new FileOutputStream(outputPath);
			f.write(byteArray, 0,  byteArray.length);
			f.close();
		} catch (IOException e) {
			e.printStackTrace();
		}
	}
}

If you want to try it you simply compile it and run some Java application; you should get  an output looking like:

running my agent against the following Java application

public class TestAgent {

	public TestAgent(String msg, int n) {
		this.msg = msg;	
	}

	public static void main(String []args) {
		TestAgent t = new TestAgent("Hallo Welt", -1);
		String msg = t.getMessage("Hello World", 42, 11.0f, true, t);
	}
	
	private String getMessage(String msg, int i, float f, boolean b, Object o) {
		return msg;
	}
	
	public String toString() {
		return "OLA MUNDO";
	}
	
	private String msg;
}

Fell free to use it anyway you like but keep in mind that this code is far from being complete or bug free or production ready (for instance it does not handle arrays) so strange errors may happen if you find one of these not supported cases.

As a last comment, if you want to play with ASM I do recommend you to take a look in the Bytecode outline plugin (but be careful to use the one in the previous link instead of the one in the Objectweb page since the later seems to have issues with newer Eclipse versions).

Hope you find this interesting!

Adriano


----

¹ Of course I could either download it from the web - after all it is an open source project- or disassembled it, but I wouldn't loose the change to play with new toys and possibly learn something new - not to mention that I severely suffer from the NIH syndrome :)


² Since I am in no way an expert in the subject, take every word with a grain of salt.

API usage rant

If you have been following me recently (Facebook, Linked-in, Tweeter, etc) you probably are aware that I have been busy taking care of the building of my new house, my job and what not (not necessarily in this order ;).

Man, building a new place proved to be a huge time sink; since we've finally finished the main construction phase I expect to blog more often.

Ok, so lets start with a rant (I am really good ranting :))

Some time ago I was playing with one of my toy projects when I got an exception while adding a new entry into a dictionary. The exception was quite self explanatory about what had happened (from the point of view of the dictionary class implementer): a duplicate key was added (which,  according to the documentation, is explicitly prohibited).

Ok, I can live with that, it was my fault after all ;) My real issue is that the message would be much more helpful had it included the actual key value (of course you override ToString() method in all of your public classes, don't you ?).
Without this information I was forced to debug the code (of course I could catch the exception and print the key but then I'd be required to change my code just to find out what was happening) to figure out what the original exception could have told me.

To my understanding, one of the reasons for this is to avoid possible sensible information from leaking and, even though IMHO this is reasonable, it makes me think whether we could do it better. 

To me looks like the dictionary class developer (product owner, or whoever) was faced with the usual coast/benefit decision: make it easier, more convenient to use or make it suitable for use when sensible information is involved and he/she decided for the later.

When writing libraries I usually tend to making it easier to be used at the coast of a higher chance of developers misuse (or using it in a way that may shut then in their foots). I confess that I had never thought about this particular scenario but now I think it could be fixed (made more flexible) by allowing the user (in this case the developer) to decide what to expose in exceptions (through an enum, an interface, you name it) and choosing a safe default value.

What do you think?

Adriano

Free (open source) RAM disk for Windows (x86 / x64)

Hi


Have you ever had the need to perform some I/O intensive operation on a relative small set of files? I do, usually when I need to compile some application.

In order to speed up this operation I've recurred to the so called "RAM Disks", applications that takes part of your computer's main memory and pretends that it is an actual disk. As you can imagine read/write/etc (I/O) operations against this virtual disk are much faster than operations against your actual hard disks (be it a traditional magnetic HD or a shine new SSD).

In the past 3 years I've used at least 3 different solutions and was happy with none. My main complain is that most of them were not capable (or it was really clumsy/hard to accomplish) of creating RAM disks dynamically, i.e, one is required to create it at OS start up time. Since my needs are dynamic (I may need more disk space at some points in time during the day) it is really hard to come up with a disk size that work for me: choose a to big disk and I am wasting precious RAM space; create a disk that's to small and I may find myself facing "out of disk space" errors (then cleaning up some junk and restarting my build tasks).

Fortunately some time ago I stumbled upon a RAM disk implementation that allows me to mount and unmount disks dynamically! Since then my work flow has been much more smooth ;) The not so good side is that it is not very easy for the "not computer savvy" user.

Bottom line is: if you need a stable and flexible (but maybe not so easy to start with) RAM disk solution you can't go wrong with this one.

What's wrong with this code - fun with assembly : the answer

In the previous post I presented the following program and asked what was the problem with it:

int fact(int n)
{
    if (n == 1)
        return 1;
    else
        n = n * fact(n - 1);
}

int main(int argc, char *argv[])
{
     int n = argc > 1 ? atoi(argv[1]) : 5;
     int i = fact(n);
   
     return printf("fact(%d): %d", n, i);
}

As I said in the previous post, the problem itself is not so hard to find out: take a close look in line 6 again! We are not missing any instruction, or are we? Actually we are missing a return  so line 6 should really look like

return n * fact(n - 1);

i.e, the developer forgot to include the return.
Note that the compiler tried to alert me with the following warning, but as you know, I just ignored it :) (as I said in the previous post, you should never ignore compiler warnings)

c:\temp\fact.c(7) : warning C4715: 'fact' : not all control paths return a value

So the interesting question is: why this program works even when it is clear that it is missing a return? In order to answer this question we are going to dive into the assembly generated code for this program. To get the assembly code just type the following in a command line (assuming you have cl.exe in your path):

cl fact.c /Fa fact.asm

Even if you have little knowledge of assembly, please, bear with me; I'll try to explain the important parts. Also I've simplified both functions (fact and main) assembly code removing not important (to this discussion) bits.

MESSAGE DB 'Fact(%d): %d', 00H
fact:
 push ebp
 mov ebp, esp
 
 cmp dword ptr [ebp+8], 1
 jne next_fact
 mov eax, 1
 jmp finish
 
next_fact:
 mov eax, [ebp+8]
 sub eax, 1
 push eax
 call fact
 add esp, 4
 imul eax, [ebp + 8]
 mov [ebp+8], eax
 
finish:
 pop ebp
 ret
 
main:
 call fact
 add esp, 4
 mov _i$[ebp], eax
 mov ecx, _i$[ebp]
 push ecx
 mov edx, _n$[ebp]
 push edx
 push OFFSET MESSAGE
 call _printf
 add esp, 12
 mov esp, ebp
 pop ebp
 ret 0

PS: If you want to generate an assembly source with more - actually lots of - information from the original C program use the following command line arguments:

cl fact.c /Fa fact.asm /FAscu

First let have some simple facts:

• fact function code starts at line 2 and extends through line 22 (ret instruction).
• Argument n is stored at address [ebp + 8]
• main function starts at line 24 and extends through line 37.
• main calls fact function on line 25 (again, please note that for brevity/simplicity reasons I removed parts of the main function, so it became easier to understand).

Now that we have a high level view of the code lets dig a little deep into the fact function code (and on main function when appropriate).

Lines 3 and 4 represents the standard C function prologue; the first interesting instruction is the one at line 6 which compares n (remember, [ebp + 8]) with 1 branching to label next_fact (line 11) if they are not equal.

Starting at line 12 the code loads n into register eax subtracts one from it and calls itself recursively. When fact returns from a previous recursive call (line 16) the code calculates n times eax i.e, the compiler used eax register to return the calculated factorial from fact. We can confirm this behavior inspecting line 27 (inside main function) which assigns eax to local variable i after calling fact.

But just using eax register to pass the return value from fact is not enough; this program works (accidentally) only because the compiler used the same register eax to perform the calculations and to pass the return value.

So we can conclude that in this particular program, omitting the return statement renders an executable equivalent to the one that would be generated had the return statement be present.

Note that different compilers (or even different versions of the same compiler) may choose other registers to perform the calculations / pass return values from functions; actually I have found at least one version of CL (from Visual Studio 6) that used register ecx to perform the multiplication and eax to pass the return values (rendering an incorrect program).

Note also that even the same version of a compiler may generate different versions of the code (one that uses eax for both, calculations and for the return values and another that uses a different set of registers) depending on options such optimization, debug, etc.


Best