inconsistencies when accessing the registry on Windows x64

ABSTRACT

This article shows a problem that exists when trying to obtain the path of Environment.SpecialFolder.CommonDocuments from a x86 process on systems running Windows x64 after the Common Documents folder has been moved from the default location. The article explains how to reproduce the problem and shows some working code to work around this bug. Additionally this article explains the background behind this problem and some implementation details of x86 emulation on Windows x64 systems.

 

Background

The 64-bit (x64) versions of Windows supports 32-bit (x86) processes through an emulation concept called 'Windows on Windows 64' (WOW64). Part of the magic are the File System Redirector and the Registry Redirector. The File System Redirector comes into play, when you try to access certain folders- e.g. %windir%\system32 - from an x86 process. It will map the request transparently to %windir%\SysWOW64. Registry Redirector does exactthe same for certain nodes in the registry. For example, HKEY_LOCAL_MACHINE\Software is redirected to HKEY_LOCAL_MACHINE\Software\Wow6432Node. 

Additionally, Windows allows users to move those special folders. For example, a user may decide to move his Public Documents folder to a network share, from which backups are created by the local administrator.

Here are the steps you need to take to move the Public Documents folder: 
Or use this alternative way for (moving the Public folders on Windows 7)

  1. Deactivate User Account Control (UAC) in the control panel
  2. Reboot Windows
  3. Navigate to %SYSTEMDRIVE%\Users\Public
  4. Right click on 'Public Documents' (or whatever is the localized name of that folder)
  5. Select 'Properties'
  6. switch to the 'Location' tab
  7. input the new path and apply settings, confirm all questions asked

Note: This is only possibly after deactivating UAC and restarting Windows, so Windows tries to protect you to some extend.

 

The Problem

When a user moves his special folders, all seems well (all IS well) – when viewed from the x64 world. When you look at the folder in Windows Explorer you see the real name (e.g. D:\Real Common Documents) replaced by the localized name ("Public Documents" or "Öffentliche Dokumente" on German versions of Windows).
But once you try to access that same information from any program that was compiled as x86 and runs on the WOW64 emulation, you'll get very different results indeed. 

For demonstration purposes, I moved the Public Documents folder on my Windows 7 x64 computer from its usual place ( C:\Users\Public\Documents ) to D:\ Real Common Documents.

To illustrate I pulled together a very short program in C# and compiled it with .NET 4.0.

namespace GetSpecialFolders
{
    class Program
    {
        static void Main(string[] args)
        {
            Console.WriteLine(Environment.GetFolderPath(Environment.SpecialFolder.CommonDocuments));
        }
    }
}

Then I fired up a command line and ran the following commands:
(Outputs are bold, actual commands are preceded by '>' )

>csc /platform:x86 Program.cs > dev.null

>Program.exe
C:\Users\Public\Documents

>csc /platform:x64 Program.cs > dev.null

>Program.exe
D:\Real Common Documents

(You can also have a look at the screen shot of this experiment)

As you can see the returned values differ between x64 and x86 versions.

 

Explaining the Behaviour

First and Foremost: This behaviour is (somwewhat) by design. The path to the Public Documents folder is stored in the registry and it is done so twice.
registry path for x64: HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Explorer\Shell Folders\Common Documents
registry path for x86: HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\Windows\CurrentVersion\Explorer\Shell Folders\Common Documents

So from my point of view it looks like moving the folder will update the x64 part, but not the x86 part of the registry. 

The following folders are also stored in the 32 bit part of the registry – and are thus likely affected by the same bug: 

  • Common Administrative Tools
  • Common AppData
  • Common Desktop
  • Common Documents
  • Common Programs
  • Common Start Menu
  • Common Startup
  • Common Templates
  • CommonMusic
  • CommonPictures
  • CommonVideo

 

Important notes: 

As you see there are no user-specific folders in this list. Doing the same steps with My Documents folder and testing with a slightly modified version of the above application showed that My Documents is NOT affected by this bug – i.e. the same path is yielded in x86 and x64.
Windows does not allow you to move all of the above folders. At least not with a convenient user interface – you might be able to pull that off through the registry however. 

 

How to solve this problem

The only work around I know of, is the application (and its derivates) I posted above – compiled with /platform:x64. 
Calling the resulting executable and parsing its output solved the problem for me and will likely solve it for you. 

I tried calling SHGetKnownFolderPath directly through P/Invoke while debugging my application, but that unsurprisingly did not help me. 

 

Further Reading:

 

UPDATE 1:

Running the same test (see above) on Windows 8 x64 Consumer Preview returns an empty string for the x86 version. 
So this matter seems to actually get worse with Windows 8!

Survey on the Status Report Framework

I'm currently quite busy with founding a new startup, which is why I post scarcely.
The idea behind my new startup is to lower costs in laboratories by increasing automation.

Expect to read more about this topic as soon as I find the time.

Meanwhile, please take part in a quick 3-5 minute survey on the Status Report Framework and let me hear which features you would like to have implemented.
 

5 tips for non-english software developers / programmers

The inspiration for today's post comes from a piece of code – more precisely from one of those famous // TODO comments.
So here's the comment alongside with the line of code:
 

                // TODO unabhängig von Sprache machen
                properties["authorizedGroup"] = "Benutzer";

You are already wondering what "unabhängig von Sprache machen" and "Benutzer" mean – right?

This is the first thing to avoid:

1. never, NEVER EVER – not even while hacking some website with a gun pointed to your head and a blonde giving you head (Swordfish)… OK maybe this is a valid excuse to not comment at all – write your comments in any other language than English.

You will write 100% of your code in English and if you decide to write your comments in German/Polish/Esperanto/Swahili you'll force your brain to switch context dozens of times while reading your code. There is conclusive evidence that switching contexts disrupts your efficiency while coding.  So just don't do it. Write in English. You'll probably improve your English skills along the way.

 

2. Learn English

Learn to read and write English – proper English. I flinch everytime someone storms into the C# chat over on StackOverflow and writes sentences like this: "I gut prob w/ ASP.NET MVC3. Do u any exp w/ it?"
This sounds like the writer has learned English from the graffitis in the background of an American movie with subtitles. Probably from the movie Dangerous Minds or worse.
Extra tip for readers from India: Putting a 'sir' behind every sentence does improve neither spelling nor grammar.

 

3. Write about programming – in English

I started this blog for exactly that reason: I want to better communicate my ideas and thoughts about programming. I want to be able to do that in English, because English is the lingua franca for software developers. I want to be a great programmer and I share Joel's opinion, that you the difference between tolerable and great is ability to communicate. After graduation from university I am going to be a military leader of 40-200 men and women. I need to be able to communicate in a concise way and yet not miss important details.
If starting your own blog seems like to much trouble for you, go over to StackOverflow and start answering questions. Try to make your answer the most awesome answer on the web.

 

4. Install only English versions of your IDE

This is especially true if your IDE is being shipped by Microsoft. For all that Microsoft does really well with its OS, the programming languages, the framework, world peace (OK maybe not with that), the fact is they… ummm… suck at translations. I know professional interpreters that outright refuse to do any work for Microsoft, because of their translation policies.
The number of times I've been trying to find the Microsoft translation (English to German) of a menu item in Visual Studio is beyond count. Often times this boils down to "third menu from the right, fourth item from top" in the end.

 

5. When logging error messages always use English messages

For the same reasons stated earlier on, I recommend you log all your exceptions / errors in English. This is especially true for exceptions that are automatically translated into the computer's locale – as happens with most exceptions in .NET. Imagine you are receiving a logfile from a customer in Libya and find the error messages written right to left in an unfamiliar alphabet. Good luck finding that bug.
This can be worked around for most exceptions in .NET (solution on StackOverflow) and I'm sure you'll find a way to make it work in your programming language.

 

6. Bonus tip: get the job done

No matter whether you are a native English-speaker or not, the main part of being a software developer is getting things done.

 

You have more tips to share?

Please post them in the comments and keep discussing

Usability fail OR hidden features of a bike computer

As long as weather permits (read: no blizzards, no ice/snow) I ride my bicycle to work as often as possible. It's a nice 12 kilometer ride one way and gives me amble time (approx. 30 minutes) to ponder about the answer to life, the universe and everything. While taking those rides I like to stay informed about my current speed, average speed, distance traveled, etc. It's just cool to know that I ride 115 kilometers in a good week and I spend 6 hours a week on my bike. So I bought a bike computer from a local DIY market for $13. That thing lasted probably less than three months and so I went to a nearby bike shop where I got the "SIGMA BC 1009" bike computer for $32 . It is not quite the state-of-the-art bike computer, but I figured spending $100 for an additional calorie counter and display of the current altitude would be a waste of money. After about six months I have to say that I am quite happy with my purchase. It has all the data I really need and it is still operational. But, for me it is quite the definition of 'usability FAIL'. Take a look at that picture:

 

As you can see it has 4 buttons. The left ones are for resetting trip time, trip distance etc. and settings menu. The other two let you switch through the displayed data. Here's a list of all the data you can access:

  • Trip distance
  • Trip time
  • current speed
  • average speed
  • maximum speed
  • Total ride time
  • Total distance
  • current time

Now while you ride your bike hitting one button will show you the current time and the other one cycles through:

  • Trip distance
  • Trip time
  • current speed
  • average speed
  • maximum speed

Noticed something? Yes! 'Total distance' and 'total ride time' are missing. Imagine poor old me riding on my bike and desperately trying everything from quick double tapping the buttons to holding the button down for 10 seconds to pressing two buttons at the same time – just to get a display of the total distance!
I literally tried for weeks to find out the total distance. I eventually resigned and looked up the function in the manual. I am a man! Real men don't read manuals!

And through reading the manual I learned that display of 'total distance' is disabled while riding the bike!

WTF? Of all the gazillion different actions I tried to get the total distance, pressing the upper right button multiple times while the bike is not moving certainly was not one of them.
And there I stood – utterly defeated and bereaved of self-esteem (after all I had to read the manual!) – wondering how this came to happen.
I'd like to think that there are some complex technical restrictions that don't allow that calculation while the bike is moving – something like the influence of the coriolis force on the trajectory of free electrons.
But the more I think about this, the more I come to the conclusion that there is nothing restricting the manufacturer from doing so. It is just that an engineer somewhere decided that this was the best way to implement this feature.
I would sure like to know why. I've never seen this behaviour in any of my previously owned devices  nor do I know of any make of car (obvious analogy for most users of bike computers).

How to extend the Status Report Framework with custom information classes

In the last article I wrote about the Status Report Framework and how it can be used to gather information about your application. This time I am going to show you how you can easily extend the Status Report Framework by adding your own data (called 'information' in this context) and putting it into the  StatusReport. If you are unfamiliar with how the framework works you should go back and read the release article now. For this article I will create an information class that adds free disk space information to the StatusReport.

Which class to inherit from?

While you could inherit from any class that implements AbstractReportInformation there are some natural choices:

  • LeafInformation – choose this if you are providing some real information, not just aggregation; this should be the right choice in most of the cases
  • CompositeInformation – choose this if you are merely putting together information in a new way
  • AbstractReportInformation itself – choose this if no other option seems right

I used AbstractReportInformation for the OneStreamInformation and FilesystemInformation classes, because they don't easily fit into the categories of 'composite' or 'leaf'. However, for the class I am about to create I will happily default to LeafInformation.

What do I need to implement?

Because I inherit directly from LeafInformation most of the desired behaviour is already implemented and I only need to provide an implementation for public override Stream DataStream {get;}. So here it goes:

public override Stream DataStream
{
    get
    {
        MemoryStream stream = new MemoryStream();
        var writer = new StreamWriter(stream);
        ulong cFreeAvailable;
        ulong cFreeTotal;
        ulong cTotal;
        bool fReturn;

        SetErrorMode(ErrorModes.SEM_FAILCRITICALERRORS);

        foreach(var path in Pathes)
        {
            fReturn = GetDiskFreeSpaceEx(path, out cFreeAvailable, out cTotal, out cFreeTotal);

            if (fReturn)
            {
                writer.WriteLine(string.Format(
                    "path: {0}, free available space:{1}, total space: {2}",
                    path, cFreeAvailable.FormatBytes(), cTotal.FormatBytes()));
            }
            else
            {
                int lastError = Marshal.GetHRForLastWin32Error();
                Exception ex = Marshal.GetExceptionForHR(lastError);

                if (null != ex)
                {
                    writer.WriteLine(string.Format("no access to path '{0}', error code: {1}, exception:{2}",
                                                   path, lastError, ex));
                }
                else
                {
                    writer.WriteLine(string.Format("no access to path '{0}', error code: {1}, no exception thrown", path, lastError));
                }
            }
        }
        SetErrorMode(ErrorModes.SYSTEM_DEFAULT);
        writer.Flush();
        stream.Seek(0, SeekOrigin.Begin);
        return stream;
    }
}

I also need a small constructor along with a private property to get the UNC pathes in. To make the call to GetDiskFreeSpaceEx work I also need the P/Invoke signature of this method.

Design decisions and implementation details

The logic of the class is completely packed into the DataStream_get property accessor, because I deliberately wanted no caching of results. When I first searched the web (a.k.a. Google) for information on how to read the number of free bytes on a disk, I found the DriveInfo class, which provides a convenient and natural way to access this data for logical drives. This is certainly good enough for most use-cases, but it just doesn't do the trick for network drives – something I would not want to miss in a framework. So I searched some more (StackOverflow this time) and quickly found GetDiskFreeSpaceEx, which works for UNC pathes. Because I use P/Invoke I have to do some more work to handle errors – including calls to Marshal.GetHRForLastWin32Error() and subsequently to Marshal.GetExceptionForHR(Int32). When I tested this for the first time with various valid and invalid UNC pathes, I got an error dialog when trying to access a removable storage device (an SD-Card reader to be precise). This can be suppressed by calling SetErrorMode() via P/Invoke.

 Recommended Reading

  • The release article for my error handling and application fitness framework

Feedback

If you have any suggestions on how to further improve the Status Report Framework, please share and discuss them in the comments

Presenting The Status Report Framework

A while back I was looking for a ready-to-use framework that would collect data about my application’s state in case of a severe meltdown – or whenever I felt the need for it.
Because my application also runs on PCs that are not connected to the internet (or any network for that matter) it had to be able to just create a zipped version of the data and throw it at the user (i.e. place the file on the desktop where the user can find it)

My requirements

  • easy-to-use (don’t want to clutter my code with this)
  • creates a ZIP file and allows me to store it on the desktop
  • add any data I want to the collection (i.e. files from hard drive, custom messages, etc)
  • automatically collect a bunch of data (version number of the application, exception data, log4net logfiles, event log data, etc.)

Additionally, I didn’t want to throw $500+ at the problem, so GIBRALTAR was out of the equation.

Well – I did not find anything fitting my needs. And so I started implementing my very own version of such a framework. After I finished it, I had it peer reviewed by @akmad from StackOverflow, whom I want to thank for his great ideas and the time he invested into reviewing my code (even more thanks for not submitting parts of it to TheDailyWTF.com).

Some information on the architecture and inner workings

Lets start off with a class diagram.

The framework is easy and straight forward to use. There are two main classes (StatusReport and AbstractReportInformation, both are abstract class).  AbstractReportInformation (and deriving classes) is based on the composite pattern. The derived classes represent some sort of data that can be put into a StatusReport.  For example the ExceptionInformation class stores an Exception object. If a class contains real information (and is not a container class), its IsLeaf property returns true. Access to the stored information is possible through the DataStream property, which will return a Stream object (in most cases a MemoryStream) with a serialized representation of the stored information. If IsLeaf returns false, the Children property grants access to the child nodes.

The currently only implementation of StatusReport is ZipReport, which stores the data into a ZIP file. When you invoke the PrepareReport() method on the ZipReport class it iterates over all AbstractReportInformation objects that have been added using the Add(AbstractReportInformation) method. This is done recursively (so all child elements will be executed). For each Information that is not a leaf a directory according to the information’s Name property will be created. For each leaf information a file with the serialized information obtained from DataStream will be created.

Have a look at the sequence diagram and see how this works:

For ease of use I have also added a InformationFactory class(not a real implementation of the factory pattern – but it was the best name that came to mind) that provides ways to create commonly desired information objects.

So how now do you use it?

That is really simple:

        private void CreateStatusReport()
        {
            using (ZipReport report = new ZipReport())
            {
                report.Add(InformationFactory.CreateMainInformation("1.0.0.0"));
                report.Add(new ExceptionInformation(CreateException()));
                report.PrepareReport();

                File.Copy(report.ReportPath, GetDesktopPath("report.zip"),true);
            }
        }
        private static Exception CreateException()
        {
            try
            {
                throw new NotImplementedException();
            }
            catch (Exception ex)
            {
                return ex;
            }
        }

        private static string GetDesktopPath(string filename)
        {
            return System.IO.Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.Desktop),
                                   filename);
        }

This will create a file named “report.zip” on your Desktop. In this ZIP you’ll find two files: exception.txt contains the created exception complete with stack trace, main.txt contains a bunch of information about your application, the system and processes currently running.
Note that I pass the version number as a string to InformationFactory.CreateMainInformation() here.

 

 Download

The file can be downloaded here: http://www.hiddenfeatures.net/BEMA.StatusReport.zip

In it you will find:

  • the Visual Studio 2010 solution
  • projects for the three components
  • a small demo application (START YOUR EXAMINATION HERE!)
  • all source codes
  • SharpZipLib and log4net, because the Status Report Framework depends on them
  • a PDF with the UML diagrams found on this page
  • a license file for log4net ( I did not find any for SharpZipLib, so if you could point me to one I will happily add it)

GitHub

As of 24.12.2011 there is a repository on GitHub where you can get the source codes and do all the other weird stuff that you folks normally do on GitHub (fork etc)
https://github.com/yas4891/BEMA.StatusReport

Feedback

I am looking forward to hearing your thoughts on this. Any suggestions for improvement are really appreciated and if you find a feature missing: tell me!

Changelog:

* * * * * Version 1.7.0 * * * * * [this is not yet reflected in the downloadable file]
-add: nicer caption for the Windows version in InformationFactory.CreateSystemInformation()

How to get information about swallowed exceptions

Did you ever stumble upon a code block looking something like this?

try
 {
 // do  something
 }
 catch(Exception)
 {
 // swallowed for no good reason
 }

Well, hopefully it wasn’t you who swallowed the exception in the first place.
There might be various reasons for swallowed exceptions and I plan on dedicating a blog post to this topic some day.
Meanwhile suffice to say that there are cases in the wild, where you can’t go back to the code, insert some logging, recompile, build and ship your software.
Might be, that you don’t have the code (third party library used) or even that the CLR swallows the exception (which happens on x64 systems under certain conditions).

However, it might be handy to get a notification every time an exception is thrown.
Luckily, the .NET Framework provides a mechanism for just that. With the AppDomain.FirstChanceException event you can access any exception before it is passed on for handling.

So how do you do this?
It is really quite simple. All you have to do is to register for the event and have an appropriate event handling method:

public static void Main(string[] args)
{
    AppDomain.CurrentDomain.FirstChanceException += FirstChanceHandler;
    try
    {
         throw new Exception("swallowed exception");
    }
    catch(Exception) {}
}

static void FirstChanceHandler(object source, FirstChanceExceptionEventArgs e)
{
    Console.WriteLine(e.Exception);
}

This will print out the exception to the console just as if it had been handled in the catch block.
Note that you can not handle the exception in the event handler! So you still want to write exception handling code.
Furthermore, throwing an exception inside the event handler will (of course) run that very same event handler again and so on, resulting in a StackOverflowException. Not something you want to happen in your app! So be extra careful in that event handler and apply proper exception handling.

This feature comes in pretty handy to get information about any swallowed exceptions that happen in your application.
Additionally, it can be used as a sort of monitoring and early warning system. While exceptions are – mostly – an effect of normal program execution (if they are handled correctly), it is interesting to know which exceptions get thrown and how often they are thrown.
In one of the future articles I will go into further detail on how to create a logging framework for first chance exceptions

Hidden Features of Java

This is a summary of  the highest ranking (by vote count) answers from the corresponding StackOverflow.com thread

Those are all the hidden features that scored 100+ votes at the time of this writing.

——————————————————————————————————————
I think this is a fine list of some rarely known (read: put to good use) framework classes (ThreadLocals), language functions (NULL checking via instanceof) and tools (VisualVM).

However, I find some of the elements on the list irritating. Take for example the labeled blocks and the infamous goto keyword. I have a hard time understanding why this keyword has been implemented in the first place. And even though it is there, I have not seen it being used in production code.
And this might be the answer to the question: It got this much votes, because people are surprised that goto was implemented in Java at all.

Motivation

This is my first post in my very own blog. So besides the (albeit) few facts you can read about me on the About page, I’d like to explain why I started this blog.

First and foremost, this is because I love programming. I am reasonably good at it, but I lack in the skills to clearly express myself to fellow programmers. I encountered this time and again, but now is the time to change something about it.
Starting now, I will write blog posts on a regular basis. I’ll try to do two posts a week – hoping that it will improve my writing skills.

Secondly, the spark that started this all came from Stackoverflow.com. The community over there is awesome. One rather common thread is the “Hidden features of [place your favorite programming language here]” thread. Unfortunately, those threads are closed regularly. This leaves those threads without a good summary (see ‘Hidden Features of Java’ for an example). So I’d like to provide summary pages with deep links.

The topics will focus on those “hidden” language features (e.g. the null-coalescing operator in C#). I am aware of the fact that no language feature is really hidden. Most languages today are reasonably standardized and you could always read the specification front to end. But frankly, not many of us do this (me included).
Plus, I enjoy reading those threads on StackOverflow.com from time to time as helpful reminders.