WinRT internals: WinMD files

One of the main design goals of the Windows Runtime is to provide a broad range of APIs that can be consumed across a variety of technologies and languages. In fact, everything designed for WinRT can be made accessible to other languages through the use of language projections. Every class Microsoft provides and every class you write in C++ and C# in Visual Studio generates .winmd files that describe to other bindings how to use that. You simply need a tool that turns those meta data files into code in your language of choice. I suggest you take a look at Microsofts documentation on how to Consume and Emit .winmd Files to get some direction on where to begin with some other language, such as Delphi.

Okay, that was a little tongue in cheek; At the time of publishing this article, that MSDN article was essentially blank, a stub for future documentation. But some other MSDN articles did give me some more clues. The overview of the Windows Runtime design describes the .winmd files: "The Windows Runtime is exposed using API metadata (.winmd files). This is the same format used by the .NET framework (Ecma-335). The underlying binary contract makes it easy for you to access the Windows Runtime APIs directly in the development language of your choice." And if that wasn't enough the new APIs to query for meta data of APIs (RoGetMetaDataFile) return a IMetaDataImport2 and a mdTypeDef. These are exactly the same things you need to inspect metadata provided by .NET assemblies, which you can already do in Delphi using the APIs in Winapi.Cor and Winapi.CorHdr (which come from cor.h and corhdr.h in the platform sdk).

Another great tip I found somewhere, because the metadata files are just Ecma-335 metadata, they look just like .NET assemblies to tools such as .NET Reflector, and Microsofts own IL Disassembler. So out of the box in Windows 8, you can use ildasm.exe to peek at the contents of .winmd files. In the image to the right here, we can see the metadata in Windows.UI.Xaml.winmd. Every class is described in detail: what interfaces it implements, what methods and properties and events are available. Every enumeration and structure is described in detail as well. All that needs to be done now is develop the tool that turns this into Delphi interfaces. While I'm not going to go into the specifics of this tool, I'd like to point out some follies I fell into when translating these files.

Namespaces

The first thing I noticed is that Windows is doing a lot to break the API down into Namespaces, and for the most part each metadata file describes one namespace. So my first reaction was to put the contents of each metadata file into one Delphi unit. This doesn't quite work though because of a few things: One metadata file may actually contain multiple namespaces, one namespace might be included in several metadata files, and there are circular dependancies between types in the various metadata files. Since in delphi one unit = one namespace, I've lumped everything into one big unit that exposes all of the Apis defined in the winmd files provided by Microsoft to work around this issue for now. If you've looked at my code on git, you'll also notice that the dotted namespaced types have been flattened, using "_" instead of ".". Nesting the classes even within this one large file becomes difficult because of the interdependencies among classes and the need for things to be defined before they can be used. A smarter tool should be able to create an tree of all the dependancies and actually construct a more OO mapping similar to the C++, C# and Javascript ones.

Classes

Classes from the point of view of Delphi are exposed in much the same way as COM objects, as interfaces. You tend to create them with RoActivateInsance, which is similar to CoCreateInstance, or RoGetActivationFactory. For classes that implement "static" methods, these methods are acquired by requesting an interface. For example, Windows.UI.Xaml.Application implements Windows.UI.Xaml.IApplicationFactory (for constructing an Windows.UI.Xaml.IApplication) and a Windows.UI.Xaml.IApplicationStatics. A Delphi wrapper will need to implement these likely by storing the reference to the interface in a class private field and implementing each method. I imagine something clever might be possible using RTTI to define class static interfaces and do something similar to dynamic aggregation (the implements keyword), injecting the interface into the class VMT.

Enums

WinRT Enums will almost certainly need to be scoped. There are enums with the same name in different namespaces, and it's very possible for the enum values to be different, especially as more third parties add their own. Microsoft ordered the values of Windows.Graphics.FontStyle (Normal, Italic, Oblique) differently than Windows.UI.Xaml.FontStyle (Normal, Oblique, Italic) for example. It shouldn't be an issue if they're scoped; the compiler will type check and it won't be a problem.

Events

Something that caught me off guard: the interfaces for Event handlers don't inherit from IInspectable, dropping the pattern that "Everything is IInspectable". This seems somewhat weird; I thought that I should be able to implement say, Windows.UI.ActivatedEventHandler on any class I want, and since it's an interface, WinRT will do QueryInterface to get the correct interface before calling Invoke. This isn't the case as all; QueryInterface is never called for EventHanders, it just assumes you're giving it the right one. Presumably this makes it faster to call the Invoke behind the scenes but it seems to me a single QI at the beginning wouldn't be too much of a hit on performance. So EventHandlers all inherit from IUnknown. In my previous post I have an example of a wrapper class for an event that takes an arbitrary procedure reference; this is likely going to be a good way to create event delegate objects.

Generic Interfaces

Generics are used pretty heavily throughout WinRT. In the C++ language bindings, they end up translating down to a template class, and the pre-compiled headers actually have to parametrize every instance of a generic class. This is pretty ugly and it'd be much nicer to just declare the interface and have the compiler do the right thing. Delphi has a similar problem for similar reasons. Generic Interfaces are supported by the compiler, but the problem arises in that you cannot specify a GUID for parametrized instances of a given generic interface. So for now, I've had to create a unique interface for every parameterized interface in WinRT. But how do you go about assigning a GUID to the interfaces? The answer lies in RoGetParameterizedTypeInstanceIID.

Lets take a look at the IL of something that uses a generic. How about Windows.Storage.FilePropoerties.IStorageItem.ExtraProperties.SavePropertiesAsync:

.method public hidebysig newslot abstract virtual 
        instance class Windows.Storage.FileProperties.SavePropertiesOperation 
        SavePropertiesAsync([in] class [Windows.Foundation]Windows.Foundation.Collections.IIterable`1<class [Windows.Foundation]Windows.Foundation.Collections.IKeyValuePair`2<string,object>> propertiesToSave) cil managed
{
  .custom instance void [Windows.Foundation]Windows.Foundation.Metadata.OverloadAttribute::.ctor(string) = ( 01 00 13 53 61 76 65 50 72 6F 70 65 72 74 69 65   // ...SavePropertie
                                                                                                             73 41 73 79 6E 63 00 00 )                         // sAsync..
  .param [1]
  .custom instance void [Windows.Foundation]Windows.Foundation.Metadata.HasVariantAttribute::.ctor() = ( 01 00 00 00 ) 
} // end of method IStorageItemExtraProperties::SavePropertiesAsync

Actually let me touch on those attributes since that's sort of interesting. There is an OverloadAttribute, which says this method overloads another method named SavePropertiesAsync. This is sort of interesting because in C++, the methods can't be overloaded with the same name: This one is SavePropertiesAsync, the other is SavePropertiesAsyncOverloadDefault, as described by it's own OverloadAttribute. This is useful information if your language doesn't support overloading methods with different arguments. Delphi doesn't have that problem, since there are different arguments, we can name them both the same.

Okay, so the method has one parameter propertiesToSave, of type Windows.Foundation.Collections.IIterable`1<Windows.Foundation.Collections.IKeyValuePair`2<string,object>>. If you're not familiar with IL, the "[Windows.Foundation]" denotes that the type is a typeref, and the complete typedef can be found in the assembly (or in this case, metadata file) Windows.Foundation. The `1 and `2 after the generic type names is an indication of how many generic arguments there are. To me this seems like an odd convention because it should be given how many arguments there are going to be, and it's strange that it's called out in every type. But it's a clue to some of the new APIs, such as RoParseTypeName which will validate if a type name is valid, given the number of generic parameters, and for the IRoSimpleMetaDataBuilder sink for actually getting the GUID. In order to use RoGetParameterizedTypeInstanceIID, we need a class that implements IRoMetaDataLocator, which uses the Locate method as a call back to query for more information about "unknown" types. It actually ends up querying about every Interface type, but knows how to handle basic types such as "int", "string", and "object". Remember, in the context of WinRT, a string is a HSTRING, and a Object ends up being some class that implements IInvokable. But for the purposes of this API, they're "string" and "object".

So we need a class that implements IRoMetaDataLocator. There's a pretty big gotcha here that caught me off guard. The documentation for IRoMetaDataLocator states "The IRoMetaDataLocator interfaces inherits from the IUnknown interface." That statement isn't surprising, but what is surprising is it's absolutely false. Look at the definition from roparameterizediid.h:

struct IRoMetaDataLocator
{
    STDMETHOD(Locate)(
        PCWSTR                         nameElement,
        _In_ IRoSimpleMetaDataBuilder& metaDataDestination
    ) const = 0;
};

If I declared the interface as

type
  IRoMetaDataLocator = interface(IUnknown)
    function Locate(nameElement: PCWSTR; metaDataDestination: IRoSimpleMetaDataBuilder): HRESULT; stdcall;
  end;

My program would crash in QueryInterface. In Delphi, there is no such way to define a naked "interface" with no IUnknown methods. So I had to fake it with a virtual abstract class.

type
  IRoMetaDataLocator = class
  protected
    function Locate(nameElement: PCWSTR; 
      metaDataDestination: IRoSimpleMetaDataBuilder): HRESULT; virtual; stdcall; abstract;
  end;

Looking at it now, potentially I should be using a record to more closely match the C++. IRoSimpleMetaDataBuilder actually has the same issue: there is no IUnknown members for that interface.

As I said before, the locate method gets called with each part of the type that is being fetched, so the first time it's invoked, we get "Windows.Foundation.Collections.IIterable`1", the second time we get "Windows.Foundation.Collections.IKeyValuePair`2". We don't get called for "string" or "object" because it handles them internally. My Locator needs to lookup each string and provide the metaDataDestination with it's guid, and specify if it's generic or not. I'll give a sneak peek at that code, although it's using some helper classes that aren't publicly available to make using the Cor interfaces easier to use.

function TLocator.Locate(nameElement: PCWSTR;
  metaDataDestination: TRoSimpleMetaDataBuilder): HRESULT;
var
  name: HSTRING;
  imp: IMetaDataImport2;
  tok: mdTypeDef;
  md: TMetaDataImport;
  info: TTypeInfo;
  genArgs: TArray;
begin
  Result := S_OK;

  WindowsCreateString(nameElement, StrLen(nameElement), name );
  try
    RoGetMetaDataFile(name, nil, nil, @imp, @tok);
    if imp = nil then
      Exit(E_UNEXPECTED);

    md := TMetaDataImport.Create(imp);
    try
      info := md.GetTypeInfo(tok, genArgs);
      if info.GenericParamCount = 0 then
        metaDataDestination.SetWinRtInterface(info.Guid)
      else
        metaDataDestination.SetParameterizedInterface(info.Guid, info.GenericParamCount);
    finally
      md.Free;
    end;
  finally
    WindowsDeleteString(name);
  end;
end;

Note there are other types that could be passed in to a parameterized type, Enums, Delegates, or Structs. I'm ignoring these for now because I didn't actually encounter any API that uses those types.

The somewhat problematic thing about this is this might be difficult to include in a compiler, because the APIs are only available on Win8, so there's no way to generate the GUIDs from a older version of Windows. It'd be nice if Microsoft provided information on how those GUIDs are constructed so other tools that run on other Windows platforms can cross-build for Win8.

I hope this helps to de-mystify some of the information provided in the Windows WinMD metadata files.