Overloaded Methods
The codeEntityReference tag allows you to link to API reference documentation (namespaces, types, methods, properties, fields). You have to prefix the fully qualified name of the entity with N: for namespaces, T: for types, P: for properties, F: for fields and M: for methods. What if you have overloaded methods and you want to link to the page that displays all of them, not one particular method? The O: prefix Visual Studio is expecting (for XML comments) does not work. Sandcastle Help File Builder (SHFB) expects Overload: instead.

  <codeEntityReference>Overload:YourNamespace.YourClass.YouOverloadedMethod</codeEntityReference>

Methods with arguments
If you want to link to API reference documentation for a method that take some parameters you must specify those parameter’s type in parenthesis.

  <codeEntityReference>M:YourNamespace.YourClass.Function(System.String)</codeEntityReference>

However, if the method takes more than one parameter, you must separate the argument types with a comma and use no whitespaces between them. If you have whitespaces the API reference will not be resolved.

  <codeEntityReference>M:YourNamespace.YourClass.Function(System.String,System.Int32[])</codeEntityReference>

Constructors
Constructors are special methods so you must use a special notation for them: instead of using their name use the #ctor tag.

  <codeEntityReference>M:YourNamespace.YourClass.#ctor</codeEntityReference>

Of course, if the constructor has some arguments you must specify them in parenthesis:

  <codeEntityReference>M:YourNamespace.YourClass.#ctor(System.String)</codeEntityReference>

Generics
When it comes to generics you must use the special notation with ` (grave accent). The grave accent (`) indicates the level and the number after it is the number of the generic types. Here are the rules (as indicated in the ECMA specification ):

• Arguments that define generic type parameters have an appended grave accent character followed by the number of type parameters (ex. T`1)
• For nested types, the number is based upon the of new type parameters on the nested type(ex. T`1.Nested`2)
• Arguments that refer to generic type parameters on types are encoded using a single grave accent character (`) followed by the zero-based index of the type parameter.
• Arguments that use generic parameters on methods use double grave accent characters (“) followed by the zero-based index of the type-parameter instead of the single grave accent used for parameters on types.

For instance, let’s say your class looks like this:

namespace foo
{
   class bar<T>
   {
   }
}

If you want to link to the generic bar type you must do it like this:

  <codeEntityReference>T:foo.bar`1</codeEntityReference>

What if you had a generic method too?

namespace foo
{
   class bar<T>
   {
      public TR func1<TR>(string s)
      {
         return (default)TR; // just something dummy
      }

      public TR func2<TR;>(string s, TR arg)
      {
         return (default)TR; // just something dummy
      }

      public void func3<T1, T2>(T a, T1 b, T2 c)
      {
      }
   }
}

Then you must refer to the function as follows:

  // function is generic, but no generic argument is used
  <codeEntityReference>M:foo.bar`1.func1``1(System.String)</codeEntityReference>

  // function is generic and has a generic argument on the method => use `` + 0-based index of type
  <codeEntityReference>M:foo.bar`1.func2``1(System.String,``0)</codeEntityReference>

  // function is generic and has generic arguments on type and on the method
  // first argument type comes from the type => `0
  // second and third argument types come from the method, in the order they were declared => ``0 and ``1
  <codeEntityReference>M:foo.bar`1.func3``1(`0,``0,``1)</codeEntityReference>

API Reference Documentation
When you author documentation you probably want to put together conceptual documentation and API reference documentation. The later you build from external sources, i.e. assemblies and XML files with the XML-comments you wrote in the source code. Question is where and how do you insert this API reference? The answer was not very easy to find; luckily a colleague of mine was able to help me with the answer.

First of all, there is only one place in the hierarchy of your topics hierarchy where you can insert it. So no matter from how many sources you build this, it all goes into one place. To set the place you must edit the topics.xml file (under Help\Settings). This file looks like this (of course, this is a dummy sample):

<?xml version="1.0" encoding="utf-8"?>
<topics>
  <topic id="cec0ab12-05b5-4a00-ab8a-391d67625997" file="topic1.aml">
    <topic id="854aeed0-510e-4ca1-b2bb-12104ba9eada" file="topic1\topic11.aml">
  </topic>
</topics>

There is a special tag you can insert as a child of a topic. It’s called stoc and looks like this:

<?xml version="1.0" encoding="utf-8"?>
<topics>
  <topic id="cec0ab12-05b5-4a00-ab8a-391d67625997" file="topic1.aml">
    <stoc project="Project" />
  </topic>
</topics>

This will tell the builder to insert under topic1 all the API reference documentation.

Example
Let’s say we have the a simple COM local server called COM3264Server.exe. There is just one interface called ICoCOM3264Server. Here is the IDL file:

[
	object,
	uuid(733C70A7-F7EC-4C15-85D2-5CDB14F4110B),
	dual,
	nonextensible,
	pointer_default(unique)
]
interface ICoCOM3264Server : IDispatch{
   [id(1), helpstring("Says hello")] HRESULT SayHello(void);
};
[
	uuid(2F25FC66-2380-42FD-8476-8B5917FB1BF1),
	version(1.0),
]
library COM3264ServerLib
{
	importlib("stdole2.tlb");
	[
		uuid(9268A299-E817-4C5D-8627-C2582B66F16D)
	]
	coclass CoCOM3264Server
	{
		[default] interface ICoCOM3264Server;
	};
};

The implementation of the method SayHello() is straight forward; it just displays a message box with a text that varies between the two architectures, x64 and x86.

STDMETHODIMP CCoCOM3264Server::SayHello(void)
{
#ifdef _WIN64
   ::MessageBox(NULL, _T("Hello from 64-bit COM server!"), _T("COM3264Server"), MB_OK);
#else
   ::MessageBox(NULL, _T("Hello from 32-bit COM server!"), _T("COM3264Server"), MB_OK);
#endif

   return S_OK;
}

Registry
When you register the COM server, the 32-bit and 64-bit versions are registered in different parts of the registry. On 64-bit machine, the registry has two views (or modes):

• a native view, for 64-bit application; (e.g. registry path for CLSIDs is HKEY_CLASSES_ROOT\CLSID\)
• a WOW64 view, which enables redirections for 32-bit applications, a process transparent to the user (e.g. registry path for CLSIDs is HKEY_CLASSES_ROOT\Wow6432Node\CLSID\)

Here is the Registry registration of the (native) 64-bit COM server (notice the registry key in the status bar of the editor, and the path to the server executable).

On, the other hand, the 32-bit COM server is registered under the Wow6432 node.

So if both versions are registered in Windows Registry, which one is picked? Well, both the server and the client can specify which architecture to use:

• the COM server can do this via the PreferredServerBitness Registry value
• the client can do this using one of the flags CLSCTX_ACTIVATE_32_BIT_SERVER and CLSCTX_ACTIVATE_64_BIT_SERVER, which overrides the server preference

If neither the client nor the server specifies a preference, then:

• If the computer that hosts the server is running Windows Server 2003 with Service Pack 1 (SP1) or a later system, then COM will try to match the server architecture to the client architecture. In other words, for a 32-bit client, COM will activate a 32-bit server if available; otherwise it will activate a 64-bit version of the server. For a 64-bit client, COM will activate a 64-bit server if available; otherwise it will activate a 32-bit server.
• If the computer that hosts the server is running Windows XP or Windows Server 2003 without SP1 or later installed, then COM will prefer a 64-bit version of the server if available; otherwise it will activate a 32-bit version of the server.

Server Preference
The server can specify its preferred architecture in the PreferredServerBitness value under AppId (available only on 64-bit Windows). This integer value can be:

• 1: Match the server architecture to the client architecture. For example, if the client is 32-bit, use a 32-bit version of the server, if it is available. If not, the client’s activation request will fail.
• 2: Use a 32-bit version of the server. If one does not exist, the client’s activation request will fail.
• 3: Use a 64-bit version of the server. If one does not exist, the client’s activation request will fail.

Here is the value set in Registry to specify the 64-bit architecture.

When you run the client, it launches the 64-bit version and in my example the following window pops-up:

If I change the value to indicate the 32-bit architecture, the other COM server is launched and the displayed message is:

Client Preference
The client code I used so far looked like this:

   ICoCOM3264Server* pServer;

   HRESULT hr = ::CoCreateInstance(
      CLSID_CoCOM3264Server,
      NULL,
      CLSCTX_LOCAL_SERVER|CLSCTX_REMOTE_SERVER,
      IID_ICoCOM3264Server,
      (void**)&pServer);

   if(SUCCEEDED(hr))
   {
      pServer->SayHello();

      pServer->Release();
   }

However, the 64-bit version of Windows added new flags to the CLSCTX enumeration.

• CLSCTX_ACTIVATE_32_BIT_SERVER: used to activate or connect to a 32-bit version of the server; fail if one is not registered.
• CLSCTX_ACTIVATE_64_BIT_SERVER: used to activate or connect to a 64 bit version of the server; fail if one is not registered.

As I mentioned earlier, if the client uses one of these flags, it overrides the server preference (specified via the PreferredServerBitness Registry value).

In the following example, the client requests the 64-bit COM server:

   HRESULT hr = ::CoCreateInstance(
      CLSID_CoCOM3264Server,
      NULL,
      CLSCTX_LOCAL_SERVER|CLSCTX_REMOTE_SERVER|CLSCTX_ACTIVATE_64_BIT_SERVER,
      IID_ICoCOM3264Server,
      (void**)&pServer);

And if you run it, no matter what the server specified, the 64-bit COM server is launched.

To read more about the subject see the MSDN links above.

Now, the same rule applies in the COM world. Your coclasses can either be thread-safe or not. If the are thread-safe, you can access one object from different threads at a time, otherwise not. Here enter the apartments. So what is an apartment? An apartment is an environment in which COM objects can live. It’s not a thread, nor a process, but it handles access from COM clients to COM objects. There are several types of apartments: single-threaded apartments (STA), multi-threaded apartments (MTA) and neutral-threaded apartments (NTA).

Single-Threaded Apartments
An STA allows only one thread at a time to access a COM object. This is achieved using a hidden window with a message pump. Calls from clients living in different threads are queued with the message pump. Only when the current call from a thread ends, the next call can proceed.

Assume you had a COM object called CoUserGroups that implements an interface IUserGroups that provides two methods: one called Add that adds a new user group, and one called Delete that deletes a user group. Since both methods work on the same list of user groups, adding and deleting is not possible at the same time without synchronizing access. But since such synchronization was not implemented, the COM object specifies that it can leave only in an STA, and let the apartment do the synchronization of calls from clients living in different threads.

Multi-Threaded Apartments
An MTA allows any number of threads to access a COM objects. However, the COM objects must be thread-safe, otherwise your application will behave unexpectedly and even crash.

Going back to the previous example, if CoUserGroups was implemented in a thread-safe manner, then it would be possible for clients living in different threads to access it. In this case there would be no need for an apartment level synchronization. The COM coclass specifies that it can live in an MTA and when simultaneous calls from different threads are received they are directed immediately to the COM object. This situation is shown in the next image.

Neutral-Threaded Apartments
NTAs, like MTAs, allow multiple threads to enter one apartment, but once a thread entered the apartment it acquires an apartment wide lock that will block the other threads, until the current thread exists the apartment. Calls into NTA (from STA or MTA in the same process) do not generate context switches; the thread leaves the apartment in which it executes and enters the NTA without any context switch, which increases performance. This model was introduced with COM+ (in Windows 2000) and is meant for components with no user interface.

A process can contain several apartments:

• zero or one MTA
• zero or one NTA
• zero, one or several STAs; the first STA created for a process is called the main STA

As a COM client, you specify the apartment you want to join with a call to CoInitializeEx(). This methods must be called from each thread.

HRESULT CoInitializeEx(void * pvReserved, DWORD dwCoInit);

The second parameters is a set of flags specifying the initialization options for the thread. To join the unique MTA, use COINIT_MULTITHREADED. To join a new or existing STA, use COINIT_APARTMENTTHREADED. Function CoInitialize() calls CoInitializeEx() specifying COINIT_APARTMENTTHREADED for the flags.

How to specify the threading model allowed for a coclass?
A coclass can specify the type of apartment it can join. If you’re using ATL you can specify that when you create the coclass. The next image shows the available options:

What they mean:

• Single: object wants to join the main STA (the first STA created into the process)
• Apartment: object wants to join one of the STAs in the process
• Both: object wants to join either an STA or the MTA
• Free: object wants to join the MTA
• Neutral: object wants to join the NTA

ATL adds the appropriate value to the registry script it creates for your coclass. COM depends entirely on the registry, and the threading model is also specified in the registry. Here is an example:

HKCR
{
	NoRemove CLSID
	{
		ForceRemove {80CFA233-86CC-44E3-9A62-BC498D8F2A0E} = s 'CoUserGroups Class'
		{
			ForceRemove Programmable
			InprocServer32 = s '%MODULE%'
			{
				val ThreadingModel = s 'Apartment'
			}
			TypeLib = s '{B8965B61-2A5A-4F34-B9F8-D8859452D345}'
			Version = s '1.0'
		}
	}
}

When that is merged into the Windows Registry, it looks like this:

The possible values in registry are:

• no value specified: equivalent of ATL ‘Single’
• Apartment: equivalent of ATL ‘Apartment’
• Free: equivalent of ATL ‘Free’
• Both: equivalent of ATL ‘Both’
• Neutral: equivalent of ATL ‘Neutral’

If you want to read more about COM apartments I suggests articles like this one.

[Guid("2F8433FE-4771-4037-B6B2-ED5F6585ED04")]
[InterfaceType(ComInterfaceType.InterfaceIsIDispatch)]
public interface IAccounts
{
      [DispId(1)]
      string[] GetUsers();
}

Method GetUsers() returns an array on string representing the user names. But what if you also wanted the user passwords or addresses? Since this is exposed as COM, you cannot return an array of User. But you can return multiple arrays of string. So, how would you deal with out string[]? This is what I want to show you in this tutorial.

This is a .NET interface exposed to COM. It has two methods, GetUsers() that returns an array of string representing user names, and GetUsers2() that returns an array of strings as an output parameters and a bool as return type, indicating whether any user was found.

namespace SampleLibrary
{
   [Guid("2F8433FE-4771-4037-B6B2-ED5F6585ED04")]
   [InterfaceType(ComInterfaceType.InterfaceIsIDispatch)]
   public interface IAccounts
   {
      [DispId(1)]
      string[] GetUsers();

      [DispId(2)]
      bool GetUsers2(out string [] users);
   }
}

And this is the implementation:

namespace SampleLibrary
{
   [Guid("C4713144-5D29-4c65-BF9C-188B1B7CD2B6")]
   [ClassInterface(ClassInterfaceType.None)]
   [ProgId("SampleLibrary.DataQuery")]
   public class Accounts : IAccounts
   {
      List< string > m_users;

      public Accounts()
      {
         m_users = new List< string > {
            "marius.bancila",
            "john.doe",
            "anna.kepler"
         };
      }

      #region IDataQuery Members

      public string[] GetUsers()
      {
         return m_users.ToArray();
      }

      public bool GetUsers2(out string[] users)
      {
         users = m_users.ToArray();

         return users.Length > 0;
      }

      #endregion
   }
}

Note: If you are trying this example make sure you set the ComVisible attribute to true, either for each type or per assembly (in AssemblyInfo.cs)

[assembly: ComVisible(true)]

Second, you have to check the “Register for COM interop” setting in the Build page of the project properties.

The first thing to do in C++ is importing the .TLB file that was generated by regasm.exe.

#import "SampleLibrary.tlb"
using namespace SampleLibrary;

If we look in the .TLB file, we can see how the IAccounts interface looks like:

struct __declspec(uuid("2f8433fe-4771-4037-b6b2-ed5f6585ed04"))
IAccounts : IDispatch
{
    //
    // Wrapper methods for error-handling
    //

    // Methods:
    SAFEARRAY * GetUsers ( );
    VARIANT_BOOL GetUsers2 (
        SAFEARRAY * * users );
};

The following C++ functions, GetUsers1() retrieves the users users list using method GetUsers() from IAccounts. It puts the users in a CStringArray (notice that this container does not have an assignment operator, so the only way to return such an array is with a reference in the parameters list).

void GetUsers1(CStringArray& arrUsers)
{
   IAccountsPtr pAccounts(__uuidof(Accounts));

   SAFEARRAY* sarrUsers = pAccounts->GetUsers();

   _variant_t varUsers;
   varUsers.parray = sarrUsers;
   varUsers.vt = VT_ARRAY | VT_BSTR;

   UnpackBstrArray(varUsers, arrUsers);
   SafeArrayDestroy(sarrUsers);

   pAccounts->Release();
}

UnpackBstrArray() is a function (see below) that extracts the elements of a SAFEARRAY and adds them to a CStringArray.

Function GetUsers2() uses the second method, GetUsers2() from IAccounts. This needs the address of a pointer to a SAFEARRAY (i.e. SAFEARRAY**) that will hold the values returned by the COM method. This time we have to create an empty SAFEARRAY and then pass its address to the COM method. The rest is similar to the previous case.

void GetUsers2(CStringArray& arrUsers)
{
   IAccountsPtr pAccounts(__uuidof(Accounts));

   SAFEARRAYBOUND aDim[1];
   aDim[0].lLbound = 0;
   aDim[0].cElements = 0;

   SAFEARRAY* sarrUsers = SafeArrayCreate(VT_BSTR, 1, aDim);

   VARIANT_BOOL ret = pAccounts->GetUsers2(&sarrUsers);
   if(ret != VARIANT_FALSE)
   {
      _variant_t varUsers;
      varUsers.parray = sarrUsers;
      varUsers.vt = VT_ARRAY | VT_BSTR;
      UnpackBstrArray(varUsers, arrUsers);
   }

   SafeArrayDestroy(sarrUsers);

   pAccounts->Release();
}

The helper method UnpackBstrArray() used previous looks like this:

void UnpackBstrArrayHelper(VARIANT* pvarArrayIn, CStringArray* pstrarrValues)
{
   if (!pstrarrValues || !pvarArrayIn || pvarArrayIn->vt == VT_EMPTY)
      return;

   pstrarrValues->RemoveAll();

   VARIANT* pvarArray = pvarArrayIn;
   SAFEARRAY* parrValues = NULL;

   SAFEARRAYBOUND arrayBounds[1];
   arrayBounds[0].lLbound = 0;
   arrayBounds[0].cElements = 0;

   if((pvarArray->vt & (VT_VARIANT|VT_BYREF|VT_ARRAY)) == (VT_VARIANT|VT_BYREF) &&
      NULL != pvarArray->pvarVal &&
      (pvarArray->pvarVal->vt & VT_ARRAY))
   {
      pvarArray = pvarArray->pvarVal;
   }

   if (pvarArray->vt & VT_ARRAY)
   {
      if (VT_BYREF & pvarArray->vt)
         parrValues = *pvarArray->pparray;
      else
         parrValues = pvarArray->parray;
   }
   else
      return;

   if (parrValues != NULL)
   {
      HRESULT hr = SafeArrayGetLBound(parrValues, 1, &arrayBounds[0].lLbound);
      hr = SafeArrayGetUBound(parrValues, 1, (long*)&arrayBounds[0].cElements);
      arrayBounds[0].cElements -= arrayBounds[0].lLbound;
      arrayBounds[0].cElements += 1;
   }

   if (arrayBounds[0].cElements > 0)
   {
      for (ULONG i = 0; i < arrayBounds[0].cElements; i++)
      {
         LONG lIndex = (LONG)i;
         CString strValue = _T("");

         VARTYPE vType;
         BSTR bstrItem;

         ::SafeArrayGetVartype(parrValues, &vType);
         HRESULT hr = ::SafeArrayGetElement(parrValues, &lIndex, &bstrItem);

         if(SUCCEEDED(hr))
         {
            switch(vType)
            {
            case VT_BSTR:
               strValue = (LPCTSTR)bstrItem;
               break;
            }

            ::SysFreeString(bstrItem);
         }

         pstrarrValues->Add(strValue);
      }
   }
}

void UnpackBstrArray( const _variant_t &var, CStringArray &strarrValues  )
{
   UnpackBstrArrayHelper( &(VARIANT)const_cast< _variant_t & >(var), &strarrValues );
}

Attached you can find a demo project (C# and C++) with the complete example show in this tutorial.

The seven lectures given by professor Feynman are:

• Law of Gravity
• The Relation of Mathematics and Physics
• The Great Conservation Principles
• Symmetry in Physical Law
• The Distinction of Past and Future
• Probability and Uncertainty – The Quantum Mechanical View of Nature
• Seeking New Laws

These are great lectures given by one of the greatest physicists of the 20th century. They really worth watching.

So far we have managed to parse the text representing an expression and build an abstract syntax tree. The only thing left, and the simplest of them all, is traversing this abstract syntax tree and evaluating the expression is represents.

In pseudo code, this would look like this:

double Evaluate(subtree)
{
   if(subtree is numeric)
      return value;
   else
   {
      op = subtree.operator
      v1 = Evaluate(subtree.left)
      v2 = Evaluate(subtree.right)
      return v1 op v2;
   }
}

We actually have to check for one more type of node, the one representing an unary expression. However, the evaluation function is as simple as this:

class Evaluator
{
   double EvaluateSubtree(ASTNode* ast)
   {
      if(ast == NULL)
         throw EvaluatorException("Incorrect syntax tree!");

      if(ast->Type == NumberValue)
         return ast->Value;
      else if(ast->Type == UnaryMinus)
         return -EvaluateSubtree(ast->Left);
      else
      {
         double v1 = EvaluateSubtree(ast->Left);
         double v2 = EvaluateSubtree(ast->Right);
         switch(ast->Type)
         {
         case OperatorPlus:  return v1 + v2;
         case OperatorMinus: return v1 - v2;
         case OperatorMul:   return v1 * v2;
         case OperatorDiv:   return v1 / v2;
         }
      }

      throw EvaluatorException("Incorrect syntax tree!");
   }

public:
   double Evaluate(ASTNode* ast)
   {
      if(ast == NULL)
         throw EvaluatorException("Incorrect abstract syntax tree");

      return EvaluateSubtree(ast);
   }
};

The exception class is defined as:

class EvaluatorException : public std::exception
{
public:
   EvaluatorException(const std::string& message):
      std::exception(message.c_str())
      {
      }
};

So let’s try it:

void Test(const char* text)
{
   Parser parser;

   try
   {
      ASTNode* ast = parser.Parse(text);

      try
      {
         Evaluator eval;
         double val = eval.Evaluate(ast);

         std::cout << text << " = " << val << std::endl;
      }
      catch(EvaluatorException& ex)
      {
		  std::cout << text << " t " << ex.what() << std::endl;
      }

      delete ast;
   }
   catch(ParserException& ex)
   {
      std::cout << text << " t " << ex.what() << std::endl;
   }
}

int main()
{
   Test("1+2+3+4");
   Test("1*2*3*4");
   Test("1-2-3-4");
   Test("1/2/3/4");
   Test("1*2+3*4");
   Test("1+2*3+4");
   Test("(1+2)*(3+4)");
   Test("1+(2*3)*(4+5)");
   Test("1+(2*3)/4+5");
   Test("5/(4+3)/2");
   Test("1 + 2.5");
   Test("125");
   Test("-1");
   Test("-1+(-2)");
   Test("-1+(-2.0)");

   Test("   1*2,5");
   Test("   1*2.5e2");
   Test("M1 + 2.5");
   Test("1 + 2&5");
   Test("1 * 2.5.6");
   Test("1 ** 2.5");
   Test("*1 / 2.5");

   return 0;
}

The output of this test program is:

1+2+3+4 = 10
1*2*3*4 = 24
1-2-3-4 = -8
1/2/3/4 = 0.0416667
1*2+3*4 = 14
1+2*3+4 = 11
(1+2)*(3+4) = 21
1+(2*3)*(4+5) = 55
1+(2*3)/4+5 = 7.5
5/(4+3)/2 = 0.357143
1 + 2.5 = 3.5
125 = 125
-1 = -1
-1+(-2) = -3
-1+(-2.0) = -3
   1*2,5         Unexpected token ',' at position 6
   1*2.5e2       Unexpected token 'e' at position 8
M1 + 2.5         Unexpected token 'M' at position 0
1 + 2&5          Unexpected token '&' at position 5
1 * 2.5.6        Unexpected token '.' at position 7
1 ** 2.5         Unexpected token '*' at position 4
*1 / 2.5         Unexpected token '*' at position 1

And that's it. Define the grammar, build the parser, insert semantic actions and build the abstract syntax tree and then traverse it and evaluate the expression. If you are interested in understanding the grammar, and the parsing in a deeper manner than I presented in this posts, I suggest you read more articles. The purpose was not to teach compilers theory, but put it to a practical purpose.

Here you can download a Visual Studio 2008 project with the code contained in this tutorial.

An abstract syntax tree is a binary tree. The inner nodes will represent operators and leafs will be numerical values.

Here is how a node in the AST will look:

It is defined like this:

enum ASTNodeType
{
   Undefined,
   OperatorPlus,
   OperatorMinus,
   OperatorMul,
   OperatorDiv,
   UnaryMinus,
   NumberValue
};

class ASTNode
{
public:
   ASTNodeType Type;
   double      Value;
   ASTNode*    Left;
   ASTNode*    Right;

   ASTNode()
   {
      Type = Undefined;
      Value = 0;
      Left = NULL;
      Right = NULL;
   }

   ~ASTNode()
   {
      delete Left;
      delete Right;
   }
};

For the expression 1+2*3, the AST will be:

We’ll build this tree by inserting semantic actions and adding nodes according to the following rules:

You’ll probably notice that based on these rules the AST shown above will be modified a little bit, with some additional nodes for operators + and *, having on the left a leaf node with the neutral element for the operation (zero for + and 1 for *), and on the right a node corresponding to a TERM or FACTOR. This won’t affect the evaluation.

The Parser class will change so that the functions corresponding to the non-terminal symbols EXP, EXP1, TERM, TERM1 and FACTOR will return an ASTNode* instead of void. That is the node created as a semantic action.

class Parser
{
   Token m_crtToken;
   const char* m_Text;
   size_t m_Index;

private:

   ASTNode* Expression()
   {
      ASTNode* tnode = Term();
      ASTNode* e1node = Expression1();

      return CreateNode(OperatorPlus, tnode, e1node);
   }

   ASTNode* Expression1()
   {
      ASTNode* tnode;
      ASTNode* e1node;

      switch(m_crtToken.Type)
      {
      case Plus:
         GetNextToken();
         tnode = Term();
         e1node = Expression1();

         return CreateNode(OperatorPlus, e1node, tnode);

      case Minus:
         GetNextToken();
         tnode = Term();
         e1node = Expression1();

         return CreateNode(OperatorMinus, e1node, tnode);
      }

      return CreateNodeNumber(0);
   }

   ASTNode* Term()
   {
      ASTNode* fnode = Factor();
      ASTNode* t1node = Term1();

      return CreateNode(OperatorMul, fnode, t1node);
   }

   ASTNode* Term1()
   {
      ASTNode* fnode;
      ASTNode* t1node;

      switch(m_crtToken.Type)
      {
      case Mul:
         GetNextToken();
         fnode = Factor();
         t1node = Term1();
         return CreateNode(OperatorMul, t1node, fnode);

      case Div:
         GetNextToken();
         fnode = Factor();
         t1node = Term1();
         return CreateNode(OperatorDiv, t1node, fnode);
      }

      return CreateNodeNumber(1);
   }

   ASTNode* Factor()
   {
      ASTNode* node;
      switch(m_crtToken.Type)
      {
      case OpenParenthesis:
         GetNextToken();
         node = Expression();
         Match(')');
         return node;

      case Minus:
         GetNextToken();
		 node = Factor();
         return CreateUnaryNode(node);

      case Number:
         {
            double value = m_crtToken.Value;
            GetNextToken();
            return CreateNodeNumber(value);
         }

      default:
         {
            std::stringstream sstr;
            sstr << "Unexpected token '" << m_crtToken.Symbol << "' at position " << m_Index;
            throw ParserException(sstr.str(), m_Index);
         }
      }
   }

   ASTNode* CreateNode(ASTNodeType type, ASTNode* left, ASTNode* right)
   {
      ASTNode* node = new ASTNode;
      node->Type = type;
      node->Left = left;
      node->Right = right;

      return node;
   }

   ASTNode* CreateUnaryNode(ASTNode* left)
   {
      ASTNode* node = new ASTNode;
      node->Type = UnaryMinus;
      node->Left = left;
      node->Right = NULL;

      return node;
   }

   ASTNode* CreateNodeNumber(double value)
   {
      ASTNode* node = new ASTNode;
      node->Type = NumberValue;
      node->Value = value;

      return node;
   }

   void Match(char expected)
   {
      if(m_Text[m_Index-1] == expected)
         GetNextToken();
      else
      {
         std::stringstream sstr;
         sstr << "Expected token '" << expected << "' at position " << m_Index;
         throw ParserException(sstr.str(), m_Index);
      }
   }

   void SkipWhitespaces()
   {
      while(isspace(m_Text[m_Index])) m_Index++;
   }

   void GetNextToken()
   {
      SkipWhitespaces();

	  m_crtToken.Value = 0;
	  m_crtToken.Symbol = 0;

      if(m_Text[m_Index] == 0)
      {
         m_crtToken.Type = EndOfText;
         return;
      }

      if(isdigit(m_Text[m_Index]))
      {
         m_crtToken.Type = Number;
		 m_crtToken.Value = GetNumber();
         return;
      }

      m_crtToken.Type = Error;

      switch(m_Text[m_Index])
      {
      case '+': m_crtToken.Type = Plus; break;
      case '-': m_crtToken.Type = Minus; break;
      case '*': m_crtToken.Type = Mul; break;
      case '/': m_crtToken.Type = Div; break;
      case '(': m_crtToken.Type = OpenParenthesis; break;
      case ')': m_crtToken.Type = ClosedParenthesis; break;
      }

      if(m_crtToken.Type != Error)
	  {
         m_crtToken.Symbol = m_Text[m_Index];
         m_Index++;
	  }
      else
      {
         std::stringstream sstr;
         sstr << "Unexpected token '" << m_Text[m_Index] << "' at position " << m_Index;
         throw ParserException(sstr.str(), m_Index);
      }
   }

   double GetNumber()
   {
      SkipWhitespaces();

      int index = m_Index;
      while(isdigit(m_Text[m_Index])) m_Index++;
      if(m_Text[m_Index] == '.') m_Index++;
      while(isdigit(m_Text[m_Index])) m_Index++;

      if(m_Index - index == 0)
         throw ParserException("Number expected but not found!", m_Index);

      char buffer[32] = {0};
      memcpy(buffer, &m_Text[index], m_Index - index);

      return atof(buffer);
   }

public:
   ASTNode* Parse(const char* text)
   {
      m_Text = text;
      m_Index = 0;
      GetNextToken();

      return Expression();
   }
};

Now the Parse() method will return the created abstract syntax tree. We will see how to evaluate the expression by traversing this tree in the next post.

Having the grammar defined, we’ll make one function for each non-terminal symbol (EXP, EXP1, TERM, TERM1, FACTOR).

Simply put the code will look like this:

   void Expression()
   {
      Term();
      Expression1();
   }

   void Expression1()
   {
      switch(current_token)
      {
      case '+':
         GetNextToken();
         Term();
         Expression1();
         break;

      case '-':
         GetNextToken();
         Term();
         Expression1();
         break;
      }
   }

However, I want to make it a little bit more organized, so the first thing to do will be defining a Token structure that will indicate the type of last extracted token and if the case its value (for numbers). A token is basically a symbol extracted (one at a time) from the input text. The possible tokens will be the arithmetical operators (‘+’, ‘-’, ‘/’, ‘*’), the parentheses (‘(‘ and ‘)’), numbers and the end of the text.

Here is how I defined the token type and the token:

enum TokenType
{
   Error,
   Plus,
   Minus,
   Mul,
   Div,
   EndOfText,
   OpenParenthesis,
   ClosedParenthesis,
   Number
};

struct Token
{
   TokenType	Type;
   double		Value;
   char		Symbol;

   Token():Type(Error), Value(0), Symbol(0)
   {}
};

To be able to do the parsing, we’ll need some helper functions:

• SkipWhitespaces(), skips all whitespaces between two tokens:

     void SkipWhitespaces()
     {
        while(isspace(m_Text[m_Index])) m_Index++;
     }
  

• GetNextToken(), extracts the next token from the text; if an illegal token appears it throws an exception

     void GetNextToken()
     {
        // ignore white spaces
        SkipWhitespaces();
  
        m_crtToken.Value = 0;
        m_crtToken.Symbol = 0;
  
        // test for the end of text
        if(m_Text[m_Index] == 0)
        {
           m_crtToken.Type = EndOfText;
           return;
        }
  
        // if the current character is a digit read a number
        if(isdigit(m_Text[m_Index]))
        {
           m_crtToken.Type = Number;
           m_crtToken.Value = GetNumber();
           return;
        }
  
        m_crtToken.Type = Error;
  
        // check if the current character is an operator or parentheses
        switch(m_Text[m_Index])
        {
        case '+': m_crtToken.Type = Plus; break;
        case '-': m_crtToken.Type = Minus; break;
        case '*': m_crtToken.Type = Mul; break;
        case '/': m_crtToken.Type = Div; break;
        case '(': m_crtToken.Type = OpenParenthesis; break;
        case ')': m_crtToken.Type = ClosedParenthesis; break;
        }
  
        if(m_crtToken.Type != Error)
        {
           m_crtToken.Symbol = m_Text[m_Index];
           m_Index++;
        }
        else
        {
           std::stringstream sstr;
           sstr << "Unexpected token '" << m_Text[m_Index] << "' at position " << m_Index;
           throw ParserException(sstr.str(), m_Index);
        }
     }
  

• GetNumber() extracts a number from the input text from the current position; the purpose of this tutorial is didactical, so this function is quite simple: it reads integers and doubles with '.' As the decimal point; it doesn't read numbers in a format like 123.3E+2.

     double GetNumber()
     {
        SkipWhitespaces();
  
        int index = m_Index;
        while(isdigit(m_Text[m_Index])) m_Index++;
        if(m_Text[m_Index] == '.') m_Index++;
        while(isdigit(m_Text[m_Index])) m_Index++;
  
        if(m_Index - index == 0)
           throw ParserException("Number expected but not found!", m_Index);
  
        char buffer[32] = {0};
        memcpy(buffer, &m_Text[index], m_Index - index);
  
        return atof(buffer);
     }
  

With these defined, we can build the parser for the specified grammar.

class Parser
{
   Token m_crtToken;
   const char* m_Text;
   size_t m_Index;

private:

   void Expression()
   {
      Term();
      Expression1();
   }

   void Expression1()
   {
      switch(m_crtToken.Type)
      {
      case Plus:
         GetNextToken();
         Term();
         Expression1();
         break;

      case Minus:
         GetNextToken();
         Term();
         Expression1();
         break;
      }
   }

   void Term()
   {
      Factor();
      Term1();
   }

   void Term1()
   {
      switch(m_crtToken.Type)
      {
      case Mul:
         GetNextToken();
         Factor();
         Term1();
         break;

      case Div:
         GetNextToken();
         Factor();
         Term1();
         break;
      }
   }

   void Factor()
   {
      switch(m_crtToken.Type)
      {
      case OpenParenthesis:
         GetNextToken();
         Expression();
         Match(')');
         break;

      case Minus:
         GetNextToken();
         Factor();
         break;

      case Number:
         GetNextToken();
         break;

      default:
         {
            std::stringstream sstr;
            sstr << "Unexpected token '" << m_crtToken.Symbol << "' at position " << m_Index;
            throw ParserException(sstr.str(), m_Index);
         }
      }
   }

   void Match(char expected)
   {
      if(m_Text[m_Index-1] == expected)
         GetNextToken();
      else
      {
         std::stringstream sstr;
         sstr << "Expected token '" << expected << "' at position " << m_Index;
         throw ParserException(sstr.str(), m_Index);
      }
   }

   void SkipWhitespaces()
   {
      while(isspace(m_Text[m_Index])) m_Index++;
   }

   void GetNextToken()
   {
      // ignore white spaces
      SkipWhitespaces();

      m_crtToken.Value = 0;
      m_crtToken.Symbol = 0;

      // test for the end of text
      if(m_Text[m_Index] == 0)
      {
         m_crtToken.Type = EndOfText;
         return;
      }

      // if the current character is a digit read a number
      if(isdigit(m_Text[m_Index]))
      {
         m_crtToken.Type = Number;
         m_crtToken.Value = GetNumber();
         return;
      }

      m_crtToken.Type = Error;

      // check if the current character is an operator or parentheses
      switch(m_Text[m_Index])
      {
      case '+': m_crtToken.Type = Plus; break;
      case '-': m_crtToken.Type = Minus; break;
      case '*': m_crtToken.Type = Mul; break;
      case '/': m_crtToken.Type = Div; break;
      case '(': m_crtToken.Type = OpenParenthesis; break;
      case ')': m_crtToken.Type = ClosedParenthesis; break;
      }

      if(m_crtToken.Type != Error)
      {
         m_crtToken.Symbol = m_Text[m_Index];
         m_Index++;
      }
      else
      {
         std::stringstream sstr;
         sstr << "Unexpected token '" << m_Text[m_Index] << "' at position " << m_Index;
         throw ParserException(sstr.str(), m_Index);
      }
   }

   double GetNumber()
   {
      SkipWhitespaces();

      int index = m_Index;
      while(isdigit(m_Text[m_Index])) m_Index++;
      if(m_Text[m_Index] == '.') m_Index++;
      while(isdigit(m_Text[m_Index])) m_Index++;

      if(m_Index - index == 0)
         throw ParserException("Number expected but not found!", m_Index);

      char buffer[32] = {0};
      memcpy(buffer, &m_Text[index], m_Index - index);

      return atof(buffer);
   }

public:
   void Parse(const char* text)
   {
      m_Text = text;
      m_Index = 0;
      GetNextToken();

      Expression();
   }
};

The exception class is defined like this:

class ParserException : public std::exception
{
   int m_Pos;

public:
   ParserException(const std::string& message, int pos):
      std::exception(message.c_str()),
      m_Pos(pos)
   {
   }
};

As you can see, the code for the grammar production is quite simple and straight forward. Now, let's put it to the test.

void Test(const char* text)
{
   Parser parser;
   try
   {
      parser.Parse(text);
      std::cout << """ << text << ""t OK" << std::endl;
   }
   catch(ParserException& ex)
   {
      std::cout << """ << text << ""t " << ex.what() << std::endl;
   }
}

int main()
{
   Test("1+2+3+4");
   Test("1*2*3*4");
   Test("1-2-3-4");
   Test("1/2/3/4");
   Test("1*2+3*4");
   Test("1+2*3+4");
   Test("(1+2)*(3+4)");
   Test("1+(2*3)*(4+5)");
   Test("1+(2*3)/4+5");
   Test("5/(4+3)/2");
   Test("1 + 2.5");
   Test("125");
   Test("-1");
   Test("-1+(-2)");
   Test("-1+(-2.0)");

   Test("   1*2,5");
   Test("   1*2.5e2");
   Test("M1 + 2.5");
   Test("1 + 2&5");
   Test("1 * 2.5.6");
   Test("1 ** 2.5");
   Test("*1 / 2.5");

   return 0;
}

The output for this testing program is:

"1+2+3+4"        OK
"1*2*3*4"        OK
"1-2-3-4"        OK
"1/2/3/4"        OK
"1*2+3*4"        OK
"1+2*3+4"        OK
"(1+2)*(3+4)"    OK
"1+(2*3)*(4+5)"  OK
"1+(2*3)/4+5"    OK
"5/(4+3)/2"      OK
"1 + 2.5"        OK
"125"    OK
"-1"     OK
"-1+(-2)"        OK
"-1+(-2.0)"      OK
"   1*2,5"       Unexpected token ',' at position 6
"   1*2.5e2"     Unexpected token 'e' at position 8
"M1 + 2.5"       Unexpected token 'M' at position 0
"1 + 2&5"        Unexpected token '&' at position 5
"1 * 2.5.6"      Unexpected token '.' at position 7
"1 ** 2.5"       Unexpected token '*' at position 4
"*1 / 2.5"       Unexpected token '*' at position 1

Which is exactly what we expected: it validates correct expressions and throws an exception when the exception is incorrect.

In the next post I'll show how to modify this code to build an abstract syntax tree.

Let’s consider this expression: 1+2*3. The value of this expression is 7. But how do you evaluate it in a language like C++ if you get it as a string? First of all this is a so called “infix” notation. There are also prefix and postfix notation. The terms infix, prefix and postfix refer to the position of the operator related to the operands:

• Prefix: operator operand1 operand2 (ex: + 1 2)
• Infix: operand1 operator operand2 (ex: 1 + 2)
• Postfix: operand1 operand2 operator (ex: 1 2 +)

The human understandable notation is infix. But it turns out that trying to parse a string with infix expression, from left to right and evaluate it is not possible. Because you cannot now what in advance and operators have different precedence; and there are parentheses too.

To solve the problem you’d have to build a helper structure representing the infix expression. There are two possibilities:

• Reverse Polish Notation (RPN) implies transforming the infix expression in a postfix expression and then evaluating it from left to right. 1 + 2*3 is transformed into 1 2 3 * +. You go from left to right until you find an operator, evaluate the expression and then replace it in the stack.

• Abstract Syntax Tree (AST) is an abstract representation of an expression, with inner nodes representing operators and leafs representing numbers.

  

The RPN is harder to build and evaluate in my opinion, so I will focus on the approach with the AST.

We build an AST while parsing the expression. First, we’ll have to define the grammar for the expression. Otherwise we wouldn’t know what to parse.

EXP -> EXP + EXP | EXP - EXP | EXP * EXP | EXP / EXP | - EXP | (EXP) | number

First, this grammar is recursive, as you can see, but another important problem is that it does not represent the precedence of the operators. For this reasons, a better grammar is this:

EXP    -> EXP + TERM |
          EXP - TERM |
          TERM
TERM   -> TERM * FACTOR |
          TERM / FACTOR |
          FACTOR
FACTOR -> ( EXP ) | - EXP | number

These rules written above are called productions. The symbols used are:

• EXP, TERM, FACTOR are called non-terminal symbols
• +, -, /, *, (, ) number are called terminal symbols
• EXT is the start symbol

While the grammar has the correct operator precedence, it’s still recursive, or more precisely, left-recursive. You can see that EXP goes into EXP then operator + then TERM. You never reach to match operator + because you have start again and again with a new expression. There are techniques for eliminating this recursion and the result is:

EXP    -> TERM EXP1
EXP1   -> + TERM EXP1 |
          - TERM EXP1 |
          epsilon
TERM   -> FACTOR TERM1
TERM1  -> * FACTOR TERM1 |
          / FACTOR TERM1 |
          epsilon
FACTOR -> ( EXP ) | - EXP | number

‘epsilon’ here means ‘nothing’.

With the theory (well, this is just the tip of the iceberg, but should be a good start for you) in place we’ll have to do three things:

• Parse the expression
• Build the abstract syntax tree
• Evaluate the abstract syntax tree

The first two steps will be done the same time, but I’ll take them one at a time and explain it in details.

Before you continue with the implementation details, I suggest you read more about both RPN and AST and grammars.

Here are several references:

• Syntax, Grammar
• Arithmetic Expressions
• Abstract Syntax
• Top-Down Parsing
]]> http://mariusbancila.ro/blog/2009/02/03/evaluating-expressions-part-1/feed/ 5 http://mariusbancila.ro/blog/2008/03/26/arrays-in-f/ http://mariusbancila.ro/blog/2008/03/26/arrays-in-f/#comments Wed, 26 Mar 2008 08:31:31 +0000 Marius Bancila http://mariusbancila.ro/blog/?p=109 Yesterday I wrote about list in F#. Today I’ll write about arrays, which unlike lists are a mutable flat storage and cannot be resized. That means you have to create a new array if you want to remove or add elements. Advantages include constant look-up time and the fact that they can store a large amount of data.

You can create a literal array in a similar way with the lists, placing the elements between [| |]:

let data1 = [|1;2;3;4|]
printfn "data1: %a" output_any data1

data1: [|1; 2; 3; 4|]

The empty literal array is [||].

To create an array you can either use Array.create or Array.init. They both create and initialize an array, but the second makes a lambda expression, which allows advance initialization possibilities. The following creates an array with 10 elements initialized to 1:

let data2 = Array.create 10 1
printfn "data2: %a" output_any data2

Here is the output:

data2: [|1; 1; 1; 1; 1; 1; 1; 1; 1; 1|]

The same can be achieved using Array.init:

let data3 = Array.init 10 (fun x -> 1)
printfn "data3: %a" output_any data3

data3: [|1; 1; 1; 1; 1; 1; 1; 1; 1; 1|]

But we can use Array.init to initialize the elements from 1 to N for instance:

let data4 = Array.init 10 (fun x -> x+1)
printfn "data4: %a" output_any data4

data4: [|1; 2; 3; 4; 5; 6; 7; 8; 9; 10|]

The arrays are mutable data structures. Elements are accessed with .[] or .(). The following code shows how to set the elements of an array:

let data5 = Array.create 10 0
for i = 0 to (Array.length data5)-1 do
   data5.[i] <- i+1

printfn "data5: %a" output_any data5

data5: [|1; 2; 3; 4; 5; 6; 7; 8; 9; 10|]

You can iterate over the elements of an array with Array.iter and Array.iteri, the second also providing access to the index of the elements.

data4 |> Array.iter (fun x -> printf "%d " x)
printfn ""

data4 |> Array.iteri (fun i x -> printfn "data4(%d) = %d" i x)

1 2 3 4 5 6 7 8 9 10
data4(0) = 1
data4(1) = 2
data4(2) = 3
data4(3) = 4
data4(4) = 5
data4(5) = 6
data4(6) = 7
data4(7) = 8
data4(8) = 9
data4(9) = 10

Retrieving the length of the array can either be done with Array.length arr or with arr.Length.

for i = 0 to data4.Length-1 do
   printfn "data4(%d) = %d" i data4.(i)

data4(0) = 1
data4(1) = 2
data4(2) = 3
data4(3) = 4
data4(4) = 5
data4(5) = 6
data4(6) = 7
data4(7) = 8
data4(8) = 9
data4(9) = 10

Like the lists, arrays provide mapping that creates a new array by applying a function on all the elements of an array (with Array.map) or two arrays (with Array.map2).

let data6 = data4 |> Array.map (fun x -> x*2)
printfn "data6: %a" output_any data6

let data7 = Array.map2 (fun x y -> x+y) data4 data6
printfn "data7: %a" output_any data7

data6: [|2; 4; 6; 8; 10; 12; 14; 16; 18; 20|]
data7: [|3; 6; 9; 12; 15; 18; 21; 24; 27; 30|]

A copy of an array can be done with Array.copy.

let data7 = Array.copy data6
printfn "data7: %a" output_any data7

data7: [|2; 4; 6; 8; 10; 12; 14; 16; 18; 20|]

Appending elements to an array is also possible with Array.append, but the result is a new array, created by concatenating two arrays.

let data8 = Array.append data7 [|100|]
printfn "data8: %a" output_any data8

data8: [|2; 4; 6; 8; 10; 12; 14; 16; 18; 20; 100|]

The last operation of arrays I'm going to mention here is the folding, which allows applying a function to all the elements of an array, threading an accumulator argument in the process. The following example shows how to compute the sum of the elements of an array.

let data9 = [|1;2;3;4|]
let sum1 = (Array.fold_left (fun acc x-> x + acc) 0 data9)
let sum2 = (Array.fold_right (fun acc x-> x + acc) data9 0)
printfn "sum1 = %d" sum1
printfn "sum2 = %d" sum2

sum1 = 10
sum2 = 10

]]> http://mariusbancila.ro/blog/2008/03/26/arrays-in-f/feed/ 7