C#, UIAutomation and Chrome, enabling via IAccessible2

C#, UIAutomation and Chrome, enabling via IAccessible2

In the course of my work implementing data readers over various applications, I came across an application that required Chrome. My application is written in C# and uses the interop UIAutomation library to do most of its screen reading. Chrome doesn't enable this interface by default, so I started pursuing options.

One option is to use a command line switch (--force-renderer-accessibility) or to manually go into the chrome accessibility settings (chrome://accessibility/) and enable it there. Since this was being installed on end users systems, this didn't seem the right approach, as they are more for development. There was also a plugin that the application pushed on the user, which includes a launcher, and would void any command line switch we added.

Looking at the accessibility page for chromium ( https://www.chromium.org/developers/design-documents/accessibility ), I noticed that IAccessible2 was the interface of choice. I couldn't find much about using this interface from C#, but I did find some examples of code that worked in C++, including this piece from chromium test suites.

https://github.com/adobe/chromium/blob/master/chrome/browser/accessibility/accessibility_win_browsertest.cc

From there I was able to design this method, which probes the IAccessible2 interface of the target chrome window, and from there all the accessibility modes are enabled, including UIAutomation.

public static void GetIAccessible2(int pid)
        {
            //Guid guid = new Guid("6C496175-9160-5C3D-93EE-7DDB1E2D1CB0");
            Guid guid = new Guid("618736e0-3c3d-11cf-810c-00aa00389b71");
            Process proc = Process.GetProcessById(pid);
            IntPtr hwnd = proc.MainWindowHandle;

            Console.WriteLine(proc.MainWindowTitle);

            IntPtr ptrToObj = Marshal.AllocCoTaskMem(Marshal.SizeOf(typeof(Guid)));

            uint OBJECT_ID = 0xFFFFFFFC; // client: 0xFFFFFFFC window: 0x00000000

            IntPtr ptrAccObj = Marshal.AllocCoTaskMem(Marshal.SizeOf(typeof(IntPtr)));
            int retAcc = Win32.AccessibleObjectFromWindow(hwnd, OBJECT_ID, guid.ToByteArray(), out ptrAccObj);
            ptrToObj = Marshal.ReadIntPtr(ptrAccObj);

            Guid iAccessibleGuid = new Guid(0x618736e0, 0x3c3d, 0x11cf, 0x81, 0xc, 0x0, 0xaa, 0x0, 0x38, 0x9b, 0x71);
            IntPtr iAccessiblePtr = Marshal.AllocCoTaskMem(Marshal.SizeOf(typeof(Guid)));

            Guid iAccServiceProvider = new Guid("6d5140c1-7436-11ce-8034-00aa006009fa");
            int retQuery = Marshal.QueryInterface(ptrAccObj, ref iAccServiceProvider, out iAccessiblePtr);

            Accessibility.IAccessible acc = (Accessibility.IAccessible)Marshal.GetTypedObjectForIUnknown(iAccessiblePtr, typeof(Accessibility.IAccessible));

            IntPtr ptrAcc2 = Marshal.AllocCoTaskMem(Marshal.SizeOf(typeof(IntPtr)));

            var serviceProvider = (Win32.IServiceProvider)acc;
            Guid IID_IAccessible2 = new Guid(0xE89F726E, 0xC4F4, 0x4c19, 0xbb, 0x19, 0xb6, 0x47, 0xd7, 0xfa, 0x84, 0x78);
            try
            {
                serviceProvider.QueryService(ref IID_IAccessible2, ref IID_IAccessible2, out ptrAcc2);
            }
            catch
            {
                // ignore - don't care!!!
            }
            Marshal.FinalReleaseComObject(serviceProvider);
            Marshal.FreeCoTaskMem(ptrAcc2);
            Marshal.FreeCoTaskMem(ptrAccObj);
            //Marshal.FreeCoTaskMem(ptrToObj); Resulted in a corrupted heap error, because it is not a marshalled intptr
            Marshal.FinalReleaseComObject(acc);
            proc = null;
            //ParseChildren(new MSAAUIItem(acc));
            acc = null;
        }

This sometimes needs to be called twice to fully enable the accessibility interfaces. The document is initially presented as a Chrome Legacy Window. After the first run, the type changes to document, but still has no children. After the second run, we find the accessibility items for the rest of the document are presented.

I check this by getting the first child of the Chrome window, and seeing if it has children. If not, probe the IA2 interface. Doing this in a loop until we get children on the first child of the window, or 10 tries.