Win7 SP1 OS bug - Do we need Windows 8 now?

It's the same Windows 7 x64 (WOW64) bug I reported in November last year (2010) - see the entry in Microsoft Connect.

And the new Windows 7 Service Pack 1 *HASN'T* fixed it! It doesn't only affect Windows 7 though, it goes back all the way to the very first version of x64 Windows - XP 64-bit.

There's no way to workaround this problem in user code. It is a fundamental design flaw in the very first version of 64-bit Windows. This makes it very hard (expensive) for Microsoft to fix - a proper fix requires a redesign of the whole emulation layer of 32-bit apps on 64-bit Windows. Microsoft charges a minimal sum for their OSes (sarcasm), so it's not like they have the resources to work on anything that doesn't affect everyone. It's very likely that Windows 7 will be forever plagued with this bug as it would seem to be easier (cheaper) to start from a clean sheet with Windows 8. Then again, why bother? It's not as if Microsoft has any real competition at all in the x64 OS market - they own more than 80% of the pie!

From the feedback of my previous post, the problem appears more widespread than Microsoft would want to acknowledge - see Discussions about the Boehm Garbage Collector (Boehm GC). Hopefully this bug gets fixed sooner rather than later.

In the meantime, if you want to test it out yourself, you can download this (extract all and run Tester.exe). Usually running it for a few minutes will get it to crash on x64. Give it 30 minutes if it doesn't. If it doesn't crash (i.e. on 32-bit OS), you'll have to kill the app via task manager after you've tested it though (an unrelated bug in the tester application prevents it from shutting down completely - the app will appear closed but the process remains running).

Fast memcpy for large blocks

Memory copy of 8MB blocks can be quite slow.

I found that both memcpy and CopyMemory won't utilize the full bandwidth of your RAM due to memory controller bottlenecks (I suspect the memory controller isn't smart enough to prefetch the right data). So this implementation by William Chan issues SSE2 prefetch instructions and gets the memory controller to literally stream the data back and forth from RAM in the fastest manner.

Note though, that you'll need to give it 16-byte aligned memory and it copies in 128-byte blocks.

The result is here (on my Core2Duo Wolfdale CPU @ 3.6GHz, dual channel DDR2 @ 800MHz):

memcpy/CopyMemory:
1871.775MB/sec

William Chan's SSE2 memcpy:
3540.471MB/sec

That's nearly double the speed of the naive memcpy!

Reporting a bug against Windows OS - possible?

Is Microsoft so arrogant that they think their OS is bug-free and that no one should ever need to report a bug against their OS?

There doesn't seem to be a way to report a bug against the Windows OS (no such category in Microsoft Connect). 2 years ago, I tried to report a bug against Vista - TreeView Indent in Vista causes HitTest to fail. The closest way would be to report it against VisualStudio. Surprise surprise, the VS product team closed it as external and did absolutely *nothing* after that. Sure, they redirected me to a non-existent website, which was supposed to be the MSDN forums, but that was where I was redirected to MS Connect in the first place! Talk about going around in circles... This really reminds me of Telstra (the ex-government owned now privatized biggest telco in Australia).

With regards to the bug that I've just reported, I'd expect them to do the same, and close it as external, not knowing which department they need to talk to. Microsoft is so big, dumb and slow that its right hand really have no idea what the left hand is doing! Sad really...


* Edit: Looks like I'm not the only one complaining:

Reporting a bug in Vista 64 WOW64
How do you file a bug report for Windows?
Problems with comdlg32.ocx, Windows Vista and long file names/extension´s

WOW64 bug: GetThreadContext() may return stale contents

My GC app runs perfectly fine under native x86 OS's starting from XP.

I've installed Windows 7 x64 recently (I've always used Win7-x86 due to 64-bit being in its infancy yet for the stuff that I do on my desktop) and to my dismay the app fails after a few minutes of running.

After hours of debugging (GC is never easy to debug, let alone a hypothesis that involves a bug in WOW64 which has been used by millions since XP-64), I found that WOW64 clobbers the ESP value (as returned by GetThreadContext(), of which the the GC thread in the app relies upon to get the current stack pointer of mutator threads) when it does a call out to long mode. I also found that ESP is always restored upon returning.

Prior to calling GetThreadContext(), the GC thread suspends all mutator threads. If it just so happens to suspend the mutator thread while it's running long mode code in user mode, the ESP value gets changed to a value indicating a higher address than the actual stack pointer (remember on IA, stack 'grows' downwards). I've seen this happen for SetEvent() and SwitchToThread() (as these are the most frequently called kernel functions in the app).

This means that either SuspendThread is suspending a thread in an incompatible way to native x86, or the thread's context in WOW64 is not being protected when the code jumps to translation mode. Either way, I was sure it's a bug.

I then found this article (difference between WOW64 and native x86 system DLLs) and while the article isn't exactly addressing the issue I'm facing, I found it very useful because this guy (Skywing from Microsoft) certainly knows WOW64 very well. I proceeded to email him and he replied with the following:

[...] there’s an issue with get and set context operations against amd64 Wow64 threads returning bad information in circumstances when the thread is running long mode code in user mode. This relates to us [Microsoft] pulling the Wow64 context from the TLS slot (as described in the [this] article) before that context structure has been updated with current contents.

That sounded very much like the issue here. So, I decided to dig deeper and put a few software traps to try and catch it in the act.

This is what I found.

The stale contents from GetThreadContext() actually came from the previous system call out (a looong way up the stack really - it's not as if it's a few instructions ago). It should've returned contents from the *current* system call out instead (or to be precise, just before the call out to long mode took place). Like Skywing said, they pulled the context before it's updated with the current contents.

With that said, we can now conclude that it is indeed an OS bug (Win7 SP1 hasn't fixed it).

Update 29 March 2014: As of Windows 8.1, this bug is still *NOT* fixed!

* I'd like to thank Skywing for his effort in assisting me to root cause this issue.

BCC32 the Optimizing Compiler?

We all know asking the bcc32 compiler team to do put more effort into making the codegen generate more efficient / optimized asm is a *BIG ASK*, seeing how they haven't even got the basics (compiler bugs that Microsoft VC++ compiler fixed since MSVC 2005) working correctly. VC++ 6 was worse than C++ Builder 6, but since then, Microsoft have worked hard and fixed up their compiler, made it fully optimizing and generate one of the fastest and most efficient code in the world. It is also now one of the most compliant C++ compilers of all.

Let's compare the following code sample.

For the uninformed, the following usage pattern is found in a lot of expanded template code (I'm using it a lot in my GC framework and Boost uses it too).

#include <tchar.h>

struct foo
{
    inline operator bool() const
    {
        return false;
    }
};

int _tmain(int argc, _TCHAR* argv[])
{
    if (foo())
        return 0; // *see footnote
    return 0;
}

*footnote: usually this is something more meaningful, but here I'm trying to illustrate how massively unintelligent bcc32
is.


bcc32 (C++ Builder XE) command line:

    bcc32 -O2 -Hs- -C8 -v- -vi test8.cpp

generated asm:

push ebp
mov ebp,esp
add esp,-$08
push edi
lea edi,[ebp-$08]
xor eax,eax
mov ecx,$00000008
rep stosb
lea eax,[ebp-$08]
xor edx,edx
test dl,dl
jz $00401201
xor eax,eax
jmp $00401203
xor eax,eax
pop edi
pop ecx
pop ecx
pop ebp
ret

cl command line:

    cl /Ox /Ot test8.cpp

generated asm:

xor eax,eax
ret

BCC32 generated 18 lines of useless opcodes when really only 2 are required. With BCC32, your code would be a few hundred times slower (taking into account of memory access latencies).

SerialPort IOException Workaround in C#

As promised, I've whipped up a quick workaround to fix the problem as described here.

Here's the code:

// Copyright 2010-2014 Zach Saw // // Licensed under the Apache License, Version 2.0 (the "License"); // you may not use this file except in compliance with the License. // You may obtain a copy of the License at // // http://www.apache.org/licenses/LICENSE-2.0 // // Unless required by applicable law or agreed to in writing, software // distributed under the License is distributed on an "AS IS" BASIS, // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. // See the License for the specific language governing permissions and // limitations under the License. using System; using System.IO; using System.IO.Ports; using System.Runtime.InteropServices; using System.Text; using Microsoft.Win32.SafeHandles; namespace SerialPortTester { public class SerialPortFixer : IDisposable { public static void Execute(string portName) { using (new SerialPortFixer(portName)) { } } #region IDisposable Members public void Dispose() { if (m_Handle != null) { m_Handle.Close(); m_Handle = null; } } #endregion #region Implementation private const int DcbFlagAbortOnError = 14; private const int CommStateRetries = 10; private SafeFileHandle m_Handle; private SerialPortFixer(string portName) { const int dwFlagsAndAttributes = 0x40000000; const int dwAccess = unchecked((int) 0xC0000000); if ((portName == null) || !portName.StartsWith("COM", StringComparison.OrdinalIgnoreCase)) { throw new ArgumentException("Invalid Serial Port", "portName"); } SafeFileHandle hFile = CreateFile(@"\\.\" + portName, dwAccess, 0, IntPtr.Zero, 3, dwFlagsAndAttributes, IntPtr.Zero); if (hFile.IsInvalid) { WinIoError(); } try { int fileType = GetFileType(hFile); if ((fileType != 2) && (fileType != 0)) { throw new ArgumentException("Invalid Serial Port", "portName"); } m_Handle = hFile; InitializeDcb(); } catch { hFile.Close(); m_Handle = null; throw; } } [DllImport("kernel32.dll", CharSet = CharSet.Auto, SetLastError = true)] private static extern int FormatMessage(int dwFlags, HandleRef lpSource, int dwMessageId, int dwLanguageId, StringBuilder lpBuffer, int nSize, IntPtr arguments); [DllImport("kernel32.dll", CharSet = CharSet.Auto, SetLastError = true)] private static extern bool GetCommState(SafeFileHandle hFile, ref Dcb lpDcb); [DllImport("kernel32.dll", CharSet = CharSet.Auto, SetLastError = true)] private static extern bool SetCommState(SafeFileHandle hFile, ref Dcb lpDcb); [DllImport("kernel32.dll", CharSet = CharSet.Auto, SetLastError = true)] private static extern bool ClearCommError(SafeFileHandle hFile, ref int lpErrors, ref Comstat lpStat); [DllImport("kernel32.dll", CharSet = CharSet.Auto, SetLastError = true)] private static extern SafeFileHandle CreateFile(string lpFileName, int dwDesiredAccess, int dwShareMode, IntPtr securityAttrs, int dwCreationDisposition, int dwFlagsAndAttributes, IntPtr hTemplateFile); [DllImport("kernel32.dll", SetLastError = true)] private static extern int GetFileType(SafeFileHandle hFile); private void InitializeDcb() { Dcb dcb = new Dcb(); GetCommStateNative(ref dcb); dcb.Flags &= ~(1u << DcbFlagAbortOnError); SetCommStateNative(ref dcb); } private static string GetMessage(int errorCode) { StringBuilder lpBuffer = new StringBuilder(0x200); if ( FormatMessage(0x3200, new HandleRef(null, IntPtr.Zero), errorCode, 0, lpBuffer, lpBuffer.Capacity, IntPtr.Zero) != 0) { return lpBuffer.ToString(); } return "Unknown Error"; } private static int MakeHrFromErrorCode(int errorCode) { return (int) (0x80070000 | (uint) errorCode); } private static void WinIoError() { int errorCode = Marshal.GetLastWin32Error(); throw new IOException(GetMessage(errorCode), MakeHrFromErrorCode(errorCode)); } private void GetCommStateNative(ref Dcb lpDcb) { int commErrors = 0; Comstat comStat = new Comstat(); for (int i = 0; i < CommStateRetries; i++) { if (!ClearCommError(m_Handle, ref commErrors, ref comStat)) { WinIoError(); } if (GetCommState(m_Handle, ref lpDcb)) { break; } if (i == CommStateRetries - 1) { WinIoError(); } } } private void SetCommStateNative(ref Dcb lpDcb) { int commErrors = 0; Comstat comStat = new Comstat(); for (int i = 0; i < CommStateRetries; i++) { if (!ClearCommError(m_Handle, ref commErrors, ref comStat)) { WinIoError(); } if (SetCommState(m_Handle, ref lpDcb)) { break; } if (i == CommStateRetries - 1) { WinIoError(); } } } #region Nested type: COMSTAT [StructLayout(LayoutKind.Sequential)] private struct Comstat { public readonly uint Flags; public readonly uint cbInQue; public readonly uint cbOutQue; } #endregion #region Nested type: DCB [StructLayout(LayoutKind.Sequential)] private struct Dcb { public readonly uint DCBlength; public readonly uint BaudRate; public uint Flags; public readonly ushort wReserved; public readonly ushort XonLim; public readonly ushort XoffLim; public readonly byte ByteSize; public readonly byte Parity; public readonly byte StopBits; public readonly byte XonChar; public readonly byte XoffChar; public readonly byte ErrorChar; public readonly byte EofChar; public readonly byte EvtChar; public readonly ushort wReserved1; } #endregion #endregion } internal class Program { private static void Main(string[] args) { SerialPortFixer.Execute("COM1"); using (SerialPort port = new SerialPort("COM1")) { port.Write("test"); } } } }

* Use Firefox to copy and paste the code above. Formatting won't be preserved if you use IE (IE bug).

.NET SerialPort Woes

Preface

This is a very long post articulating the .NET SerialPort bug and the proposed fix for Microsoft to implement in its post .NET 4.0 framework. An interim fix that doesn't involve Microsoft is also available.


Introduction

The built-in serial port support in .NET has been a major let down (which has remained largely unchanged since its introduction in v2.0 to the latest v4.0). Posts on MSDN have suggested that a lot of people (both C# and VB users alike) are in fact facing some form of difficulties using System.IO.Ports.SerialPort:

IOException when reading serial port using .NET 2.0 SerialPort.Read

Port IOException

SerialPort.Close throws IOException

IOException when SerialPort.Open()

WinCE 5.0 - IOException when serialPort.Open()

WARNING! SerialPort in .NET 3.5

... and many more, but take the last one with a grain of salt.

Yet Microsoft could't seem to be able to reproduce the bug, or worse, brushed it aside thinking it's a problem with users' code. That is likely due to the noise (i.e. incorrect answers accepted as correct answers) introduced by the forum's serial port expert pretenders (a lot of them Microsoft support staffs) - so far none of the posts were answered correctly, yet were all marked as correctly answered! Yay to the quality of MSDN forum - a forum where n00bs answer questions by other n00bs. The truth is, there's only one similarity in all of the posts - they all encountered IOException.


The Problem - IOException

To understand why IOException occurs in SerialPort (or rather, SerialStream to be exact), one would only need to look as far as the WinAPI Comm functions. SerialStream calls several comm APIs to get the real job done and when the API functions fail and return an error, SerialStream simply throws an IOException with the message returned by FormatMessage with the error code from GetLastError.

So why does the WinAPI function fail? From the posts, they all have a common error message:

"The I/O operation has been aborted because of either a thread exit or an application request." (error code 995)

While a thread exit will also cause an overlapped (asynchronous) function to implicitly abort, in this case it's aborted because the serial port is in a mode where any errors encountered by the UART chip would trigger the abort flag causing all current and subsequent serial port related calls to abort. Errors include parity error, buffer overrun, etc.

Some developers have even encountered IOException as soon as they call SerialPort.Open(), especially in slow devices such as handhelds running .NET CE. Some encounter it when the garbage collector disposes the serial port. Some encounter it when they call SerialPort.Read().

They're all due to a mistake in .NET's SerialStream implementation - neglecting to set the fAbortOnError flag in the DCB structure when initializing the serial port.

This negligence on Microsoft's part means every time you run your application you could potentially encounter a different behavior (this flag is persistent across app runs and the default is determined by BIOS and/or UART hardware vendor). Some claim that it only happens in one machine and not others. This also explains why it has remained such a pesky problem for both developers and Microsoft since the first incarnation of the SerialPort class.

When fAbortOnError flag is set to true, this is indeed the expected behavior - but is this the desired behavior Microsoft intended for its users? No. System.IO.Ports.SerialStream was never meant to work with fAbortOnError set to true, because the ClearCommError WinAPI function that goes hand-in-hand was nowhere to be found among its methods. Clearly, whoever wrote SerialStream made a mistake (and needs to be shot).


The Solution

It took me an entire day to root cause this problem. Luckily the solution is much simpler.
Here's what Microsoft needs to do to fix the problems (in reference to the .NET 4.0 source):

1) In InitializeDCB, SetDcbFlag for bit 14 to zero - this sets fAbortOnError to false. Also, retry GetCommState and SetCommState if it fails with error 995 (call ClearCommError() before retrying).

2) In SerialStream's c'tor, move InitializeDCB to the line before GetCommProperties. This fixes the problem for the folks who've been getting IOException when calling SerialPort.Open(). The reason SerialPort.Open() only failed on slow devices because between the port's CreateFile and the time GetCommProperties() is called, a comm port physical error might have already occurred.

The reason some people have claimed that their app simply crashes out when their application terminates is due to DiscardInBuffer() in SerialStream.Dispose() throwing IOException because PurgeComm failed with error 995, likely because of buffer overrun as their serial devices would've been sending and filling up the input buffer before user closes the app. And mind you, Dispose() at that point would've been called by the garbage collector thread - hence a try-catch would've been ineffective, unless of course, you've manually disposed the object prior to closing the app - causing the app to hard crash with unhandled exception.

How do you fix it in the interim? Simple. Before you call SerialPort.Open(), simply open the serial port by calling CreateFile and SetCommState's fAbortOnError to false. Now you can safely open the serial port without worrying that it might throw an IOException. I've whipped up a sample workaround in C# that you could use as a reference.