After the previous BSOD in Vista, I logged in to Vista interactively as an administrator (I usually run as a standard user), and I was greeted with a dialog informing me about a "serious error" or the like. I chose to check for updates to the problem. What came back was more than I expected, but not really all that helpful for my particular situation.

Problem caused by ATI Graphics Driver

This problem was caused by ATI Graphics Driver.

This program was created by ATI Technologies, Inc.. ATI Technologies, Inc. does not currently have a solution for the problem that you reported.



The following troubleshooting steps might prevent the problem from recurring.

Download and install an updated version of ATI Graphics Driver from one of the following locations:
Microsoft Update
ATI Technologies, Inc.

If an updated driver is not available for ATI Graphics Driver, check with your computer manufacturer.

If you are running the latest version of ATI Graphics Driver, contact ATI Technologies, Inc. for your support options.

Additional information

If this problem continues to occur after installing the latest product updates, we recommend you get assistance and troubleshooting information directly from ATI Technologies, Inc..


I am running the latest driver, and ATI has discontinued the Radeon 9600 Pro. Not a big deal, as the problem has only happened twice. Of course, I would rather that it not happen at all...



Using WinDBG to Cheat at MineSweeper

Ran across this rather unique notion the other day. It works!

eb poi(@$peb+0x8)+0x36fa c6 00 8a
My interpretation is that this "enters byte values" "c6 00 8a" into the address starting at offset 0x36fa from the value pointed to by offset 8 into the PEB. Whatever that ultimately does!



Internet Explorer VPC Refresh Available

Another update to the IE6 and IE7 Virtual PC images that the IE Team at Microsoft makes available is available for download in the Microsoft Download Center. The previous ones expired on 2007-12-07; these expire on 2008-04-01.



Recently, after resuming my Vista laptop from hibernation, I was greeted with a rather strange wait, followed by a blue screen of death. Analysis of the dump yielded the following:

The device driver is spinning in an infinite loop, most likely waiting for hardware to become idle. This usually indicates problem with the hardware itself or with the device driver programming the hardware incorrectly.
If the kernel debugger is connected and running when watchdog detects a timeout condition then DbgBreakPoint() will be called instead of KeBugCheckEx()and detailed message including bugcheck arguments will be printed to the
debugger. This way we can identify an offending thread, set breakpoints in it, and hit go to return to the spinning code to debug it further. Because KeBugCheckEx() is not called the .bugcheck directive will not return bugcheck
information in this case. The arguments are already printed out to the kernel debugger. You can also retrieve them from a global variable via
"dd watchdog!g_WdBugCheckData l5" (use dq on NT64).
On MP machines (OS builds <= 3790) it is possible to hit a timeout when the spinning thread is interrupted by hardware interrupt and ISR or DPC routine is running at the time of the bugcheck (this is because the timeout's work item can be delivered and handled on the second CPU and the same time). If this is the case you will have to look deeper at the offending thread's stack (e.g. using dds) to determine spinning code which caused the timeout to occur.
Arg1: 870246b8, Pointer to a stuck thread object. Do .thread then kb on it to find the hung location.
Arg2: 00000000, Pointer to a DEFERRED_WATCHDOG object.
Arg3: 00000000, Pointer to offending driver name.
Arg4: 00000000, Number of times this error occurred. If a debugger is attached, this error is not always fatal -- see DESCRIPTION below. On the blue screen, this will always equal 1.

Debugging Details:

PEB is paged out (Peb.Ldr = 7ffd800c). Type ".hh dbgerr001" for details

PEB is paged out (Peb.Ldr = 7ffd800c). Type ".hh dbgerr001" for details




PROCESS_NAME: Ati2evxx.exe


LAST_CONTROL_TRANSFER: from 89c2a825 to 81cace97

a53d7704 89c2a825 000000ea 870246b8 00000000 nt!KeBugCheckEx+0x1e
a53d7748 89c22bfa a53d7794 00000000 89c1d786 dxgkrnl!TdrTimedOperationBugcheckOnTimeout+0x2b
a53d7770 8b5785dc a53d7794 00000000 00000000 dxgkrnl!TdrTimedOperationDelay+0xc9
WARNING: Stack unwind information not available. Following frames may be wrong.
a53d77c0 8b576468 8b670040 a53d785c ffffffff atikmdag+0x255dc
a53d77dc 8b66782c 861bd000 a53d77f8 00000014 atikmdag+0x23468
a53d7838 8b670101 86a58008 8b670040 a53d785c atikmdag+0x11482c
a53d7868 8b6cd9da 8685b0e8 00000000 00000001 atikmdag+0x11d101
a53d7888 8b59f159 88340000 00000000 00000001 atikmdag+0x17a9da
a53d78a8 8b59505c 86a58000 86a61974 00000000 atikmdag+0x4c159
a53d78dc 8b5973e3 00000000 86a611e0 00000001 atikmdag+0x4205c
a53d7904 8b5b3be0 00000001 00000001 00000001 atikmdag+0x443e3
a53d7960 8b5b80ab 86a58000 00000000 00000001 atikmdag+0x60be0
a53d7980 8b58e38d 86a58000 a53d799c a53d7ba0 atikmdag+0x650ab
a53d79b8 8b554e80 86a58000 a53d7ba0 00000030 atikmdag+0x3b38d
a53d79dc 8b55a7de a53d7ba0 00000030 a53d7bd4 atikmdag+0x1e80
a53d7a00 8b55af33 0011000e 00000030 a53d7bd4 atikmdag+0x77de
a53d7a24 8b56bdeb 00000030 a53d7ba0 00000000 atikmdag+0x7f33
a53d7a54 8b56bf8a 00000000 a53d7b1c a53d7ba0 atikmdag+0x18deb
a53d7a74 89c4a7b2 8640a648 a53d7ab4 000000b8 atikmdag+0x18f8a
a53d7a94 89c4a455 a53d7ab4 a5b4b811 0012e910 dxgkrnl!DXGADAPTER::DdiEscape+0x3b
a53d7d38 81c4607a 0012e910 0012e94c 77940f34 dxgkrnl!DxgkEscape+0x4af
a53d7d38 77940f34 0012e910 0012e94c 77940f34 nt!KiFastCallEntry+0x12a
0012e94c 00000000 00000000 00000000 00000000 0x77940f34

STACK_COMMAND: .thread 0xffffffff870246b8 ; kb

89c2a825 cc int 3


SYMBOL_NAME: dxgkrnl!TdrTimedOperationBugcheckOnTimeout+2b


MODULE_NAME: dxgkrnl

IMAGE_NAME: dxgkrnl.sys



BUCKET_ID: 0xEA_IMAGE_dxgkrnl.sys

Followup: MachineOwner

Seems that the hardware was messed up, as I had to force the laptop to power down twice during subsequent boots, in order for Vista to make it to the logon prompt.



SetThreadPriority, Vista, and Autostart Locations

I ran across a post on the Vista Compatibility Team Blog entitled "SetThreadPriority from Run key" that discusses a change in Vista whereby calling SetThreadPriority from an application launched from the Startup folder and the "Run" key in the registry will not cause the thread's priority to be increased.

Wanting to verify and play around with this, I wrote a simple program that called SetThreadPriority to set the priority of the thread to THREAD_PRIORITY_HIGHEST. The program then immediately called GetThreadPriority to determine if the call to SetThreadPriority had any effect. Next, in a loop, the program then called SetThreadPriority / GetThreadPriority until either an error was encountered, or GetThreadPriority returned the expected priority. The program logged before and after each call to SetThreadPriority / GetThreadPriority the time, the action, and the either the parameters or the return value.

I set the program to be launched automatically by placing a shortcut in the "Startup" folder, and rebooted. Once the system came back up, I waited a bit and then examined the log. The first call to SetThreadPriority( GetCurrentThread(), THREAD_PRIORITY_HIGHEST ); returned TRUE. The first call to GetThreadPriority( GetCurrentThread() ); returned 0 indicating THREAD_PRIORITY_NORMAL. In other words, the call to SetThreadPriority had succeeded, but the priority of the thread remained unchanged. The calls to SetThreadPriority and GetThreadPriority in the loop were identical, and returned identical values. That is, until about 45 seconds into the program's execution, when the call to GetThreadPriority returned 2, indicating that the priority of the thread was THREAD_PRIORITY_HIGHEST. This matches what is mentioned in the "SetThreadPriority from Run key" blog entry, where it is stated that:

it is for about a minute or so after which the call to SetThreadPriority(THREAD_PRIORITY_HIGHEST ) will actually succeed in bumping up its priority level.

I repeated the same tests, using THREAD_PRIORITY_ABOVE_NORMAL in the call to SetThreadPriority, with the same results.

I also used THREAD_PRIORITY_BELOW_NORMAL in the call to SetThreadPriority as well as THREAD_PRIORITY_LOWEST; in these cases, the call indicated success and GetThreadPriority confirmed the change in priority immediately.

The next set of tests removed the call to SetThreadPriority in the loop - just the initial call to SetThreadPriority was made. The return indicated success, but the call to GetThreadPriority returned THREAD_PRIORITY_NORMAL for many minutes; as the loop was a tight loop, I terminated the process once it became apparent that there truly would be no change to the priority of the thread. This means that requests to increase the priority are not queued up or held for later processing. The call to increase priority indicates success, the priority is not changed, and unless the thread checks, it is none the wiser.

One other thing that I thought of trying was to see what happened when a thread in a process spawned by an "autostart" process called SetThreadPriority, as above. To do so, I modified the original program to accept a command-line parameter indicating that it should spawn another instance of itself. The thread in the spawned process behaved identically to the thread in the "autostart" process; this persisted 3 "levels" deep ("autostart" instance spawns instance x, which spawns instance y), which is as deep as I tried. The Vista Compatibility Team Blog entry only mentions the Startup folder and the "Run" key as being affected by this, but I wonder if other things may be affected. It is interesting (and a good thing!) that there is a mechanism in place to cause this behavior to affect processes spawned by autostart processes (otherwise, the "protection" offered by this feature is easily defeated).

As a last test, I invoked the test program manually as quickly as I could while Vista was still processing the login. The first attempt to change the priority of the thread succeeded, and the first call to GetThreadPriority confirmed the priority change. At the same time, Vista was processing the autostart instance of the program, which behaved as it had previously when started automatically. So there is not a blanket ban on priority boosting in the first minute or so - how a program is started truly affects what it can do.



Updated IE 6 and IE 7 Virtual PC Images Available

Previously, I had written about IE6 and IE7 Virtual PC images that the IE Team at Microsoft makes available. As the previous release of the VPCs has expired, a refresh release has been issued. The new release expires on 2007-12-07.



New Events and Errors Message Center on TechNet

Microsoft has recently published an "Events and Errors Message Center" on TechNet. It is a bit like EventId.net, but you can search on more fields than Event ID and Source (note that non-Microsoft products aren't included in the Microsoft offering).

The basic search allows for selection of a Microsoft product, and a search string. The advanced search adds a Version field for the selected product, an Event ID field, an Event Source field, a File name, and Language. The product list isn't yet all that comprehensive with only 18 entries (and some duplicates or products that could be further filtered by version), but I hope to see it grow to encompass more products and more versions (Vista is not listed yet, for example). It would also be cool if the capability to annotate specific events was made available to users, much like can be done at EventId.net. Another idea that would be nice to see is that as products are developed / maintained / updated, part of the process would involve documenting the events and their meaning, in the same database that the Events and Errors Message Center interfaces with. Perhaps QA and other groups could even add their own annotations to specific events - what caused the problem, how the problem was resolved, etc.

Hopefully, the Events and Errors Message Center keeps improving. Along similar lines, it would be nice to see the DLL Help Database get some attention - several times I have hoped to see Vista files appear but have been disappointed.



Multiple Versions of IE on the Same System?

If you need to run IE 6 and IE 7 on the same system for some reason (testing?), what are you supposed to do?

One solution is the "Multiple IE installer" from TredoSoft. I haven't tried it, but it is certainly a novel approach. Looks like you can run IE3, IE4.01, IE5, IE5.5, and IE6 with it.

TredoSoft also provides instructions and a utility that allows you to run IE7 side-by-side with IE6. Again, I haven't tried it, but it is good to know that it is available.

When I need to test different versions of the browser, I have as of late been relying on VPC images of Windows XP with IE6 and IE7 that have been provided by the IE team. The VPC images run with the free Virtual PC 2007. They expire on 2007-08-17, for various reasons, but the IE team has in the past provided a "refresh" for the IE6 VPC image when it last expired, and the expectation is that this will continue. The expiration-refresh cycle allows for control over how "old" an image is allowed to get - refreshes have the latest security patches installed so people aren't left running ancient and (overly) vulnerable VPC images.

The images won't pass WGA for obvious reasons but this hasn't affected my ability to test with them.

I tend to customize my IE settings pretty severely, so it is nice to be able to test with "stock" / "virgin" installs of IE. One might argue that using the VPC images doesn't necessarily mean one has two different versions of IE on the same system - that the introduction of a VM means another "system" is involved. While this is true, the VM solution does prevent one from needing to have multiple physical machines around just for the sake of testing. And, while the TredoSoft solution may work, I can't help but think that the possbility exists for the solution itself to be a potential cause of problems.



An Incoming Request for Troubleshooting Advice

I recently received the following via email. I responded, but never heard back. Hopefully the information will be of use to others...

Hi, Found your name on the web when I was looking at some stuff on slashdot and I thought maybe you might have a moment to point me in the right direction.
I have a custom app which I have zero access to in terms of the developer. When the app was run on new hardware, a few users would periodically get a bizarre error, but never on their old hardware. Main difference seems to be core 2 duo on the new laptops.
Anyway, this error causes a total reset of the internal windows database system used by the app and then there's an error in msvcr80.dll. Is there a way I can tell from the data below what routine went bad in that module? I could at least then tell the guy who guards the gates to tell the developer a little bit more. At this point the "guard" is convinced it is just because the dll is not the latest version. I think it is probably not that simple.

[app].exe signature
Appname [app].exe App. Ver.
Mod Name: msvcr80.dll
Mod Ver: 8.0.50215.44
Offsett: 000161fd

Error report contents
Exception info:
Code 0x0000005
Flag 0x00000000
Record 0x000000000000000
Address: 0x000000007C3861fd

I responded with the following:

It does appear that the problem may lie in the version of MSVCR80.DLL you referenced. At least, getting this to an appropriate version seems like a very reasonable first step towards troubleshooting.

According to a dude on the VC++ compiler team (Jonathan Caves @ http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=164465&SiteID=1&PageID=1 ), version 8.0.50215.44 of MSVCR80.DLL "is not an officially released version"; he links to this site (http://www.dll-files.com/dllindex/dll-files.shtml?msvcr80 ), which has a statement by another member of the VC++ team (Martyn Lovell):

"a) This is the beta 2 version of msvcr80.dll. Don\'t install it. b) This file should never be installed in system32 (except on Win9x and Windows 2000). c) The .NET framework already installs the copy correctly (in WinSxS). d) The correct source for an msvcr80 binary is from your application provider, not from this site. Martyn"

The MS DLLHelp Database entries for MSVCR80.DLL (http://support.microsoft.com/dllhelp/?dlltype=file&l=55&alpha=msvcr80.dll&S=1 ) indicate that you can get the DLL from "MS SQL 2005 Server Enterprise" or "Microsoft Visual Studio 2005 Professional". Martyn from Microsoft indicates that you should get this DLL from the application provider. He also stated that you can get it as part of the .NET Framework redistributable (version 2.0). It also appears that you can get it as a part of a package from the "Microsoft Visual C++ 2005 SP1 Redistributable Package (x86)" at http://www.microsoft.com/downloads/details.aspx?FamilyID=200B2FD9-AE1A-4A14-984D-389C36F85647&displaylang=en.

FWIW, 0x0000005 is not a valid exception code, though I suspect you meant 0xc0000005, which is the code for an access violation. You're reporting a crash at address 0x7C3861fd. With no other info it is difficult to determine what is loaded at that address (especially since the "shipping" / non-beta versions of MSVCR80.DLL are based at 0x78130000 with a virtual size of 0x9b000), but it seems likely that the beta version of MSVCR80.dll has a different base address (0x7C370000) / size which could/should include the crash address (or, the DLL could have been rebased by the loader).

I suspect that a "Dr. Watson" or "Windows Error Reporting" dump file may have been created from the application crash(es) - these are files with a .dmp extension, and they contain information useful for debugging, or getting more information about, these kinds of problems.

Hope this helps you!



Deep Understanding?

I was reading a recent blog entry by Eric Sink, and he made some comments that I thought I would take the time to share. Nothing profound on my end, mind you... ;)

In "In defense of Petzold's WPF book", the founder of SourceGear writes about a recent trend in books and tools for developers that focus on "quick results", and the trade off that goes along with this trend - loss of "deep understanding". I too have noticed this in working with various developers. While it is admirable that one is able to produce results quickly and ship a product in record time, or add some features in the blink of an eye, it seems that much of the productivity is lost when problems crop up - problems that are at least partially a lack of - you guessed it - deep understanding. As I'm one who likes to have Deep Understanding, I'm probably biased. That said, I can fully appreciate Eric's statement that "When I hire a software developer, I look for deep understanding". I think that it is important to understand how something works so you are able to use it as effectively as possible. Perhaps this is why I am inclined to consider writing software an art, as opposed to a means to an end. Sure, it's both, but how you treat it can have a significant impact on the quality of the end product. At least, that's been my experience.

To me, Deep Understanding implies passion - if one takes the time to understand something fully, to explore all venues of a topic or a technology, they truly care about what it is they're doing and are excited by it. And I can't see doing something as a profession and not being passionate about it.



Fix that Addresses Issues with SVCHOST.EXE and Windows Update / Microsoft Update

Just received the following that is related to the SVCHOST issues that I've written about in the past...

MS has released "Microsoft Security Advisory (927891) - Fix for Windows Installer (MSI)" that's not really a direct security concern, but actually addresses concerns that might prevent people from getting critical security or other updates.

As previously mentioned, it involves MS KB 927891 - "You receive an access violation error and the system may appear to become unresponsive when you try to install an update from Windows Update or from Microsoft Update", and the current revision of the article (8.0) states "This fix is one component of a two-part fix that includes a Windows Update client software update. These updates will be deployed automatically using Windows Update in May 2007 and June 2007."

Again, this update is one of two that need to be applied to fully address the issue. The other update is version 3.0 of the Windows Update Client Software, available from MS KB 932494, "When you use Automatic Updates to scan for updates or to apply updates to applications that use Windows Installer, you experience issues that involve the Svchost.exe process".

One can also hope that this will help address the 0x8ddd0009 problems that MANY have been experiencing...



Reading List

Initially, I was going to start off by saying that I've been reading more books and less content online, but that wouldn't be accurate. I've actually picked up more online reading, but I'm also more frequently finding myself with a good book in hand before bed. Currently, I'm finding my way through Eldad Eilam's Reversing: Secrets of Reverse Engineering. It's a good read so far and I hope to learn a lot.

I've ALWAYS got Russinovich & Solomon's Windows Internals open, and I've been through it cover to cover a few times as well. There's just so much good stuff there, and I've got a memory like a goldfish... =8->

Recently, I also read Hoglund and Butler's Rootkits: Subverting the Windows Kernel for the second time. The chapter on DKOM is probably my favorite.

I just finished Raymond Chen's The Old New Thing: Practical Development Throughout the Evolution of Windows. The anecdotes range from light to heavy, and the lessons to be learned are well presented. I like Raymond's writing style and it is good to hear his perspectives on why things are the way they are. For whatever reason I hadn't picked up Raymond's blog on MSDN, "The Old New Thing", though I was well aware of it. Probably had something to do with the volume; two posts per day seems quite high to me. After reading the book, though, I'm subscribed! Sigh. I'm not a fan of the "bonus online chapters" concept, though I am sure that I'm going to go and read them sometime. Kind of detracts from the whole "book" experience if the whole book isn't... well, whole, I suppose.

A later post will detail some of the stuff I'm subscribed to online.



Hey! Where'd my E: Drive Go, Vista?

I use ReadyBoost in Vista. One day, I went to power up my laptop and when Vista resumed from hibernation I noticed that the light on the USB Flash-memory Device (UFD) was not on. Finding this a bit odd, I jumped into Windows Explorer, double-clicked on the E drive (the letter assigned to the UFD), and was presented with an interesting dialog:

Title: Item Not Found
Text: Could not find this item
This is no longer located in <%3 NULL:OpText>.
Verify the item's location and try again.
Removable Disk

Try Again Cancel

So I looked at the back of my laptop to verify the item's location, and deciding that the item was still there I clicked "Try Again". (I know, I know - not really what the person who wrote the message for the dialog intended...) I don't recall if the dialog dismissed and another instance reappeared, or if it was just that nothing happened. Either way, I wasn't getting anywhere. I unplugged the UFD, and plugged it back in again and things were fine.

A few days later, the same thing happened. I suspect I'll be dealing with this for a while.

Procedurally, I hate to think that I'm going to have to eject the device prior to hibernating, and then plug the device in again when Vista resumes - that's too tedious for my tastes. ReadyBoost would have to have a significant impact on performance for me to go through that rigamarole, and at this point I'm just not convinced that's the case.

Anyone else coping with %3 NULL:OpText? How are you dealing with it?



Setting the Priority of a Service Process via Script

Previously (here and here), I've written about isolating shared services so that they run in their own process, with a specific focus on the Windows Update Automatic Updates Service (wuauserv) that typically runs in the NETSVCS SVCHOST.EXE instance. One thing that can be done once this is accomplished is to lower the priority of the process so that when the service winds up consuming 100% of the CPU, the system doesn't become unresponsive.

Since we're dealing with a service, setting the priority of such a SVCHOST.EXE process can become problematic - the service may already be running, or, because it is a service, it is not started as non-service processes are, so one is not able to use START / [LOW NORMAL HIGH REALTIME ABOVENORMAL BELOWNORMAL] to impose a priority when the process starts. One can use a utility like Task Manager or Process Explorer to set the priority of a process on an ad hoc basis, but when the service restarts or the system reboots one has to remember to set the priority again.

Though not an ideal solution the following scripts (VBS using WMI, and PowerShell) can be used to set the priority of the SVCHOST.EXE instance hosting the isolated Windows Update Automatic Updates Service service to "below normal". Note that no check is done to ensure that the SVCHOST.EXE instance is only hosting one service - if wuauserv is found to be a service inside of the process, the priority is adjusted. Note also that no error handling is implemented.

I'll try to format the code so it looks nice, but I fear I will be limited...

Here's the code for the VBS / WMI script:

Const BELOW_NORMAL = 16384
Set objWMIService = GetObject("winmgmts:\\.\root\CIMV2")
Set colServices = objWMIService.ExecQuery( _
"SELECT * FROM Win32_Service where name='wuauserv'")
For Each oService in colServices
Set colProcesses = objWMIService.ExecQuery( _
"SELECT * FROM Win32_Process where ProcessId=" & oService.ProcessId )
For Each oProcess in colProcesses

Here's the code for the PowerShell script:

(gps -id (get-wmiobject win32_service where {$_.name -eq "wuauserv"}).ProcessId).PriorityClass="BelowNormal"

The different values for the priority parameter of the SetPriority method of the Win32_Process WMI class can be found in the documentation for the SetPriority method.

The different values for the PriorityClass in the PowerShell script are "Normal", "Idle", "High", "RealTime", "BelowNormal", or "AboveNormal". Or, to get a list of the available options, one can use the following PowerShell command:


Once the script is in place and working, one can cause it to be invoked at will, or via scheduled task at specific times, or after logon, or any other way that one can get something to happen when Windows boots or a user logs on.



User-Mode Debugging Internals

Alex Ionescu has posted some publications at his blog. I've only had an opportunity to go through the three-part series on Windows XP / 2003 User-Mode Debugging Internals, but I found them to be quite interesting and I hope to go through the rest of the publications which cover topics like Process Internals, VB File Format, NTFS ADS, and Subverting Windows 2003 SP1 Kernel Integrity Protection.

One thing (probably trivial) that I am curoius about in the User-Mode Debugging Internals papers is the analysis of NtDebugActiveProcess. Alex comments in his analysis in part 3 "Don't allow debugging the initial system process". The check for the initial system process is made, and STATUS_ACCESS_DENIED is returned if indeed it is the initial system process that is the subject of the debug attempt. I am curious as to why, prior to returning in this case, the process is not dereferenced (ObDereferenceObject(Process);)? Is it simply the nature of the system process that this is not required, though perhaps it might be considered good practice to call ObDereferenceObject(Process) in this case? Or is there some other reason?

Does anyone have any thoughts on the above?



The Joy of Tagging (or Labeling)

OK... I admit it. I'm jumping on the whole "tagging" thing (or as Blogger likes to call it, "Labels") a bit late. Shortly after converting to the "new" Blogger, I made a rather unconscious decision to ignore the field that allows for tagging new items. So there are many posts that were made with the new blogger that don't have tags. I plan on tagging a few at a time, but with 140+ posts, this could take a bit of time.



Part 2: Background - What's using my CPU?

Previously (Part 1: Introduction - What's using my CPU?), I kicked off what I expect to be a multi-part series on determining what is causing excessive CPU consumption, outside of the normal "which process has the highest value in the CPU column in Task Manager".

Before I get into things, a little bit of background may prove useful or mildly entertaining. Over on "Sysinternals Forums", there were recently two similar problems that both involved excessive CPU utilization that was not attributable to a specific process. I became involved in both problems and attempted to use similar techniques to get additional information with the hopes of ultimately being able to pinpoint the problem. What may make this mildly entertaining is that in both cases, there was limited or no success in detetmining the cause of or solution to the problem. In the end, one problem was resolved by disabling the floppy disk controller, and the other problem appears to be as of yet unresolved. (In the latter case, the poster did admit that the system was experiencing hardware problems - the chipset fan was dying and there were diagnostic beep codes during / after POST. These hardware problems could be related to the problem.) Despite the lack of success in determining the cause of the problems I do feel that I learned a bit about this type of problem and gained some insight into the use of some tools that can come in handy in this situation.

In the two cases, the problem consisted of the CPU spending a lot of time servicing interrupts and deferred procedure calls (DPCs). What are interrupts and DPCs? "Windows Internals, Chapter 3 - System Mechanisms" says:

Interrupts ... are operating system conditions that divert the processor to code outside the normal flow of control. An interrupt is an asynchronous event (one that can occur at any time) that is unrelated to what the processor is executing. Interrupts are generated primarily by I/O devices, processor clocks, or timers.
A deferred procedure call (DPC) is a function that performs a system task—a task that is less time-critical than the current one. The functions are called deferred because they might not execute immediately.
It is interesting to note that one may have a problem with excessive CPU use but may not be able determine it by using Windows' Task Manager. This is because for whatever reason, Task Manager adds time the CPU spends servicing interrupts and DPCs to the "System Idle Process". Microsoft's / Sysinternals' Process Explorer includes separate "artificial" processes for interrupts and DPCs so that one can see how much time the CPU spends dealing with each. Per Process Explorer's help file, "high CPU consumption by these activities can indicate a hardware problem or device driver bug".

Another thing that could be consuming CPU is the SYSTEM process. The process of determining what system thread is consuming the CPU is similar to determining what thread in a user-mode process is utilizing the CPU. However, excessive CPU utilization by the SYSTEM process might be a little more serious as it is an indication that some driver is possibly running rampant.

Next time, I plan to introduce some tools that can be useful in exploring DPC and interrupt activity on a system, as well as discussing how to determine what driver might be inolved with excessive CPU utilization in the SYSTEM process.



Visual Studio's Tacit Endorsement of MadLibs

Formatting strings can be a tricky job...

I'm sure there's a logical explanation, but still... It really {0} me off when {1} Visual Studio 2005 SP1 decided to throw this {2} my way.



Patch that Might Help with 0x8ddd0009 as well as high SVCHOST.EXE CPU Utilization?

In the past, I've written about both high CPU utilization by SVCHOST.EXE as well as the 0x8ddd0009 Windows Update / Microsoft Update error, so I thought I would mention this...

MS KB 932494 (When you use Automatic Updates to scan for updates or to apply updates to applications that use Windows Installer, you experience issues that involve the Svchost.exe process) references problems that are addressed by MS KB 916089 (FIX: When you run Windows Update to scan for updates that use Windows Installer, including Office updates, CPU utilization may reach 100 percent for prolonged periods) and MS KB 927891 (You receive an access violation when you try to install an update from Windows Update after you apply hotfix package 916089). However, even after applying the patch associated with 927891 (which replaces the patch associated with 916089), 932494 indicates that the following problems remain:

1) Certain 100 percent CPU issues are still present when you use the Svchost.exe process.
2) An access violation may occur in the Svchost.exe process.
I (as well as others) have speculated in the past that 916089 (and its succedent patches) can also help with the 0x8ddd0009 error message that one might receive from Windows Update / Microsoft Update.



Part 1: Introduction - What's using my CPU?

Recently, I have been involved in attempting to diagnose problems with excessive CPU utilization. Often times, this type of thing is relatively easy to identify - at least as far as pointing the finger at the thing that is consuming CPU cycles. Task Manager can be used for this - simply sort the "CPU" column in descending order and note the process that is at the top of the list. One can use a similar technique with Process Explorer.

In the past (here and here), I've given examples that demonstrate various techniques that can be used to try to determine what a process is doing when it is consuming so much CPU. Sometimes, you can do something about it - if you have the debugging symbols, perhaps there is something in the stack of the thread or threads in the process that is consuming the CPU that will lead you to some setting, feature, or configuration piece that can be manipulated so as to avoid the problem. Or perhaps just knowing the module name is enough information to identify the problem software - a recently installed add-in / plug-in, or a new utility, perhaps. Sometimes you are forced to work around the problem - you don't have any control over it and don't want to stop using the program, or have no choice but to keep using the program.

But what happens when the excessive CPU utilization is not attributable to a "standard" process? In the coming series of articles, I hope to explore some of the things that can be done to diagnose and troubleshoot this type of scenario. Stay tuned...



Seriously! The dog ate it!

I'll make this one brief...

Sorry for the lack of updates lately - I'm working on a number of other things and haven't found a lot of time to write. I have something kind of fun planned that may be a three or four parter but it will probably be a bit slow in coming. It will also be a learning experience for me, so that should be cool. I will post things as I finish them, but there may be revisions to the content when the later parts are posted. I hope to have the first one (introduction) up by the end of the coming weekend.

I appreciate your patience, understanding, and continued readership. Thank you.



Ideas For Features / Enhancements to Sysinternals' Process Monitor

I've written about Sysinternals' Process Monitor utility before:

I have had a few months now to work with Process Monitor, and it certainly is amazing. The filtering capabilities are great, and the fact that the filters are not destructive makes slicing and dicing the data many ways quite simple. The ability to capture all of the data that the utility can capture makes it quite powerful, and the ability to get stack traces for each event is extremely useful.

Going off the observation that Process Monitor is currently at version 1.01, and the assumption that the utility will see further development, I have hopes that the following relatively small ideas will be taken into consideration for future releases, and that further discussion and conversation is sparked.

1) Allow for the use of CTRL+C to copy selected data to the clipboard. Some data can be copied in this fashion, but from my experience not much. In many cases, one can right-click and choose "Copy" from the context menu, but that's inconvenient. For example, on Event properties, on the Event tab for a Profiling Interrupt, I can select the User and Kernel times and press CTRL+C, and the data will go to the clipboard. However, if I select the date, the result, or the sequence #, the only way to copy the selection is by using the mouse.

2) Along the same lines, it would be useful to have a "Copy details to clipboard" button on each tab of the Event properties - it would simply grab all of the displayed data and copy it to the clipboard with the click of a button. Or the use of a hotkey accelerator. This would have come in handy on the Stack tab, but there I was at least able to save the data to a CSV file and work with it in that fashion.

3) I would like to be able to sort by each of the columns displayed on Process tab of the Event's properties, in the "DLLs" section. It might also be nice to toggle display of path (this would affect sort as well), or perhaps add another column for just the module name.

4) In Options --> "History Depth", the edit part of the spin control could be a bit wider - there is plenty of room on the dialog...

5) It might be inferred that I prefer to avoid using the mouse. As such, it pains me when there are no hotkeys on a dialog box. I find myself constantly wanting to "ALT+A" to add a filter on the "Process Monitor Filter" dialog, or "ALT+R" to remove, or whatever. Other dialogs, such as Configure Symbols, Select Columns, Show Unique Values, etc, could also benefit from hotkeys. I do love the fact that so many things in the main UI are accessible with the CTRL key - CTRL+L for the filter, CTRL+J for jump to, etc.

For further exploration / discussion:
-- I realize it can be difficult to determine what to do for an implementation of "Jump to" for certain classes - what would one "jump to" for a profiling interrupt, for example? For Process and Thread activity, one might wish to "Jump to" Dependency Walker for "Process Create" and "Load Image" operations. For "Thread Create" and "Thread Exit" events, however, a relevant action is not clear. Does anyone have any thoughts or ideas?

[Note: The spirit of the above was posted by me to the Process Monitor forum on Sysinternals' Forums at "Process Monitor - Feature Requests". Reprinting here with my own permission. Apologies if you've seen this before. I also apologize for my client's rude behavior.]



More on Isolating Shared Services in Windows

Previously (Troubleshooting Performance Issues with Automatic Updates / How to Isolate A Shared Service Hosted by SVCHOST.EXE) I had detailed some steps that one could follow to isolate a shared service hosted by SVCHOST.EXE, in the context of the Windows Update Automatic Updates service. There are a couple of other ways to isolate a shared service, with different implications.

The intention / desire of a service to be "shared" or not is typically indicated when the service is created - the fifth parameter to CreateService is dwServiceType, which for our interests can be, among other things, SERVICE_WIN32_OWN_PROCESS (0x10) or SERVICE_WIN32_SHARE_PROCESS (0x20). Once a service has been created, the dwServiceType setting can be changed by calling ChangeServiceConfig with the appropriate parameters.

Windows XP and Server 2003 ship with a utility program called "SC.exe". (I seem to recall using SC.EXE on NT 4.0, after it was included in some resource kit, but I could be mistaken. The DLL Help Database listing for SC.EXE puts the earliest version as having shipped with Visual Studio .NET 2002.) SC presumably stands for "Service Controller". At any rate, one can use SC to change the configuration of a service, including the "service type" - "shared" or "own". So, using wuauserv - the Windows Update Automatic Updates service - as a guinea pig, one could execute from a CMD prompt:

sc config wuauserv type= own
to cause the service to run in its own process. (The service will need to be restarted for this to happen.) To change it back to a shared service, use the following command:

sc config wuauserv type= share
Note that in both commands, the space after the '=' character is critical. For this change (back to "shared") to take effect, the system will likely need to be rebooted as the "original" SVCHOST group that this service was a part of is already running. This is also the case when un-doing the configuration to make the service run in its own SVCHOST group.

The SC commands above are actually just manipulating the registry - you could make the change directly to achieve the same outcome. The setting in question (again, using wuauserv as a guinea pig) is located at [HKEY_LOCAL_MACHINE\SYSTEM\ CurrentControlSet\Services\wuauserv] - the "Type" REG_DWORD value. The type corresponds to the aforementioned dwServiceType, which can be (for our purposes here) SERVICE_WIN32_OWN_PROCESS (0x10) or SERVICE_WIN32_SHARE_PROCESS (0x20). To isolate the service, set type to 0x10, and to undo the change, set type back to 0x20 (keeping in mind the restart considerations previously mentioned).

Another thing to keep in mind is the consideration that perhaps a service is not meant to be isolated; those concerns are discussed a wee bit more here.

One subtle difference in techniques is that if you isolate a service into its own SVCHOST group, the command line for the svchost.exe process changes to reflect the group name you speficied when setting up the configuration. However, if you use the SC.EXE / "Type" registry value change technique, the command line for the svchost.exe process remains the same - in the case of wuauserv, the "netsvcs" group is still specified on the command line even though the SVCHOST instance will only be hosting one service.



Unspecified Potential Security Risk! Take 2

Some additional questions have been raised here and elsewhere about what precisely causes the "Unspecified Potential Security Risk" dialog - the one from Internet Explorer that looks like:

Internet Explorer

This page has an unspecified potential security risk.
Would you like to continue?

The dialog is displayed when the setting "Launching applications and unsafe files" is set to "Prompt" for the security zone that Windows / Internet Explorer believes itself to be operating in.

Changing the setting to "Enable" for the specific zone eliminates the dialog, while changing the setting to "Disable" produces a "Security Alert" dialog stating that "Your current security settings do not allow this action."



Troubleshooting Performance Issues with Automatic Updates

How to Isolate A Shared Service Hosted by SVCHOST.EXE

There are a number of articles in the Microsoft Knowledge Base about performance issues with Windows Update / Microsoft Update, as well as other problems related to the scanning mechanisms used by automatic-type update services. There are some fixes, and fixes for some fixes, but no "ultimate" solution. Stepping back a bit, how can you even determine if Automatic Updates is causing performance issues on your system?

The Automatic Updates service is not a stand-alone process, so it is not sufficient to simply look for which process is consuming the most CPU time or the most memory. Rather, Automatic Updates is integrated into the "netsvcs" SVCHOST service hosting instance. On the systems that I have examined, this instance hosts over 20 services - 25 services on the system I am using to write this. How can you see what services are running inside of a process? One way is to use "tasklist /svc", and examine the "Services" column. Another way is to use Process Explorer - simply hover the mouse pointer over a process and any services that are contained in the process are listed in a tooltip. Or, view the Process' "Properties" page and examine the "Services" tab for more details.

This sharing of services in one process isn't a bad thing - Windows has been doing this for some time. There are times when it makes sense, and times when it doesn't. Basically, processes are expensive and the more you have the more resources they consume. If you are able to share services in the address space of a process, you are conserving resources. But if the services have different security needs, for example, then you should probably split them into two separate processes to "isolate" the functionality that requires greater privileges.

Back to the task at hand... The fact that services can share a process is nice, but this really gets in the way of troubleshooting a service you suspect may be causing problems. So it can be useful to extract a shared service and make it run in its own process. With services hosted by SVCHOST, the configuration is controlled in the registry. Microsoft doesn't publicly document the interfaces for SVCHOST.EXE as it doesn't want people writing services and making them run in the same address space of processes that host Windows built-in services - if the service is poorly written it can cause SVCHOST.EXE to crash, and subsequently kill all of the other services running in that instance of SVCHOST.EXE. That doesn't mean you can't manipulate the built-in Windows services to use a configuration you desire, though...

The SVCHOST services are controlled by registry settings in [HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\SvcHost]. (Standard warnings about editing the registry apply.) Each REG_MULTI_SZ in this key represents a SVCHOST group containing a list of one or more services to run in an instance of SVCHOST.EXE. So if one wishes to isolate the Automatic Updates service, one needs to find which group it is in. The "name" of the Automatic Updates service is "wuauserv" - Windows Update Automatic Updates service. This service resides in the "netsvcs" group. So, since the desire is to create a new SVCHOST instance to run the service in, remove wuauserv from the list in the netsvcs value. Then, create a new REG_MULTI_SZ value and give it an appropriate name, such as AutomaticUpdates. Add wuauserv to this value.

Next, navigate to [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet \Services\wuauserv] and change the ImagePath (which specifies the program and arguments the Service Control Manager is to use to invoke the service) from:
%systemroot%\system32\svchost.exe -k netsvcs
%systemroot%\system32\svchost.exe -k AutomaticUpdates

That's it. Stop and restart the Automatic Updates service (net stop wuauserv / net start wuauserv) and you should see a new instance of SVCHOST.EXE that contains only the Automatic Updates service. Now you can monitor the performance of this process, drop its priority (Task Manager or Process Explorer), etc.

The same technique can be applied to isolate other SVCHOST hosted services as well. However, some caution and investigation should be applied on a case-by-case basis- it should be noted that some services may have some dependence on residing in the same address space as another service. This may or may not be intentional; if intentional I suspect that it probably has to have some relation to performance. If not intentional, it is likely a bug.



Vista Features You'll Never See

A quick one this week...

Ran into an interesting posting at shell: revealed about features of Longhorn that didn't make the cut. The post paints the picture that after the Longhorn reset (described here), a lot of features were yanked, and then goes on to describe some rather humorous things that just make you think... "What if?"



Fix Available for Performance Problems with Internet Explorer 7's Phishing Filter

I don't know how I missed it (well, I suspect it's related to quirks in the method(s) in which Microsoft makes notifications of new KB articles available), but it seems that Microsoft has released an update to Internet Explorer 7 to address the performance issues with the Phishing Filter that I had previously (here and here) encountered.

The KB article (The computer may respond very slowly as the Phishing Filter evaluates Web page contents in Internet Explorer 7) is dated December 12, 2006, and contains links to download pages for various versions of Windows that Internet Explorer 7 can run on. Your copy of Windows has to pass the Windows Genuine Advantage (WGA) check before Microsoft will allow you to download the fix. Once downloaded, installation is straightforward; depending on what programs you have open at the time, you may be required to reboot after the installation finishes.

It is interesting to note that the Cause section (below) explains exactly the conditions I was operating in, and the behavior I observed.


This problem occurs when one or more of the following conditions are true:
• The Web page contains many frames.
• You browse many frames in a short time.
Internet Explorer 7 evaluates the whole Web page when you browse a frame. Therefore, CPU usage may be very high.

Additionally, the workaround (at the bottom of the article) is to disable the Phishing Filter on the "Advanced" tab of the "Internet Options".

Prior to installing the update, I set the Phishing Filter to "Turn off automatic website checking" ("Advanced" tab of "Internet Options") and made sure it was "Enabled" for the Internet Explorer Security Zone I was working in. I then verified that I was able to recreate the behavior I witnessed in The Case of the Sluggish Internet Explorer 7.

I installed the update and needed to reboot (I forgot I had SharpReader open, and it had loaded MSHTML.DLL). Upon reboot I attempted to cause the CPU to spike by doing precisely the things that had caused the problem in the past. I didn't have any luck in doing so. In fact, none of the threads in the iexplore.exe process were consuming an inordinate amount of CPU. It would appear that the fix involved, to some extent at least, changing the technique Internet Explorer uses to queue up requests to have something evaluated by the Phishing Filter (previously, I had hypothesized that Internet Explorer 7 was using the ThreadPool API and was creating a new thread for each request). Based on my explorations so far, the fix takes care of the problem. Kudos to Microsoft for recognizing the problem and taking appropriate steps to address it.

The fix updates 2 Windows / Internet Explorer program files in %windir%\system32 - ieapfltr.dll (the "Microsoft Phishing Filter") and mshtml.dll (the "Microsoft (R) HTML Viewer"). I didn't take the time to save off copies of the previous versions of these DLLs to try to compare differences (perhaps an exercise for another day). However, the new MSHTML.DLL has a date of 2006-11-09, and the new IEAPFLTR.DLL has a date of 2006-11-08. Internet Explorer 7.0 was released on 2006-10-18. So it appears that Microsoft knew of problems with the Phishing Filter performance prior to launch; obviously this wasn't a showstopper issue. It also would appear to have taken just over 1 month for the fix to make its way through the testing / release process. I guess that's not too bad, considering that some high-priority / critical security updates take at least that long. On the other hand, this is a new feature so it didn't have the legacy behind it that some of the security updates have to contend with. I guess I should just be happy to have a fix... ;)



Unspecified Potential Security Risk!

Oh, my. Two weeks in a row with ambiguous security-related messages from a web browser.

This week, we have the following:

Internet Explorer

This page has an unspecified potential security risk.
Would you like to continue?

If it wasn't for the fact that I wasn't browsing the web - I was trying to open a ZIP file on a network share - I probably would have said "No". But since I really needed to get into the ZIP file, I decided to take my unspecified potential chances. I think I'm OK.



Puritanical Security? And a Few Other Notes on HTML Help

I was going through some CHM help files the other day and I wound up copying one of the links to the clipboard and tossing it into Maxthon. I wasn't even really aware of what I was doing (just plodding along mindlessly) so I was rather surprised when I was presented with the following dialog:

Security Warning !

Using MK: protocol in browser may cause puritanical security problems.
Do you really want to enable this protocol during this session?

Of course, I had no desire to cause puritanical security problems, so I went with the default "No". The URL I had copied was in fact a "Microsoft Infotech" protocol link in the form of:

The InfoTech protocol has changed several times over the last few years to reduce security vulnerabilities in HTML help. See MS05-026: A vulnerability in HTML Help could allow remote code execution and MS04-023: Vulnerability in HTML Help could allow code execution for more information.

Another issue that is seen rather frequently is the inability to open CHM / HTML Help files from a network path (UNC path or mapped drive). The article "You cannot open remote content by using the InfoTech protocol after you install security update 896358, security update 840315, or Windows Server 2003 Service Pack 1" discusses various registry settings that can be manipulated to allow the display of content in CHM files in this scenario.



mtpstoolkit RatingBehavior related to AJAX?

Unfortunately, I forgot what website I was visiting when I encountered the following dialog:

Assertion Failed: Unrecognized tag mtpstoolkit:RatingBehavior

Break into debugger?

I have no concrete idea what the dialog is related to, but I suspect it has to do with some AJAX framework. Interestingly, googling mtpstoolkit (groups search here) currently proves to be rather fruitless. The stack of the thread "owning" the dialog (with a large number of calls in jscript.dll) seems to support the AJAX theory:


Anyone know what precisely mtpstoolkit is a part of?