Author: Chuck

KB5009624 Causing Some Domain Controllers to Reboot

Update: This issue has been resolved with new updates released by Microsoft.

Today, I had the pleasure of troubleshooting why a domain controller was continually, and randomly, rebooting itself. Though, rebooting is a light term, when the reality was the server was entering a fault state (aka, crashing) and rebooting to recover. Inspection of the event logs revealed Event ID 1074, which stated:

The process wininit.exe has initiated the restart of computer on behalf of user32 for the following reason: No title for this reason could be found

Reason Code: 0x50006

Shutdown Type: restart

Comment: The system process ‘C:\Windows\system32\lsass.exe’ terminated unexpectedly with status code [blank]. The system will now shut down and restart.

One or two Google searches later, refined to display only from the past week, I found a post where someone figured out that Microsoft Window’s Server 2012 R2 Security Update KB5009624 was the cause of the reboots, further detailed here.

Microsoft did release an emergency out of band update (KB5010794) though it is listed as an Optional update. So if you are not in the habit of installing those optional updates, now is a good time to start.

However, if you don’t want to install that optional update (for whatever reason), this fix is simple enough: either go to Windows Update -> Installed Updates -> and select it and click Uninstall; or, go to Control Panel -> Programs and Features -> View Installed Updates -> and select it and click Uninstall. (Note: a reboot will be required).

Using an Intel NUC as a Server

Some organizations have servers that are old or perhaps they just have a single server. This can pose a problem if they utilize an Active Directory environment. I have seen an organization have only one Domain Controller, which doubled as a file server, which was infected by Ransomware. Luckily, we were able to restore the server from a recent backup. However, not all organizations are lucky. Had there been a catastrophic hardware failure or no good backup, they would have had to rebuild their environment. Another alternative issue is organizations that have several branch offices and thus need a Domain Controller at every location.

Unfortunately, not all organizations can afford another physical server. One solution is to remove them entirely from Active Directory and place them into a workgroup. However, not all organizations want that. As such, another cost-effective solution needed to be found.

Enter the Intel NUC. They are fast, cheap, customizable, but really, they are cheap. Also, depending on the one you buy, they are compatible with Windows Server 2019. They are not ideal as a standalone domain controller, let me be clear on that (unless you operate a home lab). However, they are suitable for branch offices that are interconnected via VPN tunnels, and organizations that have a physical server in place but need redundancy.

Not all Intel NUCs are compatible with Windows Server and those that are pose their own challenges. Through testing, the best and most compatible NUC we have found is the NUC7i5DNHE (we prefer utilizing the Tall version).  The NUC7i5DNHE can be customized with NVMe SSD or a traditional 2.5” SSD or HDD, with up to 32GB DDR4 Memory. Our usual build-out for clients is a 250 to a 1 TB 2.5” SSD and between 8 GB and 16 GB of memory. Regardless of the two options for SSD and memory, these things are fast. With Windows Server 2019 standard installed, we see full patch times, including boot times, to be within 5 minutes. Straight boot times usually fall inside 45 seconds.

Sounds good, right? So what issues have we found? Mainly driver support issues. With the NUC7i5DNHE, two drivers will not work out of the box (nor have we been successful in getting them to work, not that we put much effort into it anyway). The built in WiFi and Bluetooth drivers will not work. However, everything else installs perfectly with Windows Server 2019. We did have Ethernet issues working right with Windows Server 2012 R2, but there are drivers out there if you can find them to manually install via a USB drive. Windows Server 2016 on the other hand was a bit more complicated, so I would avoid that OS entirely if possible. Another note: BIOS updates through windows will not work, so it is best done to update the BIOS through the BIOS itself (which is fairly easy).

Keep in mind, we are using these only as a redundancy domain controller for smaller organizations or organizations with several branches. If one of these units die, it’s trivial and cost effective to order a replacement and have the redundancy restored in a day. Want an extra step of redundancy? Utilize an external HDD to do bare metal backups via Windows Server Backup (or another backup solution of your choice).

We have deployed nearly a dozen of these over the last year in different organizations and each seem to meet the needs of the clients nicely.

Dislike this idea or have questions? Let me know in the comments!

Polycom VVX 411 and Skype for Business

We recently purchased several Polycom VVX 411 phones to use with Skype for Business Online. However, we discovered it didn’t work right out of the box. Turns out, it requires a firmware update (which tipped me off since Skype for Business was still referred to as Lync on the phone). Fair enough, however, even after updating to the latest version (5.9) it STILL didn’t work. We would get stuck at “Unable to fetch user certificate” (or some variation thereof).  What could be going on? Well, it turns out 5.9 isn’t yet supported by Microsoft (as of Jan 30th, 2019), so Skype for Business Online won’t work. The second latest version, 5.8.2, does work. So, if you are having trouble with the phone and signing into Skype for Business Online, try updating the firmware (or downgrading) and try again.

Azure AD Connect Not Synchronizing to Office 365 – No Errors

Recently we had a very frustrating, yet somewhat easy to fix issue involving Office 365 and Azure Active Directory Connect. Essentially, the synchronization was happening but none of the Active Directory objects were syncing. As a matter of fact, all previous synced users stopped syncing and ended up existing in the cloud only.

When looking at the Office 365 admin portal, there were no synchronization errors shown. Rather, it was showing the synchronization as happening every 30 minutes as usual. When looking at the synchronization service itself, there were no errors shown there either.

Yet distribution groups no longer contained users. New users existed only in Active Directory and not in Office 365.

Many hours were spent talking to Microsoft support over this issue. Installing Azure AD Connect on different servers and different domain controllers gave the same behavior. PowerShell commands did nothing. It was only when I was looking at our other clients AD Connect settings that I discovered that the Source Anchor was wrong.

See, we had been installing Azure AD Connect with express settings. Yet rather than defaulting to ms-ds-consistencyguid for the Source Anchor, express settings set the Source Anchor as domain.onmicrosoft.com – aad. Anyone who knows Azure AD Connect should immediately recognize that this will not work. Even Microsoft says that if the Source Anchor is wrong, there will be no synchronization errors while nothing synchronizes.

So why did the Source Anchor default to something so wrong? Why did it default wrong on different servers? This part remains a mystery.

The fix used to be simple. Even Microsoft’s current documentation explains to use the Azure AD Connect Wizard to change the Source Anchor. However, that is no longer an option with the newer versions of Azure AD Connect.

Microsoft says Configure Source Anchor should be here, but it is not.

In the end, we had to uninstall Azure AD Connect and reinstall it manually (versus the express settings). Once there, we were able to select the correct Source Anchor (ms-ds-consistencyguid) and proceed with the install. Within minutes, all items synchronized properly and as expected.

Go here to change the SourceAnchor

This did not randomly break. Sync had broken two months prior and Microsoft instructed us to reinstall Azure AD Connect using express settings. They had even looked at the installed settings and said it was correct. It was then that we figure the Source Anchor changed its default from the default to domain.onmicrosoft.com -aad instead. After that, it took weeks for anyone to realize anything was wrong (again, as the sync service showed as operating normally and Active Directory changes are few and far between). When we did finally discover this, Microsoft spent many hours troubleshooting on the phone with us and within remote sessions and still didn’t catch that the Source Anchor was wrong.

Proper Source Anchor settings. This was changed to domain.onmicrosoft.com -aad for some reason, causing synchronization to break.

So, there you have it, if your synchronization is ‘working’ but nothing is synching, make sure your Source Anchor did not default to something else other than ms-ds-consistencyguid.

iPhone Contact Details Not Saving

My wife recently came to me with a simple problem – when she tried to update certain contact details on her iPhone, they would disappear and not save. I verified this issue by adding a birthday to someone’s name. As soon as the contacts window was closed and reopened, the birthday was gone. Even a simple change such as changing the number type from ‘mobile’ to ‘iPhone’ wouldn’t stick, reverting back to ‘mobile’.

iPhone as the phone number type, vs mobile. This would not save after the contacts window was closed.

A quick look at her settings revealed her contacts was synced to Gmail, which I assumed was the culprit. However, a cursory search didn’t come up with much as most others had synchronization issues. I had to dig pretty deep, but I did discover someone having a similar issue –  Birthdays wouldn’t save unless the year was included. Sure enough, my wife wasn’t putting the year in for birthdays. I tested this by adding a birthday year in with a contact of hers. It saved. Ah ha. It turns out, Gmail can only save certain details in certain formats. This differs from iCloud, which allows phone number types such as ‘iPhone’ to be saved, along side year less birthdays.

The fix was easy enough – we moved her from using Gmail as her contacts sync to iCloud. Now she can change the little details and add birthdays to contacts whom she knows the days, but not the years they were born.

Misc. details about this specific post:
iPhone 8
iOS 12.0
Issue date: Oct. 6th, 2018

Critical Error when entering Office 365 Exchange

UPDATE: This issue has since been resolved.

We got a ticket today about a user getting a critical error when trying to enter Office 365 Exchange. I recall a similar issue happening to me a week prior, so I took a look. Sure enough, when entering exchange, I get this error:

Here is the full report:

Client Information
——————

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.140 Safari/537.36

CPU Class: undefined
Platform: Win32
System Language: undefined
User Language: en-US
CookieEnabled: true

—————–
Exception Details
—————–

Date: Thu Feb 08 2018 09:31:21 GMT-0700 (Mountain Standard Time)
Message: Script error.
Url:
Line: 0

Call Stack
———-

Error

    at Function.ErrorHandling.$Ee (https://r1.res.office365.com/ecp/15.20.464.16/scripts/common.js:1:189941)

    at ErrorHandling.showUnhandledException (https://r1.res.office365.com/ecp/15.20.464.16/scripts/common.js:1:189026)

Dump Event
———-

                isTrusted = true
                message = Script error.
                filename =
                lineno = 0
                colno = 0
                error = null
                NONE = 0
                CAPTURING_PHASE = 1
                AT_TARGET = 2
                BUBBLING_PHASE = 3
                type = error
                target = [object Window]
                currentTarget = [object Window]
                eventPhase = 2
                bubbles = false
                cancelable = true
                defaultPrevented = false
                composed = false
                timeStamp = 7115.5000000144355
                srcElement = [object Window]
                returnValue = true
                cancelBubble = false
                path = [object Window]
                composedPath = function composedPath() { [native code] }
                stopPropagation = function stopPropagation() { [native code] }
                stopImmediatePropagation = function stopImmediatePropagation() { [native code] }
                preventDefault = function preventDefault() { [native code] }
                initEvent = function initEvent() { [native code] }

Detailed Call Stack
———–

This error message occurs across two of three admin accounts on three computers. Oddly, the third account does not suffer this error. It also occurs across at least three browsers – Edge, Internet Explorer 11, and Chrome (Ver 64, both normal and incognito).

As of now, I know of no fix, and several cursory google searches where fruitless. This only started to occur within the past month (as I go into exchange on a monthly basis) and sadly there are no recent posts anywhere about this. If anyone has a clue on how to fix this, or if we simply have to wait until Microsoft does something on their end, let me know. Otherwise we just have to deal with this very minor nuisance.

SMTP Stopped Sending Mail to Office 365

Had a client whose SMTP server suddenly stopped sending scan to e-mails to their domain cloud Office 365 e-mail from their on-premise SMTP server. The queue folder quickly filled up with over a hundred e-mails, so what could have caused this problem?

After three days of troubleshooting, I finally figured it out. In the event log, I was seeing the following issue:

A total of three IP addresses were rejecting the mail, which never even left the queue folder to generate a log in the badmail folder. The three IP addresses were ‘207.46.163.42’, ‘207.46.163.74’, and ‘216.32.180.10’. A cursory search on Google revealed these to be Microsoft servers.

Now this worked perfectly just a few days ago, with no error message like it in the event logs prior to the 23rd of January, 2018. Could it really be this simple of a fix? I tried resetting IIS 6.0, started and stopped the SMTP service many time. I added smtp.office365.com as a smart host, I added a relay as detailed in several websites (including Microsoft’s), and I event played with different settings. Nothing worked… but I did notice occasionally a few e-mails would kick off at a time before failing again as I made changes. Now, I can’t say I know a lot about TLS, but it is my understanding that TLS required authentication and had to go across port 587.

Apparently not. I went back into IIS 6.0 Manager and right clicking on SMTP Virtual server and went to properties. Under the delivery tab, I clicked Outbound Security… at the bottom. There I clicked TLS encryption. I made zero other changes.

I clicked OK and then clicked apply. I didn’t even have to restart the SMTP service – all the e-mail in the queue folder cleared out within moments. Finally, a victory. As a final test, I sent test scans to our domain e-mail account from the Xerox SMTP configuration as well as sending another IT technician to send scans at the printers to themselves. It all worked like it should.

I’m not entirely sure why this works, but I would love if someone could fill me in. Remember, it worked days ago, then suddenly it stopped working until I enabled TLS encryption. I did not configure any ports other than 25.

KB4056894 May Break Hyper-V VMs

UPDATE: This issue has since been resolved.

KB4056894 has the potential to break Windows 2008 R2 Hyper-V hosts. The server itself comes up just fine, however the VMs get stuck in restoring mode at 0%. This poses a huge problem. So far this has only happened to one of our host servers with the rest coming up normally (Server 2012, Server 2012 R2, and one other Server 2008 R2 host). So how do you fix this issue? So long as the host comes up, simply uninstall the offending patch and restart. When we did this, the VMs immediately booted back up.

It is interesting to note that at one point Microsoft pulled these patches, but has obviously made them available again. Thankfully, the only issue we have had affected only this one server (so far) out of several hosts and over a hundred VMs. Of course, now Intel recommends that you skip those patches completely.

Random Black Screens with nVidia Graphics Card and Windows 7

For several months, I had dealt with an increasingly frustrating issue with my monitor’s screen suddenly going to sleep. Nothing I could do, aside from unplug it from the desktop and plug it back in, could get it to wake up. It certainly wasn’t a monitor issue (though I had those too), as swapping the monitor resulted in the same behavior. It wasn’t a cable issue, nor was it an issue with Windows power settings. I had narrowed it down to single point of failure – driver issues. My NVIDIA driver kept crashing and its frequency was increasing. It went from only happening in games to happening in Office and general browsing.

I had taken some steps to fix this – reverting to old drivers, installing new drivers from a completely clean slate. Nothing worked. Then I found this thread on the EVGA forums. The thread included details that matches my symptoms, down to the event log. According to that thread, it was an issue between the driver series and Windows 7. Even reverting to old drivers didn’t work as I simply hadn’t gone back long enough, as this issue has plagued the driver series for months.

So, I did what any sane person would do – I upgraded to Windows 10.

It has been two weeks now and not a single crash or blip in the event logs about a driver crash.

Now if you are having a similar issue, what do you do? Upgrading to Windows 10 for free ends December 31st, so after that you will need to pay a hefty sum for a license. I wish I can say I have a fix for users still on Windows 7 (aside from going back long enough on the driver history to avoid the bad drivers), but perhaps the thread above will.

Firewall unable to turn on (Windows Error code 0x80070422)

An odd issue, Windows Firewall was automatically off. This impacted a user who had numerous issues with their computer. The issue wasn’t hard to fix, but was certainly not easy to find a fix for.

A co-worker approached me asking for help to troubleshoot the firewall issue. We employ GPO for the firewall, but to my knowledge we do not disable the firewall.

After looking, it was obvious that this was no GPO that disabled the firewall. However, when we went to enable it we got an error code that the firewall could not be re-enabled, with error code 0x80070422.

 

After some digging, we discovered that the firewall was disabled automatically via Services. We simply had to choose “Automatic” for startup, then start the firewall through Services.

 

Once that was done, we checked back on the firewall. Sure enough, the Firewall was re-enabled. We restarted the computer to ensure that the firewall would automatically start, which it did.

Keep in mind, you must be an administrator to make changes to services.